Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Brite
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Backup failing (flaky CIFS mount)

Backup failing (flaky CIFS mount)

Scheduled Pinned Locked Moved Solved Support
backupshetznermountsshfs
26 Posts 7 Posters 4.3k Views 7 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by girish
    #1

    I got this notification:

    Backup failed: Unknown system error -128: Unknown system error -128, mkdir. Logs are available here.

    Here are the logs:

    Dec 24 03:51:13 box:backupupload v8 heap : used 31653696 total: 43773952 max: 1526909922
    Dec 24 03:51:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27187M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:51:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27236M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:51:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27267M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.905Z box:backupupload process: rss: 42160128 heapTotal: 43773952 heapUsed: 32011952 external: 5064058
    Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.908Z box:backupupload v8 heap : used 32012088 total: 43773952 max: 1526909922
    Dec 24 03:51:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27296M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:52:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27335M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:52:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27383M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload process: rss: 61669376 heapTotal: 42725376 heapUsed: 30804040 external: 4619642
    Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload v8 heap : used 30804176 total: 42725376 max: 1526909922
    Dec 24 03:52:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27435M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:52:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27485M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:52:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27516M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.904Z box:backupupload process: rss: 38522880 heapTotal: 41152512 heapUsed: 30619232 external: 7814522
    Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.912Z box:backupupload v8 heap : used 30619368 total: 41152512 max: 1526909922
    Dec 24 03:52:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27549M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:53:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27594M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:53:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27642M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:53:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:13.897Z box:backupupload process: rss: 62439424 heapTotal: 41152512 heapUsed: 30215640 external: 11222906
    Dec 24 03:53:13 box:backupupload v8 heap : used 30215776 total: 41152512 max: 1526909922
    Dec 24 03:53:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27691M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:53:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27739M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:53:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27766M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:53:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:43.897Z box:backupupload process: rss: 50860032 heapTotal: 40103936 heapUsed: 29142336 external: 3502970
    Dec 24 03:53:43 box:backupupload v8 heap : used 29142472 total: 40103936 max: 1526909922
    Dec 24 03:53:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27797M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:54:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27843M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:54:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27894M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.896Z box:backupupload process: rss: 59416576 heapTotal: 40103936 heapUsed: 28492712 external: 455034
    Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.897Z box:backupupload v8 heap : used 28492848 total: 40103936 max: 1526909922
    Dec 24 03:54:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27944M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:54:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27991M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:54:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28022M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.913Z box:backupupload process: rss: 35647488 heapTotal: 40103936 heapUsed: 28482184 external: 2145146
    Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.923Z box:backupupload v8 heap : used 28482320 total: 40103936 max: 1526909922
    Dec 24 03:54:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28058M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:02 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28103M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:12 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28158M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:13.902Z box:backupupload process: rss: 39378944 heapTotal: 41152512 heapUsed: 29247864 external: 12602234
    Dec 24 03:55:13 box:backupupload v8 heap : used 29248000 total: 41152512 max: 1526909922
    Dec 24 03:55:22 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28211M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:32 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28259M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:42 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28288M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:43.902Z box:backupupload process: rss: 37691392 heapTotal: 40103936 heapUsed: 29441000 external: 16565114
    Dec 24 03:55:43 box:backupupload v8 heap : used 29441136 total: 40103936 max: 1526909922
    Dec 24 03:55:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28322M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:55:58 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28344M@4MBps (office.uniteddiversity.coop)"}
    Dec 24 03:56:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:13.902Z box:backupupload process: rss: 51838976 heapTotal: 43249664 heapUsed: 31012632 external: 1346066
    Dec 24 03:56:13 box:backupupload v8 heap : used 31012768 total: 43249664 max: 1526909922
    Dec 24 03:56:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:43.902Z box:backupupload process: rss: 52428800 heapTotal: 43249664 heapUsed: 31273968 external: 2607346
    Dec 24 03:56:43 box:backupupload v8 heap : used 31274104 total: 43249664 max: 1526909922
    Dec 24 03:57:03 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:03.845Z box:storage/filesystem [/mnt/backup/snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f.tar.gz] upload: out stream error. { [Error: EBADF: bad file descriptor, close] errno: -9, code: 'EBADF', syscall: 'close' }
    Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload process: rss: 53137408 heapTotal: 43249664 heapUsed: 31447408 external: 3270754
    Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload v8 heap : used 31447544 total: 43249664 max: 1526909922
    Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:19.109Z box:backupupload upload completed. error: { BoxError: Unknown system error -128: Unknown system error -128, mkdir
    at /home/yellowtent/box/src/storage/filesystem.js:93:36
    at FSReqWrap.args [as oncomplete] (fs.js:140:20)
    name: 'BoxError',
    reason: 'External Error',
    details: {},
    message:
    'Unknown system error -128: Unknown system error -128, mkdir' }
    Dec 24 03:57:19 box:backups runBackupUpload: result - {"result":"Unknown system error -128: Unknown system error -128, mkdir"}
    Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f code: 50, signal: null
    Dec 24 03:57:19 box:backups office.uniteddiversity.coop Unable to backup { BoxError: Unknown system error -128: Unknown system error -128, mkdir
    at /home/yellowtent/box/src/backups.js:874:29
    at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
    at ChildProcess.emit (events.js:198:13)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
    name: 'BoxError',
    reason: 'External Error',
    details: {},
    message:
    'Unknown system error -128: Unknown system error -128, mkdir' }
    Dec 24 03:57:19 box:taskworker Task took 7038.095 seconds
    Dec 24 03:57:19 box:tasks setCompleted - 3286: {"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
    Dec 24 03:57:19 box:tasks 3286: {"percent":100,"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
    

    I figured I'd start by just trying to run the backup again, but it fails immediately and I get this:

    Command failed with exit code 1 (EPERM): df -kPT /mnt/backup

    Which I guess means the backup mount isn't mounted.... so I rebooted to see if that would fix that...which it did... so now seeing if a backup will now complete...

    I use Cloudron with Gandi & Hetzner

    jdaviescoatesJ P girishG 3 Replies Last reply
    1
    • jdaviescoatesJ jdaviescoates

      I got this notification:

      Backup failed: Unknown system error -128: Unknown system error -128, mkdir. Logs are available here.

      Here are the logs:

      Dec 24 03:51:13 box:backupupload v8 heap : used 31653696 total: 43773952 max: 1526909922
      Dec 24 03:51:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27187M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:51:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27236M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:51:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27267M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.905Z box:backupupload process: rss: 42160128 heapTotal: 43773952 heapUsed: 32011952 external: 5064058
      Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.908Z box:backupupload v8 heap : used 32012088 total: 43773952 max: 1526909922
      Dec 24 03:51:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27296M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:52:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27335M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:52:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27383M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload process: rss: 61669376 heapTotal: 42725376 heapUsed: 30804040 external: 4619642
      Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload v8 heap : used 30804176 total: 42725376 max: 1526909922
      Dec 24 03:52:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27435M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:52:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27485M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:52:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27516M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.904Z box:backupupload process: rss: 38522880 heapTotal: 41152512 heapUsed: 30619232 external: 7814522
      Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.912Z box:backupupload v8 heap : used 30619368 total: 41152512 max: 1526909922
      Dec 24 03:52:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27549M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:53:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27594M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:53:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27642M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:53:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:13.897Z box:backupupload process: rss: 62439424 heapTotal: 41152512 heapUsed: 30215640 external: 11222906
      Dec 24 03:53:13 box:backupupload v8 heap : used 30215776 total: 41152512 max: 1526909922
      Dec 24 03:53:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27691M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:53:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27739M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:53:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27766M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:53:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:43.897Z box:backupupload process: rss: 50860032 heapTotal: 40103936 heapUsed: 29142336 external: 3502970
      Dec 24 03:53:43 box:backupupload v8 heap : used 29142472 total: 40103936 max: 1526909922
      Dec 24 03:53:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27797M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:54:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27843M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:54:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27894M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.896Z box:backupupload process: rss: 59416576 heapTotal: 40103936 heapUsed: 28492712 external: 455034
      Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.897Z box:backupupload v8 heap : used 28492848 total: 40103936 max: 1526909922
      Dec 24 03:54:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27944M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:54:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27991M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:54:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28022M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.913Z box:backupupload process: rss: 35647488 heapTotal: 40103936 heapUsed: 28482184 external: 2145146
      Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.923Z box:backupupload v8 heap : used 28482320 total: 40103936 max: 1526909922
      Dec 24 03:54:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28058M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:02 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28103M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:12 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28158M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:13.902Z box:backupupload process: rss: 39378944 heapTotal: 41152512 heapUsed: 29247864 external: 12602234
      Dec 24 03:55:13 box:backupupload v8 heap : used 29248000 total: 41152512 max: 1526909922
      Dec 24 03:55:22 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28211M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:32 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28259M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:42 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28288M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:43.902Z box:backupupload process: rss: 37691392 heapTotal: 40103936 heapUsed: 29441000 external: 16565114
      Dec 24 03:55:43 box:backupupload v8 heap : used 29441136 total: 40103936 max: 1526909922
      Dec 24 03:55:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28322M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:55:58 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28344M@4MBps (office.uniteddiversity.coop)"}
      Dec 24 03:56:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:13.902Z box:backupupload process: rss: 51838976 heapTotal: 43249664 heapUsed: 31012632 external: 1346066
      Dec 24 03:56:13 box:backupupload v8 heap : used 31012768 total: 43249664 max: 1526909922
      Dec 24 03:56:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:43.902Z box:backupupload process: rss: 52428800 heapTotal: 43249664 heapUsed: 31273968 external: 2607346
      Dec 24 03:56:43 box:backupupload v8 heap : used 31274104 total: 43249664 max: 1526909922
      Dec 24 03:57:03 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:03.845Z box:storage/filesystem [/mnt/backup/snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f.tar.gz] upload: out stream error. { [Error: EBADF: bad file descriptor, close] errno: -9, code: 'EBADF', syscall: 'close' }
      Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload process: rss: 53137408 heapTotal: 43249664 heapUsed: 31447408 external: 3270754
      Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload v8 heap : used 31447544 total: 43249664 max: 1526909922
      Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:19.109Z box:backupupload upload completed. error: { BoxError: Unknown system error -128: Unknown system error -128, mkdir
      at /home/yellowtent/box/src/storage/filesystem.js:93:36
      at FSReqWrap.args [as oncomplete] (fs.js:140:20)
      name: 'BoxError',
      reason: 'External Error',
      details: {},
      message:
      'Unknown system error -128: Unknown system error -128, mkdir' }
      Dec 24 03:57:19 box:backups runBackupUpload: result - {"result":"Unknown system error -128: Unknown system error -128, mkdir"}
      Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f code: 50, signal: null
      Dec 24 03:57:19 box:backups office.uniteddiversity.coop Unable to backup { BoxError: Unknown system error -128: Unknown system error -128, mkdir
      at /home/yellowtent/box/src/backups.js:874:29
      at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
      at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
      at ChildProcess.emit (events.js:198:13)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
      name: 'BoxError',
      reason: 'External Error',
      details: {},
      message:
      'Unknown system error -128: Unknown system error -128, mkdir' }
      Dec 24 03:57:19 box:taskworker Task took 7038.095 seconds
      Dec 24 03:57:19 box:tasks setCompleted - 3286: {"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
      Dec 24 03:57:19 box:tasks 3286: {"percent":100,"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
      

      I figured I'd start by just trying to run the backup again, but it fails immediately and I get this:

      Command failed with exit code 1 (EPERM): df -kPT /mnt/backup

      Which I guess means the backup mount isn't mounted.... so I rebooted to see if that would fix that...which it did... so now seeing if a backup will now complete...

      jdaviescoatesJ Offline
      jdaviescoatesJ Offline
      jdaviescoates
      wrote on last edited by
      #2

      Backing up post-reboot worked fine. So my guess is that my Hetzner Storage Box had a wobble and somehow became unmounted mid backup.

      I use Cloudron with Gandi & Hetzner

      jdaviescoatesJ 1 Reply Last reply
      0
      • robiR Offline
        robiR Offline
        robi
        wrote on last edited by
        #3

        we need better retries for mounts and DB connections during long running processes such as backups.

        Conscious tech

        1 Reply Last reply
        3
        • jdaviescoatesJ jdaviescoates

          I got this notification:

          Backup failed: Unknown system error -128: Unknown system error -128, mkdir. Logs are available here.

          Here are the logs:

          Dec 24 03:51:13 box:backupupload v8 heap : used 31653696 total: 43773952 max: 1526909922
          Dec 24 03:51:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27187M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:51:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27236M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:51:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27267M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.905Z box:backupupload process: rss: 42160128 heapTotal: 43773952 heapUsed: 32011952 external: 5064058
          Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.908Z box:backupupload v8 heap : used 32012088 total: 43773952 max: 1526909922
          Dec 24 03:51:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27296M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:52:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27335M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:52:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27383M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload process: rss: 61669376 heapTotal: 42725376 heapUsed: 30804040 external: 4619642
          Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload v8 heap : used 30804176 total: 42725376 max: 1526909922
          Dec 24 03:52:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27435M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:52:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27485M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:52:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27516M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.904Z box:backupupload process: rss: 38522880 heapTotal: 41152512 heapUsed: 30619232 external: 7814522
          Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.912Z box:backupupload v8 heap : used 30619368 total: 41152512 max: 1526909922
          Dec 24 03:52:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27549M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:53:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27594M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:53:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27642M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:53:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:13.897Z box:backupupload process: rss: 62439424 heapTotal: 41152512 heapUsed: 30215640 external: 11222906
          Dec 24 03:53:13 box:backupupload v8 heap : used 30215776 total: 41152512 max: 1526909922
          Dec 24 03:53:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27691M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:53:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27739M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:53:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27766M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:53:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:43.897Z box:backupupload process: rss: 50860032 heapTotal: 40103936 heapUsed: 29142336 external: 3502970
          Dec 24 03:53:43 box:backupupload v8 heap : used 29142472 total: 40103936 max: 1526909922
          Dec 24 03:53:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27797M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:54:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27843M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:54:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27894M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.896Z box:backupupload process: rss: 59416576 heapTotal: 40103936 heapUsed: 28492712 external: 455034
          Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.897Z box:backupupload v8 heap : used 28492848 total: 40103936 max: 1526909922
          Dec 24 03:54:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27944M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:54:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27991M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:54:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28022M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.913Z box:backupupload process: rss: 35647488 heapTotal: 40103936 heapUsed: 28482184 external: 2145146
          Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.923Z box:backupupload v8 heap : used 28482320 total: 40103936 max: 1526909922
          Dec 24 03:54:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28058M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:02 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28103M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:12 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28158M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:13.902Z box:backupupload process: rss: 39378944 heapTotal: 41152512 heapUsed: 29247864 external: 12602234
          Dec 24 03:55:13 box:backupupload v8 heap : used 29248000 total: 41152512 max: 1526909922
          Dec 24 03:55:22 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28211M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:32 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28259M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:42 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28288M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:43.902Z box:backupupload process: rss: 37691392 heapTotal: 40103936 heapUsed: 29441000 external: 16565114
          Dec 24 03:55:43 box:backupupload v8 heap : used 29441136 total: 40103936 max: 1526909922
          Dec 24 03:55:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28322M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:55:58 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28344M@4MBps (office.uniteddiversity.coop)"}
          Dec 24 03:56:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:13.902Z box:backupupload process: rss: 51838976 heapTotal: 43249664 heapUsed: 31012632 external: 1346066
          Dec 24 03:56:13 box:backupupload v8 heap : used 31012768 total: 43249664 max: 1526909922
          Dec 24 03:56:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:43.902Z box:backupupload process: rss: 52428800 heapTotal: 43249664 heapUsed: 31273968 external: 2607346
          Dec 24 03:56:43 box:backupupload v8 heap : used 31274104 total: 43249664 max: 1526909922
          Dec 24 03:57:03 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:03.845Z box:storage/filesystem [/mnt/backup/snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f.tar.gz] upload: out stream error. { [Error: EBADF: bad file descriptor, close] errno: -9, code: 'EBADF', syscall: 'close' }
          Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload process: rss: 53137408 heapTotal: 43249664 heapUsed: 31447408 external: 3270754
          Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload v8 heap : used 31447544 total: 43249664 max: 1526909922
          Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:19.109Z box:backupupload upload completed. error: { BoxError: Unknown system error -128: Unknown system error -128, mkdir
          at /home/yellowtent/box/src/storage/filesystem.js:93:36
          at FSReqWrap.args [as oncomplete] (fs.js:140:20)
          name: 'BoxError',
          reason: 'External Error',
          details: {},
          message:
          'Unknown system error -128: Unknown system error -128, mkdir' }
          Dec 24 03:57:19 box:backups runBackupUpload: result - {"result":"Unknown system error -128: Unknown system error -128, mkdir"}
          Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f code: 50, signal: null
          Dec 24 03:57:19 box:backups office.uniteddiversity.coop Unable to backup { BoxError: Unknown system error -128: Unknown system error -128, mkdir
          at /home/yellowtent/box/src/backups.js:874:29
          at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
          at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
          at ChildProcess.emit (events.js:198:13)
          at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
          name: 'BoxError',
          reason: 'External Error',
          details: {},
          message:
          'Unknown system error -128: Unknown system error -128, mkdir' }
          Dec 24 03:57:19 box:taskworker Task took 7038.095 seconds
          Dec 24 03:57:19 box:tasks setCompleted - 3286: {"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
          Dec 24 03:57:19 box:tasks 3286: {"percent":100,"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
          

          I figured I'd start by just trying to run the backup again, but it fails immediately and I get this:

          Command failed with exit code 1 (EPERM): df -kPT /mnt/backup

          Which I guess means the backup mount isn't mounted.... so I rebooted to see if that would fix that...which it did... so now seeing if a backup will now complete...

          P Offline
          P Offline
          p44
          translator
          wrote on last edited by
          #4

          @jdaviescoates I had a similar problem. I don't know if @nebulon reached Hetzner provider to ask more info's about long-time mounting of Box service.

          1 Reply Last reply
          1
          • I Offline
            I Offline
            iqweb
            wrote on last edited by
            #5

            same here on a Hetzner storage box 2 days back - couldn't manually remount either. Had to reboot & then remounted automatically as per /etc/fstab.

            1 Reply Last reply
            2
            • jdaviescoatesJ jdaviescoates

              I got this notification:

              Backup failed: Unknown system error -128: Unknown system error -128, mkdir. Logs are available here.

              Here are the logs:

              Dec 24 03:51:13 box:backupupload v8 heap : used 31653696 total: 43773952 max: 1526909922
              Dec 24 03:51:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27187M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:51:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27236M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:51:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27267M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.905Z box:backupupload process: rss: 42160128 heapTotal: 43773952 heapUsed: 32011952 external: 5064058
              Dec 24 03:51:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:51:43.908Z box:backupupload v8 heap : used 32012088 total: 43773952 max: 1526909922
              Dec 24 03:51:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27296M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:52:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27335M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:52:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27383M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload process: rss: 61669376 heapTotal: 42725376 heapUsed: 30804040 external: 4619642
              Dec 24 03:52:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:13.895Z box:backupupload v8 heap : used 30804176 total: 42725376 max: 1526909922
              Dec 24 03:52:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27435M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:52:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27485M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:52:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27516M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.904Z box:backupupload process: rss: 38522880 heapTotal: 41152512 heapUsed: 30619232 external: 7814522
              Dec 24 03:52:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:52:43.912Z box:backupupload v8 heap : used 30619368 total: 41152512 max: 1526909922
              Dec 24 03:52:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27549M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:53:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27594M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:53:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27642M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:53:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:13.897Z box:backupupload process: rss: 62439424 heapTotal: 41152512 heapUsed: 30215640 external: 11222906
              Dec 24 03:53:13 box:backupupload v8 heap : used 30215776 total: 41152512 max: 1526909922
              Dec 24 03:53:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27691M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:53:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27739M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:53:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27766M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:53:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:53:43.897Z box:backupupload process: rss: 50860032 heapTotal: 40103936 heapUsed: 29142336 external: 3502970
              Dec 24 03:53:43 box:backupupload v8 heap : used 29142472 total: 40103936 max: 1526909922
              Dec 24 03:53:51 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27797M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:54:01 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27843M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:54:11 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27894M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.896Z box:backupupload process: rss: 59416576 heapTotal: 40103936 heapUsed: 28492712 external: 455034
              Dec 24 03:54:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:13.897Z box:backupupload v8 heap : used 28492848 total: 40103936 max: 1526909922
              Dec 24 03:54:21 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27944M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:54:31 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 27991M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:54:41 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28022M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.913Z box:backupupload process: rss: 35647488 heapTotal: 40103936 heapUsed: 28482184 external: 2145146
              Dec 24 03:54:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:54:43.923Z box:backupupload v8 heap : used 28482320 total: 40103936 max: 1526909922
              Dec 24 03:54:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28058M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:02 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28103M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:12 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28158M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:13.902Z box:backupupload process: rss: 39378944 heapTotal: 41152512 heapUsed: 29247864 external: 12602234
              Dec 24 03:55:13 box:backupupload v8 heap : used 29248000 total: 41152512 max: 1526909922
              Dec 24 03:55:22 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28211M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:32 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28259M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:42 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28288M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:55:43.902Z box:backupupload process: rss: 37691392 heapTotal: 40103936 heapUsed: 29441000 external: 16565114
              Dec 24 03:55:43 box:backupupload v8 heap : used 29441136 total: 40103936 max: 1526909922
              Dec 24 03:55:52 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28322M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:55:58 box:tasks 3286: {"percent":72.4285714285714,"message":"Uploading backup 28344M@4MBps (office.uniteddiversity.coop)"}
              Dec 24 03:56:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:13.902Z box:backupupload process: rss: 51838976 heapTotal: 43249664 heapUsed: 31012632 external: 1346066
              Dec 24 03:56:13 box:backupupload v8 heap : used 31012768 total: 43249664 max: 1526909922
              Dec 24 03:56:43 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:56:43.902Z box:backupupload process: rss: 52428800 heapTotal: 43249664 heapUsed: 31273968 external: 2607346
              Dec 24 03:56:43 box:backupupload v8 heap : used 31274104 total: 43249664 max: 1526909922
              Dec 24 03:57:03 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:03.845Z box:storage/filesystem [/mnt/backup/snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f.tar.gz] upload: out stream error. { [Error: EBADF: bad file descriptor, close] errno: -9, code: 'EBADF', syscall: 'close' }
              Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload process: rss: 53137408 heapTotal: 43249664 heapUsed: 31447408 external: 3270754
              Dec 24 03:57:13 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:13.904Z box:backupupload v8 heap : used 31447544 total: 43249664 max: 1526909922
              Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f (stdout): 2020-12-24T03:57:19.109Z box:backupupload upload completed. error: { BoxError: Unknown system error -128: Unknown system error -128, mkdir
              at /home/yellowtent/box/src/storage/filesystem.js:93:36
              at FSReqWrap.args [as oncomplete] (fs.js:140:20)
              name: 'BoxError',
              reason: 'External Error',
              details: {},
              message:
              'Unknown system error -128: Unknown system error -128, mkdir' }
              Dec 24 03:57:19 box:backups runBackupUpload: result - {"result":"Unknown system error -128: Unknown system error -128, mkdir"}
              Dec 24 03:57:19 box:shell backup-snapshot/app_d0e0fa7c-e358-4d93-869b-e227feec349f code: 50, signal: null
              Dec 24 03:57:19 box:backups office.uniteddiversity.coop Unable to backup { BoxError: Unknown system error -128: Unknown system error -128, mkdir
              at /home/yellowtent/box/src/backups.js:874:29
              at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
              at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
              at ChildProcess.emit (events.js:198:13)
              at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
              name: 'BoxError',
              reason: 'External Error',
              details: {},
              message:
              'Unknown system error -128: Unknown system error -128, mkdir' }
              Dec 24 03:57:19 box:taskworker Task took 7038.095 seconds
              Dec 24 03:57:19 box:tasks setCompleted - 3286: {"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
              Dec 24 03:57:19 box:tasks 3286: {"percent":100,"result":null,"error":{"stack":"BoxError: Unknown system error -128: Unknown system error -128, mkdir\n at /home/yellowtent/box/src/backups.js:874:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Unknown system error -128: Unknown system error -128, mkdir"}}
              

              I figured I'd start by just trying to run the backup again, but it fails immediately and I get this:

              Command failed with exit code 1 (EPERM): df -kPT /mnt/backup

              Which I guess means the backup mount isn't mounted.... so I rebooted to see if that would fix that...which it did... so now seeing if a backup will now complete...

              girishG Do not disturb
              girishG Do not disturb
              girish
              Staff
              wrote on last edited by
              #6

              @jdaviescoates Did this get sorted out?

              1 Reply Last reply
              0
              • jdaviescoatesJ jdaviescoates

                Backing up post-reboot worked fine. So my guess is that my Hetzner Storage Box had a wobble and somehow became unmounted mid backup.

                jdaviescoatesJ Offline
                jdaviescoatesJ Offline
                jdaviescoates
                wrote on last edited by jdaviescoates
                #7

                @girish yes:

                @jdaviescoates said in Backup failing:

                Backing up post-reboot worked fine. So my guess is that my Hetzner Storage Box had a wobble and somehow became unmounted mid backup.

                Sounds like others have been experiencing similar too, so I'm guessing a (hopefully temporary) issue with Hetzner storage boxes.

                Although like @robi said it'd be cool if there was some way Cloudron backups could try to survive such wobbles.

                I use Cloudron with Gandi & Hetzner

                girishG 1 Reply Last reply
                2
                • jdaviescoatesJ jdaviescoates

                  @girish yes:

                  @jdaviescoates said in Backup failing:

                  Backing up post-reboot worked fine. So my guess is that my Hetzner Storage Box had a wobble and somehow became unmounted mid backup.

                  Sounds like others have been experiencing similar too, so I'm guessing a (hopefully temporary) issue with Hetzner storage boxes.

                  Although like @robi said it'd be cool if there was some way Cloudron backups could try to survive such wobbles.

                  girishG Do not disturb
                  girishG Do not disturb
                  girish
                  Staff
                  wrote on last edited by
                  #8

                  @jdaviescoates Yes, at some point we want to make it possible to add mounts easily. When we do that, we can attempt to re-mount automatically, I guess.

                  robiR 1 Reply Last reply
                  2
                  • girishG girish

                    @jdaviescoates Yes, at some point we want to make it possible to add mounts easily. When we do that, we can attempt to re-mount automatically, I guess.

                    robiR Offline
                    robiR Offline
                    robi
                    wrote on last edited by
                    #9

                    @girish not sure this is the right context..

                    this is about existing mounts and their resilience to disconnects.

                    • monitoring mounts for health
                    • allowing for self-healing if it exists from the protocol
                    • if not, try harder, remount command for example
                    • if still not, try unmount, then remount
                    • if still not, notify to reboot

                    Conscious tech

                    P 1 Reply Last reply
                    1
                    • robiR robi

                      @girish not sure this is the right context..

                      this is about existing mounts and their resilience to disconnects.

                      • monitoring mounts for health
                      • allowing for self-healing if it exists from the protocol
                      • if not, try harder, remount command for example
                      • if still not, try unmount, then remount
                      • if still not, notify to reboot
                      P Offline
                      P Offline
                      p44
                      translator
                      wrote on last edited by
                      #10

                      @robi right suggestions.

                      @girish I had this problem again: all backup folder this morning where unmounted...

                      I 1 Reply Last reply
                      0
                      • P p44

                        @robi right suggestions.

                        @girish I had this problem again: all backup folder this morning where unmounted...

                        I Offline
                        I Offline
                        iqweb
                        wrote on last edited by
                        #11

                        @p44 said in Backup failing:

                        @robi right suggestions.

                        @girish I had this problem again: all backup folder this morning where unmounted...

                        I've had the same thing again too

                        girishG P 2 Replies Last reply
                        1
                        • I iqweb

                          @p44 said in Backup failing:

                          @robi right suggestions.

                          @girish I had this problem again: all backup folder this morning where unmounted...

                          I've had the same thing again too

                          girishG Do not disturb
                          girishG Do not disturb
                          girish
                          Staff
                          wrote on last edited by
                          #12

                          @iqweb Are you also using CIFS/NFS?

                          I 1 Reply Last reply
                          1
                          • I iqweb

                            @p44 said in Backup failing:

                            @robi right suggestions.

                            @girish I had this problem again: all backup folder this morning where unmounted...

                            I've had the same thing again too

                            P Offline
                            P Offline
                            p44
                            translator
                            wrote on last edited by
                            #13

                            @iqweb Still on Hetzner storage?

                            1 Reply Last reply
                            0
                            • girishG girish

                              @iqweb Are you also using CIFS/NFS?

                              I Offline
                              I Offline
                              iqweb
                              wrote on last edited by
                              #14

                              @girish said in Backup failing:

                              @iqweb Are you also using CIFS/NFS?

                              Yes - storage provider: CIFS Mount

                              @p44 said in Backup failing:

                              @iqweb Still on Hetzner storage?

                              Yes - Hetzner storage box

                              & again - I had to reboot to be able to remount.

                              P 1 Reply Last reply
                              1
                              • I iqweb

                                @girish said in Backup failing:

                                @iqweb Are you also using CIFS/NFS?

                                Yes - storage provider: CIFS Mount

                                @p44 said in Backup failing:

                                @iqweb Still on Hetzner storage?

                                Yes - Hetzner storage box

                                & again - I had to reboot to be able to remount.

                                P Offline
                                P Offline
                                p44
                                translator
                                wrote on last edited by
                                #15

                                @iqweb Maybe Hetzner sometimes reboot Box instance? From my side is about one week CIFS mount is stable...

                                1 Reply Last reply
                                0
                                • nebulonN Offline
                                  nebulonN Offline
                                  nebulon
                                  Staff
                                  wrote on last edited by
                                  #16

                                  I also have to remount my cifs share about once a month. In my case I thought it was related to my residential home internet connection. Either way I think we have to handle such mount points better and maybe even attempt a remount automatically.

                                  P girishG 2 Replies Last reply
                                  3
                                  • nebulonN nebulon

                                    I also have to remount my cifs share about once a month. In my case I thought it was related to my residential home internet connection. Either way I think we have to handle such mount points better and maybe even attempt a remount automatically.

                                    P Offline
                                    P Offline
                                    p44
                                    translator
                                    wrote on last edited by
                                    #17

                                    @nebulon Good idea to add a health-check to keep alive mounting

                                    1 Reply Last reply
                                    1
                                    • nebulonN nebulon

                                      I also have to remount my cifs share about once a month. In my case I thought it was related to my residential home internet connection. Either way I think we have to handle such mount points better and maybe even attempt a remount automatically.

                                      girishG Do not disturb
                                      girishG Do not disturb
                                      girish
                                      Staff
                                      wrote on last edited by
                                      #18

                                      @nebulon Does cifs mounting have some retry flags? (i.e the fstab options)

                                      nebulonN 1 Reply Last reply
                                      0
                                      • girishG girish

                                        @nebulon Does cifs mounting have some retry flags? (i.e the fstab options)

                                        nebulonN Offline
                                        nebulonN Offline
                                        nebulon
                                        Staff
                                        wrote on last edited by
                                        #19

                                        @girish not that I have found and https://linux.die.net/man/8/mount.cifs does not mention anything.

                                        Apparently systemd has some automounting feature https://www.freedesktop.org/software/systemd/man/systemd.automount.html which mounts the drive once the mountpoint is accessed by a process! Can't tell if it would reconnect such a mount though.

                                        girishG 1 Reply Last reply
                                        1
                                        • nebulonN nebulon

                                          @girish not that I have found and https://linux.die.net/man/8/mount.cifs does not mention anything.

                                          Apparently systemd has some automounting feature https://www.freedesktop.org/software/systemd/man/systemd.automount.html which mounts the drive once the mountpoint is accessed by a process! Can't tell if it would reconnect such a mount though.

                                          girishG Do not disturb
                                          girishG Do not disturb
                                          girish
                                          Staff
                                          wrote on last edited by
                                          #20

                                          @nebulon Maybe https://askubuntu.com/questions/1210867/remount-cifs-on-network-reconnect ? (just expands on your link)

                                          imc67I 1 Reply Last reply
                                          1
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search