SSHFS Backup Mount again not working with version 6.3.6

[Amahr]

Hi,
sadly today after installing the version 6.3.6 update the SSHFS backup stopped working again.

To get SSHFS to work with prior 6.3.5 versions i had use a workaround already.
With version 6.3.5 I needed to revert all changes that I had made to get SSHFS working again.
Now with version 6.3.6 it's not working again as it won't mount the directory.

I removed the directory, changed the backup to local, changed the backup back to SSHFS so it will create a new folder and mountpoint. Everything seems to be fine until the backup begins.

backup
Aug 27 14:28:59 box:settings initCache: pre-load settings
Aug 27 14:28:59 box:taskworker Starting task 2192. Logs are at /home/yellowtent/platformdata/logs/tasks/2192.log
Aug 27 14:28:59 box:tasks 2192: {"percent":2,"error":null}
Aug 27 14:28:59 box:tasks 2192: {"percent":1,"message":"Backing up XXXXXXXX"}
Aug 27 14:28:59 box:backups XXXXXXXX Backed up. Took 0.003 seconds
Aug 27 14:28:59 box:tasks 2192: {"percent":21,"message":"Backing up XXXXXXXX"}
Aug 27 14:28:59 box:tasks 2192: {"percent":41,"message":"Snapshotting app XXXXXXXX"}
Aug 27 14:28:59 box:services XXXXXXXX backupAddons
Aug 27 14:28:59 box:services XXXXXXXX backupAddons: Backing up ["postgresql","sendmail","ldap","redis","localstorage","scheduler","turn"]
Aug 27 14:28:59 box:services XXXXXXXX Backing up postgresql
Aug 27 14:29:00 box:services XXXXXXXX Backing up redis
Aug 27 14:29:00 box:backups XXXXXXXX snapshotApp: took 1.577 seconds
Aug 27 14:29:00 box:tasks 2192: {"percent":41,"message":"Uploading app snapshot XXXXXXXX"}
Aug 27 14:29:00 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a tgz {"localRoot":"/home/yellowtent/appsdata/6118113f-e41a-4b8e-b15b-6bffb0c19f5a","layout":[]}
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.375Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/6118113f-e41a-4b8e-b15b-6bffb0c19f5a","layout":[]} to snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.402Z box:settings initCache: pre-load settings
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.411Z box:backupupload process: rss: 72269824 heapTotal: 38535168 heapUsed: 25351672 external: 2996080
Aug 27 14:29:01 box:backupupload v8 heap : used 25356000 total: 38535168 max: 4345298944
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.411Z box:backups upload: id snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a format tgz dataLayout {"localRoot":"/home/yellowtent/appsdata/6118113f-e41a-4b8e-b15b-6bffb0c19f5a","layout":[]}
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.432Z box:storage/filesystem checkPreconditions: getting disk usage of /home/yellowtent/appsdata/6118113f-e41a-4b8e-b15b-6bffb0c19f5a
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.479Z box:storage/filesystem checkPreconditions: 1415554334 bytes
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a (stdout): 2021-08-27T12:29:01.489Z box:backupupload upload completed. error: BoxError: /mnt/cloudronbackup is not mounted
at /home/yellowtent/box/src/storage/filesystem.js:80:77
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
reason: 'FileSystem Error',
details: {}
Aug 27 14:29:01 box:backups runBackupUpload: result - {"result":"/mnt/cloudronbackup is not mounted"}
Aug 27 14:29:01 box:shell backup-snapshot/app_6118113f-e41a-4b8e-b15b-6bffb0c19f5a code: 50, signal: null
Aug 27 14:29:01 box:backups XXXXXXXX Unable to backup BoxError: /mnt/cloudronbackup is not mounted
at /home/yellowtent/box/src/backups.js:903:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:77:9)
at ChildProcess.emit (events.js:315:20)
at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) {
reason: 'External Error',
details: {}
Aug 27 14:29:01 box:taskworker Task took 2.183 seconds
Aug 27 14:29:01 box:tasks setCompleted - 2192: {"result":null,"error":{"stack":"BoxError: /mnt/cloudronbackup is not mounted\n at /home/yellowtent/box/src/backups.js:903:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:77:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"/mnt/cloudronbackup is not mounted"}}
Aug 27 14:29:01 box:tasks 2192: {"percent":100,"result":null,"error":{"stack":"BoxError: /mnt/cloudronbackup is not mounted\n at /home/yellowtent/box/src/backups.js:903:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:77:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"/mnt/cloudronbackup is not mounted"}}

{
  "taskId": "2192",
  "errorMessage": "/mnt/cloudronbackup is not mounted",
  "timedOut": false,
  "backupId": null
}

Directory rights as made by Cloudron:

[nebulon]

@amahr can you verify that your server has a systemd unit file at /etc/systemd/system/mnt-cloudronbackup.mount as well as check if you can remount it then with systemd restart mnt-cloudronbackup

Finally does the mountpoint show up in the df output?

[Amahr]

@nebulon the server has no systemd unit file at /etc/systemd/system/mnt-cloudronbackup.mount and it does not show up in the df output.

If I remove the directory, set local backup, and reconfigure SSHFS:

• I get a systemd unit file at /etc/systemd/system/mnt-cloudronbackup.mount
  Content of mnt-cloudronbackup.mount:

• df shows no mount

• systemd restart mnt-cloudronbackup says "Excess arguments."

[nebulon]

@amahr sorry the command to restart/remount would be:

systemctl restart mnt-cloudronbackup.mount

[Amahr]

@nebulon all good, thank you.
I also tried systemctl restart mnt-cloudronbackup.mount, wasn't working too.

I now tried again a ssh connection from the cloudron server to the backup server with the privkey, it did not work.
Both on the cloudron and on the backup server the rights seemed to have changed for the .ssh directories. Everything else was configured as before.

Now the backups are working again, but I remain clueless why this happened.

Thank you @nebulon for your help and excellent response time!

[robi]

@nebulon said in SSHFS Backup Mount again not working with version 6.3.6:

@amahr sorry the command to restart/remount would be:

systemctl restart mnt-cloudronbackup.mount

It would be nice to have this exposed in the storage UI