Hetzner Storage Box & Backup: Failed to mount

[uwcrbc]

Hi all,

This could be linked to symptoms described here but also maybe not.

This is happening on 3 out of 6 servers all on Cloudron v8.0.6 and Ubuntu v24.04lts:

• on 3 servers I am unable to mount Hetzner backup (storage box via SSHFS). The error given in Cloudron is the following: "Failed to mount (inactive): Could not determine mount failure reason"
  

• looking in the log of the 2 servers I see the following entries when trying to mount the Backup Storage:

ct 01 15:15:28 box:shell remountMount /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remountmount.sh /mnt/cloudronbackup
[no timestamp]  dependency job for mnt-cloudronbackup.mount failed. See 'journalctl -xe' for details.
Oct 01 15:15:29 box:shell remountMount: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remountmount.sh /mnt/cloudronbackup errored BoxError: remountMount exited with code 1 signal null
[no timestamp]  at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
[no timestamp]  reason: 'Shell Error',
[no timestamp]  details: {},
[no timestamp]  code: 1,
[no timestamp]  signal: null
[no timestamp]  }
Oct 01 15:15:30 box:apphealthmonitor app health: 2 running / 0 stopped / 0 unresponsive
Oct 01 15:15:31 box:shell getStatus execArgs: mountpoint ["-q","--","/mnt/cloudronbackup"]
Oct 01 15:15:31 box:shell getStatus: mountpoint with args -q -- /mnt/cloudronbackup errored Error: Command failed: mountpoint -q -- /mnt/cloudronbackup
[no timestamp] 
[no timestamp]  at genericNodeError (node:internal/errors:984:15)
[no timestamp]  at wrappedFn (node:internal/errors:538:14)
[no timestamp]  at ChildProcess.exithandler (node:child_process:422:12)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at maybeClose (node:internal/child_process:1105:16)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
[no timestamp]  code: 32,
[no timestamp]  killed: false,
[no timestamp]  signal: null,
[no timestamp]  cmd: 'mountpoint -q -- /mnt/cloudronbackup'
[no timestamp]  }

• all servers (the 3 failing and the 3 working) backups are configured the same and backing up to the same Hetzner Storage Box
• backup configurations on all servers have been set long time ago and numerous backups were successful.
• each server use a different hetzner subaccount and own credential/key. This is the only setting that differs from one server to the next
• all failing servers jumped from 8.0.4 to 8.0.6, with no successful backup since 8.0.4
• some working servers also jumped from 8.0.4 to 8.0.6 but not all.
• restarting a failing server does not solve the issue

So at the moment, I have 3 out of 6 servers not able to back things up.
Based on this, does anyone have any idea what is going on and where to find/dig up more info?

Thanks in advance for any pointers

[nebulon]

Just to rule out some potential issues, can you SSH into the storage box from one of the Cloudrons which fail to mount it?

The instructions to SSH are at https://docs.hetzner.com/storage/storage-box/access/access-ssh-rsync-borg/

[uwcrbc]

Hi @nebulon

Yes, it is possible to ssh into the storage box both from a local machine as well as from a Cloudron server.

[nebulon]

And are you able to mount this manually using fuse sshfs ? Cloudron would generate a systemd .mount file like this:

$ cat /etc/systemd/system/mnt-cloudronbackup.mount 
[Unit]
Description=backup

Requires=unbound.service
After=unbound.service
Before=docker.service


[Mount]
What=u123456@u123456.your-storagebox.de:.
Where=/mnt/cloudronbackup
Options=allow_other,port=23,IdentityFile=/home/yellowtent/platformdata/sshfs/id_rsa_u123456.your-storagebox.de,StrictHostKeyChecking=no,reconnect
Type=fuse.sshfs

[Install]
WantedBy=multi-user.target

[uwcrbc]

Not too sure what you mean from fuse sshfs, but from the cloudron server I can SSHFS and mount the remote storage backup location just fine, using the same/relevant subaccount.

Did you mean something else?

[nebulon]

Ok so that part works then. Looks like some issue on the Cloudron platform side, we may have to debug this on one of the failing servers then. Could you open remote SSH for us and send a mail to support@cloudron.io with the info of that server where you have the manual mount working but Cloudron can't do this?

[uwcrbc]

Thanks for this - Email sent.

[nebulon]

We were able to resolve this. The root cause was the missing unbound-anchor ubuntu package. After intstalling this it worked.

More info on that issue is at https://forum.cloudron.io/topic/12496/unbound-anchor-not-found-in-ubuntu-24-04?_=1727881329798