Cloudron 9.0.8 Backup to site sshfs rsync hetzner storagebox

[avatar1024]

@joseph how come as this been marked as solved? There have been no response and it clearly wasn't as of 9.0.12.

[joseph]

I thought this was fixed, but I see there is a new fix by @nebulon in https://git.cloudron.io/platform/box/-/commit/c2378d33b453a441709c585ab235102d61bcc1d4 . @nebulon should this issue be considered fixed?

[nebulon]

Yes the first was the fix to the backup and second then the fix to the cleanup, which I forgot initially

[avatar1024]

Just for info, another instance auto updated last night from 8.3.2 to 9.0.13 and something weird happened: I got the same error on the backup site (with two failed backup notifications) AND that backup site got duplicated into a new one but for some reasons as a tar archive instead of rsync like the original site. No idea what happened there.

[girish]

@avatar1024 in the previous versions, one could change the backup format between tgz and rsync . We used to track the format of each backup (for restore from old backups to work). in Cloudron 9, you cannot change the backup format of a site once set. instead, you can just create a new backup site if you want another format.

The migration code probably detected that you have backups in both formats. So, it would create two sites - one tgz and one rsync. That's what you are seeing .

[avatar1024]

Hello, I seem to be facing the issue again on one instance.

One backup site on one instance throw this error (BoxError: SSH connection error: ssh exited with code 255 signal null) repeatedly on scheduled backup. When I remount the volume manually and start backup now manually it works fine, but then fail again at the next scheduled backup.

I have another instance using the exact same volume with same config (sshfs + rsync) and it works just fine. Both are on Cloudron 10.0.5.

Here is the log:

2026-10-14T00:00:01.889Z taskworker: Starting task 24823. Logs are at /home/yellowtent/platformdata/logs/tasks/24823.log
(node:3648779) ExperimentalWarning: The supports Web Crypto API method is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:3648779) ExperimentalWarning: The ML-DSA-44 Web Crypto API algorithm is an experimental feature and might change at any time
2026-10-14T00:00:01.925Z taskworker: Running task of type backup
2026-10-14T00:00:01.965Z tasks: updating task 24823 with: {"percent":4.448275862068965,"message":"Backing up pdf.catalystcollective.org (1/26). Waiting for lock"}
2026-10-14T00:00:01.977Z locks: write: current locks: {"full_backup_task_547c3c3f-b9ee-42ff-afce-c7c84d259d2a":null,"app_backup_018ee793-8184-4666-aaee-0fcfa2e9eb8d":"24823"}
2026-10-14T00:00:01.977Z locks: acquire: app_backup_018ee793-8184-4666-aaee-0fcfa2e9eb8d
2026-10-14T00:00:01.978Z tasks: updating task 24823 with: {"percent":4.448275862068965,"message":"Snapshotting app pdf.catalystcollective.org"}
2026-10-14T00:00:01.982Z services: backupAddons: backing up ["localstorage","oidc"]
2026-10-14T00:00:01.983Z backuptask: snapshotApp: pdf.catalystcollective.org took 0.005 seconds
2026-10-14T00:00:01.984Z tasks: updating task 24823 with: {"percent":4.448275862068965,"message":"Uploading app snapshot pdf.catalystcollective.org"}
2026-10-14T00:00:01.984Z backuptask: runBackupUpload: adjusting heap size to 2816M
2026-10-14T00:00:01.985Z shell: backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d 547c3c3f-b9ee-42ff-afce-c7c84d259d2a {"localRoot":"/home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d","layout":[]}
2026-10-14T00:00:03.719Z backupupload: Backing up {"localRoot":"/home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d","layout":[]} to snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d
2026-10-14T00:00:03.721Z backuptask: upload: path snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a dataLayout {"localRoot":"/home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d","layout":[]}
(node:3649474) ExperimentalWarning: The supports Web Crypto API method is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:3649474) ExperimentalWarning: The ML-DSA-44 Web Crypto API algorithm is an experimental feature and might change at any time
2026-10-14T00:00:03.751Z shell: mounts: mountpoint -q -- /mnt/cloudronbackup
2026-10-14T00:00:03.766Z backuptask: checkPreconditions: mount point status is {"state":"active","message":""}
2026-10-14T00:00:03.766Z shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup
2026-10-14T00:00:03.787Z backuptask: checkPreconditions: getting disk usage of /home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d
2026-10-14T00:00:03.787Z shell: backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d
2026-10-14T00:00:03.792Z backuptask: checkPreconditions: total required=356352 available=768645791744
2026-10-14T00:00:03.793Z backupformat/rsync: upload: uploading to site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a path snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d (encrypted: false) dataLayout {"localRoot":"/home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d","layout":[]}
2026-10-14T00:00:03.793Z shell: backupformat/rsync: find /home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d -type d -empty
2026-10-14T00:00:03.799Z shell: backupformat/rsync: find /home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d -type f -executable
2026-10-14T00:00:03.803Z shell: backupformat/rsync: find /home/yellowtent/appsdata/018ee793-8184-4666-aaee-0fcfa2e9eb8d -type l
2026-10-14T00:00:03.809Z syncer: sync: loading cache file /home/yellowtent/platformdata/backup/547c3c3f-b9ee-42ff-afce-c7c84d259d2a/018ee793-8184-4666-aaee-0fcfa2e9eb8d.sync.cache
2026-10-14T00:00:03.813Z backupformat/rsync: sync: processing 0 deletes, 2 additions and 11 unchanged
2026-10-14T00:00:03.814Z backupformat/rsync: sync: done processing deletes. error: null
2026-10-14T00:00:03.817Z tasks: updating task 24823 with: {"percent":4.448275862068965,"message":"Uploaded 0/2 files. Current: config.json (pdf.catalystcollective.org)"}
2026-10-14T00:00:03.865Z backupformat/rsync: sync: done processing adds. error: null
2026-10-14T00:00:03.866Z syncer: finalize: patching in integrity information into /home/yellowtent/platformdata/backup/547c3c3f-b9ee-42ff-afce-c7c84d259d2a/018ee793-8184-4666-aaee-0fcfa2e9eb8d.sync.cache
2026-10-14T00:00:03.870Z backuptask: upload: path snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a uploaded: {"transferred":7672,"size":254217,"fileCount":13}
2026-10-14T00:00:03.893Z backupupload: upload completed. error: null
2026-10-14T00:00:03.894Z backuptask: runBackupUpload: result - {"result":{"stats":{"transferred":7672,"size":254217,"fileCount":13},"integrity":{"signature":"5c31c321a9f85da989a030297e542fe6f4d6879ae5ce5356179077e9a0ce3d89e7a0b9d613c6edd34c355a91cdf06fd664c73a5a316942b5e3532f5c2bc2b40b"}}}
2026-10-14T00:00:03.922Z backuptask: uploadAppSnapshot: pdf.catalystcollective.org uploaded to snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d. 1.938 seconds
2026-10-14T00:00:03.925Z backuptask: backupAppWithTag: rotating pdf.catalystcollective.org snapshot of 547c3c3f-b9ee-42ff-afce-c7c84d259d2a to path 2026-10-14-000001-930/app_pdf.catalystcollective.org_v2.44.0
2026-10-14T00:00:03.942Z tasks: updating task 24823 with: {"percent":4.448275862068965,"message":"Copying /mnt/cloudronbackup/snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d to /mnt/cloudronbackup/2026-10-14-000001-930/app_pdf.catalystcollective.org_v2.44.0"}
2026-10-14T00:00:03.942Z shell: filesystem: ssh -o StrictHostKeyChecking no -i /tmp/identity_file_22cca67b-0dc8-421d-bcf2-f27d3bf2008e -p 23 u273486@u273486.your-storagebox.de cp -aRl /home/Catalyst/snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d /home/Catalyst/2026-10-14-000001-930/app_pdf.catalystcollective.org_v2.44.0
2026-10-14T00:00:03.979Z shell: filesystem: ssh -o StrictHostKeyChecking no -i /tmp/identity_file_22cca67b-0dc8-421d-bcf2-f27d3bf2008e -p 23 u273486@u273486.your-storagebox.de cp -aRl /home/Catalyst/snapshot/app_018ee793-8184-4666-aaee-0fcfa2e9eb8d /home/Catalyst/2026-10-14-000001-930/app_pdf.catalystcollective.org_v2.44.0 errored with code 255 and signal null timeout false terminated false - stdout: "" - stderr: "ssh: Could not resolve hostname u273486.your-storagebox.de: Temporary failure in name resolution
\n"
2026-10-14T00:00:03.979Z backuptask: copy: copy to 2026-10-14-000001-930/app_pdf.catalystcollective.org_v2.44.0 errored. error: SSH connection error: ssh exited with code 255 signal null
2026-10-14T00:00:03.981Z backuptask: fullBackup: app pdf.catalystcollective.org backup finished. Took 2.004 seconds
2026-10-14T00:00:03.984Z locks: write: current locks: {"full_backup_task_547c3c3f-b9ee-42ff-afce-c7c84d259d2a":null}
2026-10-14T00:00:03.984Z locks: release: app_backup_018ee793-8184-4666-aaee-0fcfa2e9eb8d
2026-10-14T00:00:03.984Z tasks: setCompleted - 24823: {"result":null,"error":{"message":"SSH connection error: ssh exited with code 255 signal null","reason":"External Error","stack":"BoxError: SSH connection error: ssh exited with code 255 signal null\\n    at copyInternal (file:///home/yellowtent/box/src/storage/filesystem.js:221:49)\\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\\n    at async Object.copyDir (file:///home/yellowtent/box/src/storage/filesystem.js:245:12)\\n    at async Object.copy (file:///home/yellowtent/box/src/backupformat/rsync.js:148:5)"},"percent":100}
2026-10-14T00:00:03.984Z tasks: updating task 24823 with: {"completed":true,"result":null,"error":{"message":"SSH connection error: ssh exited with code 255 signal null","reason":"External Error","stack":"BoxError: SSH connection error: ssh exited with code 255 signal null\\n    at copyInternal (file:///home/yellowtent/box/src/storage/filesystem.js:221:49)\\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\\n    at async Object.copyDir (file:///home/yellowtent/box/src/storage/filesystem.js:245:12)\\n    at async Object.copy (file:///home/yellowtent/box/src/backupformat/rsync.js:148:5)"},"percent":100}
2026-10-14T00:00:03.987Z taskworker: Task took 2.103 seconds
2026-10-14T00:00:03.987Z BoxError: SSH connection error: ssh exited with code 255 signal null\n    at copyInternal (file:///home/yellowtent/box/src/storage/filesystem.js:221:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Object.copyDir (file:///home/yellowtent/box/src/storage/filesystem.js:245:12)\n    at async Object.copy (file:///home/yellowtent/box/src/backupformat/rsync.js:148:5)
2026-10-14T00:00:03.987Z Exiting with code 0

[nebulon]

Seems like ssh: Could not resolve hostname u273486.your-storagebox.de: Temporary failure in name resolution is the clue here. Is there anything specific about DNS resolving on this failing instance compared to your working one?

[avatar1024]

@nebulon said:

Is there anything specific about DNS resolving on this failing instance compared to your working one?

Thanks. How could I check that?

One thing I can say is that I haven't made any tweaks or specific DNS config on that instance.

[james]

Hello @avatar1024

DNS test from your Cloudron server

From your Cloudron server in an ssh session run:

dig A u273486.your-storagebox.de

This should return the IPv4 address:

; <<>> DiG 9.20.24-1ubuntu0.3-Ubuntu <<>> A u273486.your-storagebox.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43382
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;u273486.your-storagebox.de.    IN      A

;; ANSWER SECTION:
u273486.your-storagebox.de. 60  IN      A       91.98.245.216

;; Query time: 2 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Oct 14 10:04:28 UTC 2026
;; MSG SIZE  rcvd: 71

You can test a different DNS server like that:

dig A u273486.your-storagebox.de @1.1.1.1

If the first command does not deliver a result there is a DNS issue on your system.
But that could resolve itself after some time.

---

Important:
Hetzner Storage Boxes and SSHFS have a fail2ban set up.
If your Cloudron system fails to authenticate multiple times your Cloudron IP address will get blocked for 15 to 30 minutes.

---

Local testing:

You can check the sshfs connection to the Hetzner Storage Box locally with:

sshfs -p 23 u273486@u273486.your-storagebox.de:/home $WHERE_YOU_WANT_IT_MOUNTED

If you have the ssh key configured correctly on your Hetzner Storage Box and your local system it should mount it without asking for a password.
When it asks for a password, the ssh key authentication failed and it falls back to password.

[avatar1024]

@james Thanks for this.

dig A u273486.your-storagebox.de returns exactly that:

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> A u273486.your-storagebox.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4122
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;u273486.your-storagebox.de.    IN      A

;; ANSWER SECTION:
u273486.your-storagebox.de. 30  IN      A       91.98.245.216

;; Query time: 8 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Oct 14 10:30:49 UTC 2026
;; MSG SIZE  rcvd: 71

But actually I've just tried to "run back now" and it just worked without having to remount the volume...but I bet I'm still gonna get a "fail to backup" error at 2am later on.

[avatar1024]

...wait I spoke too quickly, the backup did not complete right now. Here is the truncated log (too long otherwise):

2026-10-14T10:36:11.666Z backupupload: Backing up {"localRoot":"/home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884","layout":[]} to snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884
2026-10-14T10:36:11.669Z backuptask: upload: path snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884 site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a dataLayout {"localRoot":"/home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884","layout":[]}
(node:3802437) ExperimentalWarning: The supports Web Crypto API method is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:3802437) ExperimentalWarning: The ML-DSA-44 Web Crypto API algorithm is an experimental feature and might change at any time
2026-10-14T10:36:11.693Z shell: mounts: mountpoint -q -- /mnt/cloudronbackup
2026-10-14T10:36:11.703Z backuptask: checkPreconditions: mount point status is {"state":"active","message":""}
2026-10-14T10:36:11.703Z shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup
2026-10-14T10:36:11.720Z backuptask: checkPreconditions: getting disk usage of /home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884
2026-10-14T10:36:11.720Z shell: backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884
2026-10-14T10:36:11.729Z backuptask: checkPreconditions: total required=495616 available=768729284608
2026-10-14T10:36:11.730Z backupformat/rsync: upload: uploading to site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a path snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884 (encrypted: false) dataLayout {"localRoot":"/home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884","layout":[]}
2026-10-14T10:36:11.730Z shell: backupformat/rsync: find /home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884 -type d -empty
2026-10-14T10:36:11.733Z shell: backupformat/rsync: find /home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884 -type f -executable
2026-10-14T10:36:11.736Z shell: backupformat/rsync: find /home/yellowtent/appsdata/94ee96d3-7845-40a1-9abf-bc621082c884 -type l
2026-10-14T10:36:11.740Z syncer: sync: loading cache file /home/yellowtent/platformdata/backup/547c3c3f-b9ee-42ff-afce-c7c84d259d2a/94ee96d3-7845-40a1-9abf-bc621082c884.sync.cache
2026-10-14T10:36:11.743Z backupformat/rsync: sync: processing 0 deletes, 3 additions and 3 unchanged
2026-10-14T10:36:11.743Z backupformat/rsync: sync: done processing deletes. error: null
2026-10-14T10:36:11.745Z tasks: updating task 24827 with: {"percent":52.72413793103449,"message":"Uploaded 0/3 files. Current: config.json (email.collectiveownership.co.uk)"}
2026-10-14T10:36:11.825Z backupformat/rsync: sync: done processing adds. error: null
2026-10-14T10:36:11.825Z syncer: finalize: patching in integrity information into /home/yellowtent/platformdata/backup/547c3c3f-b9ee-42ff-afce-c7c84d259d2a/94ee96d3-7845-40a1-9abf-bc621082c884.sync.cache
2026-10-14T10:36:11.829Z backuptask: upload: path snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884 site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a uploaded: {"transferred":458889,"size":459061,"fileCount":6}
2026-10-14T10:36:11.852Z backupupload: upload completed. error: null
2026-10-14T10:36:11.852Z backuptask: runBackupUpload: result - {"result":{"stats":{"transferred":458889,"size":459061,"fileCount":6},"integrity":{"signature":"63fa907073cc0e71334619e2fa63f420daa7fc971c9e3a5836d2d45932b17f592af6dff0991061e09b9b6b38c352659434a13c35b73d66f57c8ce1199ac96301"}}}
2026-10-14T10:36:11.873Z backuptask: uploadAppSnapshot: email.collectiveownership.co.uk uploaded to snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884. 1.592 seconds
2026-10-14T10:36:11.876Z backuptask: backupAppWithTag: rotating email.collectiveownership.co.uk snapshot of 547c3c3f-b9ee-42ff-afce-c7c84d259d2a to path 2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4
2026-10-14T10:36:11.882Z tasks: updating task 24827 with: {"percent":52.72413793103449,"message":"Copying /mnt/cloudronbackup/snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884 to /mnt/cloudronbackup/2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4"}
2026-10-14T10:36:11.882Z shell: filesystem: ssh -o StrictHostKeyChecking no -i /tmp/identity_file_7e45d22e-2d28-4a03-8e72-408377dbf77e -p 23 u273486@u273486.your-storagebox.de cp -aRl /home/Catalyst/snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884 /home/Catalyst/2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4
2026-10-14T10:36:12.273Z backuptask: copy: copied successfully to 2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4. Took 0.391 seconds
2026-10-14T10:36:12.274Z tasks: updating task 24827 with: {"percent":52.72413793103449,"message":"Copying /mnt/cloudronbackup/snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884.backupinfo to /mnt/cloudronbackup/2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4.backupinfo"}
2026-10-14T10:36:12.274Z shell: filesystem: ssh -o StrictHostKeyChecking no -i /tmp/identity_file_47b1f455-ce5a-4d17-8571-3cac4cf27c46 -p 23 u273486@u273486.your-storagebox.de cp -al /home/Catalyst/snapshot/app_94ee96d3-7845-40a1-9abf-bc621082c884.backupinfo /home/Catalyst/2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4.backupinfo
2026-10-14T10:36:12.640Z backuptask: copy: copied backupinfo successfully to 2026-10-14-103420-487/app_email.collectiveownership.co.uk_v2.10.4.backupinfo
2026-10-14T10:36:12.644Z backuptask: fullBackup: app email.collectiveownership.co.uk backup finished. Took 2.498 seconds
2026-10-14T10:36:12.647Z locks: write: current locks: {"full_backup_task_547c3c3f-b9ee-42ff-afce-c7c84d259d2a":null}
2026-10-14T10:36:12.647Z locks: release: app_backup_94ee96d3-7845-40a1-9abf-bc621082c884
2026-10-14T10:36:12.647Z tasks: updating task 24827 with: {"percent":56.17241379310346,"message":"Backing up row.collectiveownership.co.uk (16/26). Waiting for lock"}
2026-10-14T10:36:12.651Z locks: write: current locks: {"full_backup_task_547c3c3f-b9ee-42ff-afce-c7c84d259d2a":null,"app_backup_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b":"24827"}
2026-10-14T10:36:12.651Z locks: acquire: app_backup_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b
2026-10-14T10:36:12.651Z tasks: updating task 24827 with: {"percent":56.17241379310346,"message":"Snapshotting app row.collectiveownership.co.uk"}
2026-10-14T10:36:12.652Z services: backupAddons: backing up ["postgresql","sendmail","redis","localstorage"]
2026-10-14T10:36:12.652Z services: Backing up postgresql
pipeRequestToFile: connected with status code 200
pipeRequestToFile: success
2026-10-14T10:36:18.045Z services: Backing up redis
pipeRequestToFile: connected with status code 200
pipeRequestToFile: success
2026-10-14T10:36:18.094Z backuptask: snapshotApp: row.collectiveownership.co.uk took 5.443 seconds
2026-10-14T10:36:18.094Z tasks: updating task 24827 with: {"percent":56.17241379310346,"message":"Uploading app snapshot row.collectiveownership.co.uk"}
2026-10-14T10:36:18.095Z backuptask: runBackupUpload: adjusting heap size to 2816M
2026-10-14T10:36:18.095Z shell: backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b 547c3c3f-b9ee-42ff-afce-c7c84d259d2a {"localRoot":"/home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b","layout":[]}
2026-10-14T10:36:19.424Z backupupload: Backing up {"localRoot":"/home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b","layout":[]} to snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b
2026-10-14T10:36:19.427Z backuptask: upload: path snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a dataLayout {"localRoot":"/home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b","layout":[]}
(node:3802476) ExperimentalWarning: The supports Web Crypto API method is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:3802476) ExperimentalWarning: The ML-DSA-44 Web Crypto API algorithm is an experimental feature and might change at any time
2026-10-14T10:36:19.452Z shell: mounts: mountpoint -q -- /mnt/cloudronbackup
2026-10-14T10:36:19.462Z backuptask: checkPreconditions: mount point status is {"state":"active","message":""}
2026-10-14T10:36:19.462Z shell: df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup
2026-10-14T10:36:19.481Z backuptask: checkPreconditions: getting disk usage of /home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b
2026-10-14T10:36:19.481Z shell: backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b
2026-10-14T10:36:19.781Z backuptask: checkPreconditions: total required=273408000 available=768729153536
2026-10-14T10:36:19.781Z backupformat/rsync: upload: uploading to site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a path snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b (encrypted: false) dataLayout {"localRoot":"/home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b","layout":[]}
2026-10-14T10:36:19.782Z shell: backupformat/rsync: find /home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b -type d -empty
2026-10-14T10:36:19.796Z shell: backupformat/rsync: find /home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b -type f -executable
2026-10-14T10:36:19.821Z shell: backupformat/rsync: find /home/yellowtent/appsdata/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b -type l
2026-10-14T10:36:19.836Z syncer: sync: loading cache file /home/yellowtent/platformdata/backup/547c3c3f-b9ee-42ff-afce-c7c84d259d2a/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b.sync.cache
2026-10-14T10:36:19.990Z backupformat/rsync: sync: processing 0 deletes, 4 additions and 7678 unchanged
2026-10-14T10:36:19.996Z backupformat/rsync: sync: done processing deletes. error: null
2026-10-14T10:36:19.998Z tasks: updating task 24827 with: {"percent":56.17241379310346,"message":"Uploaded 0/4 files. Current: config.json (row.collectiveownership.co.uk)"}
2026-10-14T10:36:20.317Z backupformat/rsync: sync: done processing adds. error: null
2026-10-14T10:36:20.317Z syncer: finalize: patching in integrity information into /home/yellowtent/platformdata/backup/547c3c3f-b9ee-42ff-afce-c7c84d259d2a/99de5b4d-6ad8-4f37-b08f-f6e110be4b2b.sync.cache
2026-10-14T10:36:20.410Z backuptask: upload: path snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b site 547c3c3f-b9ee-42ff-afce-c7c84d259d2a uploaded: {"transferred":34096594,"size":253477399,"fileCount":7682}
2026-10-14T10:36:20.474Z backupupload: upload completed. error: null
2026-10-14T10:36:20.475Z backuptask: runBackupUpload: result - {"result":{"stats":{"transferred":34096594,"size":253477399,"fileCount":7682},"integrity":{"signature":"ec763ed7a16f91c6afa7455cdbc83ad6d7b07393d7a1575a6e14999f219a2a5d9bf27fa46799203110e16fb4a70965b5e3ba6ef21a11441e3ee7bcb8b3f4cd01"}}}
2026-10-14T10:36:20.507Z backuptask: uploadAppSnapshot: row.collectiveownership.co.uk uploaded to snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b. 2.413 seconds
2026-10-14T10:36:20.509Z backuptask: backupAppWithTag: rotating row.collectiveownership.co.uk snapshot of 547c3c3f-b9ee-42ff-afce-c7c84d259d2a to path 2026-10-14-103420-487/app_row.collectiveownership.co.uk_v1.40.0
2026-10-14T10:36:20.521Z tasks: updating task 24827 with: {"percent":56.17241379310346,"message":"Copying /mnt/cloudronbackup/snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b to /mnt/cloudronbackup/2026-10-14-103420-487/app_row.collectiveownership.co.uk_v1.40.0"}
2026-10-14T10:36:20.521Z shell: filesystem: ssh -o StrictHostKeyChecking no -i /tmp/identity_file_1f662b17-d900-44e5-b6b5-51c29c075f1c -p 23 u273486@u273486.your-storagebox.de cp -aRl /home/Catalyst/snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b /home/Catalyst/2026-10-14-103420-487/app_row.collectiveownership.co.uk_v1.40.0
2026-10-14T10:36:31.461Z shell: filesystem: ssh -o StrictHostKeyChecking no -i /tmp/identity_file_1f662b17-d900-44e5-b6b5-51c29c075f1c -p 23 u273486@u273486.your-storagebox.de cp -aRl /home/Catalyst/snapshot/app_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b /home/Catalyst/2026-10-14-103420-487/app_row.collectiveownership.co.uk_v1.40.0 errored with code 255 and signal null timeout false terminated false - stdout: "" - stderr: "ssh: Could not resolve hostname u273486.your-storagebox.de: Temporary failure in name resolution
\n"
2026-10-14T10:36:31.461Z backuptask: copy: copy to 2026-10-14-103420-487/app_row.collectiveownership.co.uk_v1.40.0 errored. error: SSH connection error: ssh exited with code 255 signal null
2026-10-14T10:36:31.464Z backuptask: fullBackup: app row.collectiveownership.co.uk backup finished. Took 18.813 seconds
2026-10-14T10:36:31.467Z locks: write: current locks: {"full_backup_task_547c3c3f-b9ee-42ff-afce-c7c84d259d2a":null}
2026-10-14T10:36:31.467Z locks: release: app_backup_99de5b4d-6ad8-4f37-b08f-f6e110be4b2b
2026-10-14T10:36:31.468Z tasks: setCompleted - 24827: {"result":null,"error":{"message":"SSH connection error: ssh exited with code 255 signal null","reason":"External Error","stack":"BoxError: SSH connection error: ssh exited with code 255 signal null\\n    at copyInternal (file:///home/yellowtent/box/src/storage/filesystem.js:221:49)\\n    at async Object.copyDir (file:///home/yellowtent/box/src/storage/filesystem.js:245:12)\\n    at async Object.copy (file:///home/yellowtent/box/src/backupformat/rsync.js:148:5)"},"percent":100}
2026-10-14T10:36:31.468Z tasks: updating task 24827 with: {"completed":true,"result":null,"error":{"message":"SSH connection error: ssh exited with code 255 signal null","reason":"External Error","stack":"BoxError: SSH connection error: ssh exited with code 255 signal null\\n    at copyInternal (file:///home/yellowtent/box/src/storage/filesystem.js:221:49)\\n    at async Object.copyDir (file:///home/yellowtent/box/src/storage/filesystem.js:245:12)\\n    at async Object.copy (file:///home/yellowtent/box/src/backupformat/rsync.js:148:5)"},"percent":100}
2026-10-14T10:36:31.470Z taskworker: Task took 131.017 seconds
2026-10-14T10:36:31.470Z BoxError: SSH connection error: ssh exited with code 255 signal null\n    at copyInternal (file:///home/yellowtent/box/src/storage/filesystem.js:221:49)\n    at async Object.copyDir (file:///home/yellowtent/box/src/storage/filesystem.js:245:12)\n    at async Object.copy (file:///home/yellowtent/box/src/backupformat/rsync.js:148:5)
2026-10-14T10:36:31.470Z Exiting with code 0

[nebulon]

So the problem of flaky DNS hasn't changed then. Do you have any custom nameservers set in /etc/systemd/resolved.conf or /etc/systemd/resolved.conf.d/* ? are those two cloudron servers with the same VPS provider if the nameservers get set by the provider?

[avatar1024]

@nebulon said:

are those two cloudron servers with the same VPS provider if the nameservers get set by the provider?

They are quite different actually. One seem to have a specific config, while the other doesn't. They are both netcup RS servers configured the same way so I'm not sure why they are different. vi /etc/systemd/resolved.conf gives:

Broken instance:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=46.38.225.230 46.38.252.230 2a03:4000:8000::fce6 2a03:4000:0:1::e1e6
#FallbackDNS=
#Domains=
LLMNR=no
MulticastDNS=no
DNSSEC=allow-downgrade
Cache=yes
DNSStubListener=yes

Working instance:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file (or a copy of it placed in
# /etc/ if the original file is shipped in /usr/), or by creating "drop-ins" in
# the /etc/systemd/resolved.conf.d/ directory. The latter is generally
# recommended. Defaults can be restored by simply deleting the main
# configuration file and all drop-ins located in /etc/.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
#StaleRetentionSec=0           

[avatar1024]

Also without doing anything, now re running "backup now" completed just fine . Anyway, should I just remove the specific DNS config in the broken instance?

By the way, both instances have ipv6 disabled, and normally should be exactly the same network wise.

[james]

Hello @avatar1024

@avatar1024 said:

Also without doing anything, now re running "backup now" completed just fine

This does point to either a temporary DNS issue or the Hetzner fail2ban expired.

With:

sshfs -p 23 u273486@u273486.your-storagebox.de:/home $WHERE_YOU_WANT_IT_MOUNTED

You can always test a mount from either your local computer or any cloudron server.

Also, keep in mind, Hetzner Storage Boxes have a concurrent connections limit of 10.
So if you re-use the same Hetzner Storage Box for multiple Cloudron servers or even other things, this limit can be hit.
Each sshfs mount from a Cloudron Backup Site is 1x connection that stays active.

[avatar1024]

@james said:

You can always test a mount from either your local computer

Local sshfs mount work just fine.

@james said:

Also, keep in mind, Hetzner Storage Boxes have a concurrent connections limit of 10.

There are only 3 concurrent connections on the box, two Cloudron instances (one mount in each) and one local.

But the specific DNS config on the broken instance might be the source of the issue?

[Resolve]
DNS=46.38.225.230 46.38.252.230 2a03:4000:8000::fce6 2a03:4000:0:1::e1e6
#FallbackDNS=
#Domains=
LLMNR=no
MulticastDNS=no
DNSSEC=allow-downgrade
Cache=yes
DNSStubListener=yes

[nebulon]

I am not sure if resolvd correctly dismisses the ipv6 DNS resolver while you have ipv6 apparently disabled as you mentioned that. So maybe try to copy that DNS=46.38.225.230 46.38.252.230 2a03:4000:8000::fce6 2a03:4000:0:1::e1e6 line and comment it out then add a new one with just ipv4 DNS=46.38.225.230

[avatar1024]

For now I've commented everything line in the working instance and rebooted the server. If there is an issue I'll re-add the ipv4 DNS. Many thanks to you two for looking into this and helping. I'll report tomorrow to let you if it's resolved.

[avatar1024]

All good. A few backups have happened since the change and it all went smoothly. What has created the issue for the past few days is a mystery (since instance has been running like this for years) but thanks to you it's all back on track.