Cloudron Forum

Thank you! I'm working through the platform updates now. I didn't realize it was so out of date, as I had automatic updates turned on. Discovered that the same issue causing my backups to fail was also preventing the platform updates from completing. Manually stopping the Snipe-IT app allowed the updates to complete successfully, so I'm assuming this will be resolved, but will follow up here if not.

Really appreciate the assistance.

Right, yes, that would make sense!

It was happening across multiple servers at the same time yesterday. Looks to be resolved now.

The issue is now resolved. The problem was with the GitLab database, where we found 160 GB of logs. The database dump was failing because there wasn’t enough free disk space. We have purged the database, and backups are now successful.

Thank you for your support.

Update: cloudron-support --troubleshoot returns:

[WARN] netplan configuration is empty. this might be OK depending on your networking setup [FAIL] DNS is not resolving

Further, /etc/netplan/ is empty. How do I generate the correct .yaml file to include there?

Final Update: I was able to crate a netplan .yaml file by following these instructions.

@nebulon Hetzner storagebox (Finland) and sshfs, rsync and encrypted filenames. 😄 The host fs (cloudron) is not a ssd (netcup).

Restic: https://restic.net
Supported repos out of the box: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
Other repos when combining restic with rclone: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone

A starting point: https://docs.cloudron.io/guides/community/restic-rclone/

I'm thinking about the possibility that the length of the encrypted filenames might be too much for Wasabi... I'll dig deeper...

There is currently no generic way to do this from Cloudron side. Depending on your backup storage you are using, you probably can download directly from there.

It looks like the problem is indeed gone with 8.1
Thank you!

This was resolved via support. For some reason unbound was not running. Restarting that made the backup storage work again (we may have a wrong configuration, where unbound is a systemd dependency for the storage mount, which seems like not needed anymore)

You should be able to download the whole task logs from the logviewer, which should give you more context which app was currently backed up.

@Usman said in Cloudron Backups to BackBlaze And RC Backup:

Right now, a 2 GB app backup is causing 4GB storage. (one 2 gb file in snapshot and one 2gb file in Date directory.

Yes, this is the case. But also, the system is optimized for having many backups. If you have 30 full backups, this is just 62GB. The 2GB is negligible in this context.

There is no "backup" options in RocketChat itself. The only option I have configured is Backups in cloudron. So this is done by Cloudron Backup procedure.

I don't see this in our Rocket.chat backup atleast. There is no second mongodb dump.

Top level:
c0d501f9-d319-41af-bb24-edca5ad89810-image.png

Inside data:

d25f6c57-a23e-4c0d-b667-589b68a1d9ab-image.png

@cocam123 whoops, you are right! I forgot there is an option in the UI for this 🙂

The exact checkbox text is Remote file system supports chown

Awesome, I couldn't get NFS to mount, so just set the tarball option instead, thanks.

@daixiwen ah ok. I would ignore the errors. It's normal for upstream services to go down once in a while...

I just checked https://status.backblaze.com/ and unfortunately no history of uptime here. But it says "Backblaze performs regular maintenance every Thursday from 11:30 am to 1:30 p.m. Pacific Time with the aim to consistently improve our systems and services. "

Just today I got a notification about an erroneous backup of the server where I have Nextcloud installed (150GB size) and I backup once a week.. I should add that the previous backups were successful. The provider is Backblaze B2.

Oct 06 01:02:32 box:storage/s3 Upload progress: {"loaded":1073741824,"part":2,"key":"102/snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7.tar.gz.enc"} Jan 01 01:24:00 node:events:496 [no timestamp] throw er; // Unhandled 'error' event [no timestamp] ^ [no timestamp] [no timestamp] write EPIPE [no timestamp] at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) [no timestamp] 'error' event on TLSSocket instance at: [no timestamp] at emitErrorNT (node:internal/streams/destroy:169:8) [no timestamp] at emitErrorCloseNT (node:internal/streams/destroy:128:3) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { [no timestamp] errno: -32, [no timestamp] code: 'EPIPE', [no timestamp] syscall: 'write' [no timestamp] } [no timestamp] [no timestamp] v20.12.2 Oct 06 01:02:32 box:shell backup-snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7 tgz {"localRoot":"/home/yellowtent/appsdata/50d07945-bf1b-4345-8e8c-c146e86c12a7","layout":[]} errored BoxError: backup-snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7 exited with code 1 signal null [no timestamp] at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19) [no timestamp] at ChildProcess.emit (node:events:518:28) [no timestamp] at ChildProcess.emit (node:domain:488:12) [no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:294:12) { [no timestamp] reason: 'Shell Error', [no timestamp] details: {}, [no timestamp] code: 1, [no timestamp] signal: null [no timestamp] } Oct 06 01:02:32 box:taskworker Task took 152.05 seconds Oct 06 01:02:32 box:tasks setCompleted - 2443: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Oct 06 01:02:32 box:tasks update 2443: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} [no timestamp] Backuptask crashed [no timestamp] at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [no timestamp] at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5) [no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5) [no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)

On other servers the backups are successful.

Yeah, I guessed the same after reading the debianforum wiki 🙂

Results after running it a couple of times:

sudo hdparm -t /dev/sdb1 /dev/sdb1: Timing buffered disk reads: 192 MB in 3.00 seconds = 63.92 MB/sec Timing buffered disk reads: 200 MB in 3.00 seconds = 66.58 MB/sec Timing buffered disk reads: 224 MB in 3.01 seconds = 74.44 MB/sec Timing buffered disk reads: 236 MB in 3.01 seconds = 78.31 MB/sec Timing buffered disk reads: 248 MB in 3.01 seconds = 82.33 MB/sec Timing buffered disk reads: 234 MB in 3.01 seconds = 77.69 MB/sec

Switching to hetzner feels good enough for now 😂. Full backup takes 30min for 21 apps. I've cancelled contabo storage service.
Hope this stays so snappy for the long run