Cloudron Forum

I'm thinking about the possibility that the length of the encrypted filenames might be too much for Wasabi... I'll dig deeper...

It looks like the problem is indeed gone with 8.1
Thank you!

This was resolved via support. For some reason unbound was not running. Restarting that made the backup storage work again (we may have a wrong configuration, where unbound is a systemd dependency for the storage mount, which seems like not needed anymore)

You should be able to download the whole task logs from the logviewer, which should give you more context which app was currently backed up.

@Usman said in Cloudron Backups to BackBlaze And RC Backup:

Right now, a 2 GB app backup is causing 4GB storage. (one 2 gb file in snapshot and one 2gb file in Date directory.

Yes, this is the case. But also, the system is optimized for having many backups. If you have 30 full backups, this is just 62GB. The 2GB is negligible in this context.

There is no "backup" options in RocketChat itself. The only option I have configured is Backups in cloudron. So this is done by Cloudron Backup procedure.

I don't see this in our Rocket.chat backup atleast. There is no second mongodb dump.

Top level:
c0d501f9-d319-41af-bb24-edca5ad89810-image.png

Inside data:

d25f6c57-a23e-4c0d-b667-589b68a1d9ab-image.png

@cocam123 whoops, you are right! I forgot there is an option in the UI for this 🙂

The exact checkbox text is Remote file system supports chown

Awesome, I couldn't get NFS to mount, so just set the tarball option instead, thanks.

@daixiwen ah ok. I would ignore the errors. It's normal for upstream services to go down once in a while...

I just checked https://status.backblaze.com/ and unfortunately no history of uptime here. But it says "Backblaze performs regular maintenance every Thursday from 11:30 am to 1:30 p.m. Pacific Time with the aim to consistently improve our systems and services. "

Just today I got a notification about an erroneous backup of the server where I have Nextcloud installed (150GB size) and I backup once a week.. I should add that the previous backups were successful. The provider is Backblaze B2.

Oct 06 01:02:32 box:storage/s3 Upload progress: {"loaded":1073741824,"part":2,"key":"102/snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7.tar.gz.enc"} Jan 01 01:24:00 node:events:496 [no timestamp] throw er; // Unhandled 'error' event [no timestamp] ^ [no timestamp] [no timestamp] write EPIPE [no timestamp] at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) [no timestamp] 'error' event on TLSSocket instance at: [no timestamp] at emitErrorNT (node:internal/streams/destroy:169:8) [no timestamp] at emitErrorCloseNT (node:internal/streams/destroy:128:3) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { [no timestamp] errno: -32, [no timestamp] code: 'EPIPE', [no timestamp] syscall: 'write' [no timestamp] } [no timestamp] [no timestamp] v20.12.2 Oct 06 01:02:32 box:shell backup-snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7 tgz {"localRoot":"/home/yellowtent/appsdata/50d07945-bf1b-4345-8e8c-c146e86c12a7","layout":[]} errored BoxError: backup-snapshot/app_50d07945-bf1b-4345-8e8c-c146e86c12a7 exited with code 1 signal null [no timestamp] at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19) [no timestamp] at ChildProcess.emit (node:events:518:28) [no timestamp] at ChildProcess.emit (node:domain:488:12) [no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:294:12) { [no timestamp] reason: 'Shell Error', [no timestamp] details: {}, [no timestamp] code: 1, [no timestamp] signal: null [no timestamp] } Oct 06 01:02:32 box:taskworker Task took 152.05 seconds Oct 06 01:02:32 box:tasks setCompleted - 2443: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Oct 06 01:02:32 box:tasks update 2443: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} [no timestamp] Backuptask crashed [no timestamp] at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [no timestamp] at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5) [no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5) [no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)

On other servers the backups are successful.

Yeah, I guessed the same after reading the debianforum wiki 🙂

Results after running it a couple of times:

sudo hdparm -t /dev/sdb1 /dev/sdb1: Timing buffered disk reads: 192 MB in 3.00 seconds = 63.92 MB/sec Timing buffered disk reads: 200 MB in 3.00 seconds = 66.58 MB/sec Timing buffered disk reads: 224 MB in 3.01 seconds = 74.44 MB/sec Timing buffered disk reads: 236 MB in 3.01 seconds = 78.31 MB/sec Timing buffered disk reads: 248 MB in 3.01 seconds = 82.33 MB/sec Timing buffered disk reads: 234 MB in 3.01 seconds = 77.69 MB/sec

Switching to hetzner feels good enough for now 😂. Full backup takes 30min for 21 apps. I've cancelled contabo storage service.
Hope this stays so snappy for the long run

@bazinga sorry, did this get solved? If not can you please ping me again on support@ ?

@girish Ok, great! Thank you very much for the information. I really appreciate your time. Thanks again!

Wonderful. Thank you @Joseph, that change to the endpoint URL did the trick. The configuration saved successfully.

@Wenhardt try Scaleway - not perfect but works generally

@Beardmancer said in Failed backups and email config issue since 8.0.3 upgrade:

I decided to go ahead and try adding this to /etc/systemd/resolved.conf since it's easy enough to comment out.

DNS=8.8.8.8 1.1.1.1

After doing that and restarting the service with systemctl restart systemd-resolved, things seem to be working correctly. The file /etc/resolv.conf lists nameservers now and a manually-initiated backup just completed successfully. Thanks for the hlep @nebulon !

In the future, is there anything I should look out for to avoid this? I don't know what caused it. Maybe it was the 8.0.3 upgrade, or maybe it was an Ubuntu update? I think I rebooted Cloudron for the latter recently. If there's a configuration I might have setup wrong, I'd love to correct it.

It did turn out to be a DNC issue which Joseph at Clourdon support fixed and then created a backup.

@girish One thing to note, all stopped apps where not imported through the restore method even tho they had backups retained with them. We are still trying to find and recover those apps atm, but they must be buried in a random 2024-04-08-030000-684 folder. Never the less.

Is it possible to recover data from the snapshot folder? what is that for if backups pull from the other folders? I think our stopped apps are in there but Im not sure how to recover them through the front end that way.

One thing that would have made this process so much more seamless is just including that restore config file in the snapshot folder, and the corresponding configs for each app backup, would have saved us hours of finding the correct backup id manually for each method, since Im assuming they are slightly different?

Thanks for the report, this is fixed for the next release.