Backups keep failing

[joseph]

@OrezBodajoy 8.2.4 has nothing backup related, so I think that won't resolve this issue. If you can use something other than B2, it will help atleast to isolate if this is a backblaze issue.

There's been many reports on this forum already about B2 having network issues. Maybe they throttle traffic after some upload sizes and maybe Cloudron code has to be smarter somehow.

[OrezBodajoy]

I am trying now with AWS.
I didn't get an epipe error but the "Task 3061 timed out" after 1h exactly.
My photo album is quite big and I don't expect it to be uploaded within an hour. Is there a way to increase this timeout limit for the backup task? I don't see anything about that in the advanced section of the backup configuration.

Thanks

[nebulon]

Can you share the logs for that also? There shouldn't be any 1h cutoff timeout set by Cloudron itself, so likely this is some other root cause.

[OrezBodajoy]

Quick update, sorry for the delay.
After migrating from backblaze to aws s3 it seems I no longer have the EPIPE errors.

Here is what I have on the latest failed backup logs:

Mar 04 01:53:23 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:29 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/2f/1d/2f1d57dd-f397-4715-a214-96838b38d205.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:37 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/30/f8/30f835f2-08c2-4366-9915-fd4a92064959.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:43 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70
[no timestamp]  new Error('Socket timed out without establishing a connection'),
[no timestamp]  ^
[no timestamp] 
[no timestamp]  Error [TimeoutError]: Socket timed out without establishing a connection
[no timestamp]  at Timeout.connectTimeout [as _onTimeout] (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70:15)
[no timestamp]  at listOnTimeout (node:internal/timers:581:17)
[no timestamp]  at process.processTimers (node:internal/timers:519:7) {
[no timestamp]  code: 'TimeoutError',
[no timestamp]  time: 2025-03-04T00:53:48.827Z,
[no timestamp]  region: 'eu-central-1',
[no timestamp]  hostname: 'impressive-dangly-smut.s3.eu-central-1.amazonaws.com',
[no timestamp]  retryable: true,
[no timestamp]  statusCode: 200
[no timestamp]  }
[no timestamp] 
[no timestamp]  Node.js v20.18.0
Mar 04 01:54:07 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask fullBackup: app photos.mysuper.website backup finished. Took 4302.722 seconds
Mar 04 01:54:07 box:locks write: current locks: {"backup_task":null}
Mar 04 01:54:07 box:locks release: app_14dd2bbe-1d61-4f3d-91d3-4af16e366076
Mar 04 01:54:07 box:taskworker Task took 6845.852 seconds
Mar 04 01:54:07 box:tasks setCompleted - 3297: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 box:tasks update 3297: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 BoxError: Backuptask crashed
Mar 04 01:54:07 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)
Mar 04 01:54:07 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Mar 04 01:54:07 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)
Mar 04 01:54:07 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)

[mdc773]

use idrive e2 my friend and my advise if your server is working proper right now. remove the backup config completely and setup a new bucket and then try that seems to fix my issue all the time.

[nebulon]

@OrezBodajoy so instead of an EPIPE error you are now seeing upload stalling (repeated @0MBps) until you get the timeout?

[OrezBodajoy]

@mdc773 you mean https://www.idrive.com/s3-storage-e2/ ? I will give it a try, thanks for the suggestion.

@nebulon yes. I am using a vpn, maybe it's linked to that.

[OrezBodajoy]

@nebulon is there a way to update the app without doing the backup first?
I am now running a 6+ month old version without backups and this is really getting annoying now.

Note: switching to AWS S3 and changing the backup config has improved the situation but I still get an error during backup after a few hours.

Here is my log for reference. I only have a "Task 4059 crashed with code 1", I don't know what caused it.
https://upload.disroot.org/r/B82DZPjw#3JC3Iw05uQQVfACMKuVNPBtDjYha2BfXiKaSC82w3+E=

Is there no way to know what's causing the issue?

[scooke]

Is there no other way to share the logs? I don't feel like downloading an unknown file from a forum.

[Dont-Worry]

@scooke You can use SquareX extension to dl and open unknown files safely.

[scooke]

Nah, no thanks. good luck.

[nebulon]

@OrezBodajoy there don't seem to be any info on a backup task in those logs. Have you tried to create a manual backup from the app and then download the logs?

[jdaviescoates]

@OrezBodajoy said in Backups keep failing:

@nebulon is there a way to update the app without doing the backup first?

Yes, just check the Skip backup checkbox

[OrezBodajoy]

Thanks @jdaviescoates ! I will try that!

@nebulon I just checked the system logs and there are a bunch of out of memory error logs:
Out of memory: Killed process XXXXX (ruby2.3 - and sometimea node) total-vm: 1704896kB, anon-rss: 891324kB, file-rss: 0kB, shmem-rss: 0kB, UID: XXXX pgtables: 2072kB, ooom_score_adj: 0

I will check to see if there are still some backup memory settings I need to increase.

[nebulon]

given that ruby is mentioned there, it maybe that the system itself runs out of memory causing things to become unstable. At least the backup task would run via nodejs not ruby. Immich also isn't using ruby so this comes from some other app then.