Since I added a storage volume to my infrastructure, I've been getting this backup error message very regularly

[Dont-Worry]

Hello,

I'm coming back to you because this problem is still unresolved. I haven't been able to find a solution to get the backups working again.

And I can now confirm that it is indeed all the backups that are not working. Every day the backup fails.

[Dont-Worry]

[no timestamp]  at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:110:19)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:291:12) {
[no timestamp]  at ChildProcess.emit (node:domain:489:12)
[no timestamp]  at ChildProcess.emit (node:events:513:28)
[no timestamp]  at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)
[no timestamp]  at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)
[no timestamp]  at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[no timestamp]  at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)
[no timestamp]  code: 50,
[no timestamp]  details: {},
[no timestamp]  reason: 'Shell Error',
[no timestamp]  signal: null
May 25 07:32:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 67499M@41MBps (emby.example.com)"}
May 25 07:32:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 67910M@41MBps (emby.example.com)"}
May 25 07:32:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 68326M@42MBps (emby.example.com)"}
May 25 07:32:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 68735M@41MBps (emby..com)"}
May 25 07:32:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 69152M@42MBps (emby.example.com)"}
May 25 07:32:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 69562M@41MBps (emby.example.com)"}
May 25 07:33:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 69970M@41MBps (emby.example.com)"}
May 25 07:33:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 70385M@42MBps (emby.example.com)"}
May 25 07:33:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 70795M@41MBps (emby.example.com)"}
May 25 07:33:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 71207M@41MBps (emby.example.com)"}
May 25 07:33:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 71626M@42MBps (emby.example.com)"}
May 25 07:33:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 72039M@41MBps (emby.example.com)"}
May 25 07:34:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 72453M@41MBps (emby.example.com)"}
May 25 07:34:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 72867M@41MBps (emby.example.com)"}
May 25 07:34:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 73281M@41MBps (emby.example.com)"}
May 25 07:34:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 73696M@42MBps (emby.example.com)"}
May 25 07:34:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 74095M@40MBps (emby.example.com)"}
May 25 07:34:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 74510M@41MBps (emby.example.com)"}
May 25 07:35:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 74917M@41MBps (emby.example.com)"}
May 25 07:35:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 75321M@40MBps (emby.example.com)"}
May 25 07:35:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 75728M@41MBps (emby.example.com)"}
May 25 07:35:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 76141M@41MBps (emby.example.com)"}
May 25 07:35:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 76552M@41MBps (emby.example.com)"}
May 25 07:35:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 76950M@40MBps (emby.example.com)"}
May 25 07:36:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 77361M@41MBps (emby.example.com)"}
May 25 07:36:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 77777M@42MBps (emby.example.com)"}
May 25 07:36:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 78189M@41MBps (emby.example.com)"}
May 25 07:36:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 78611M@42MBps (emby.example.com)"}
May 25 07:36:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 79020M@41MBps (emby.example.com)"}
May 25 07:36:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 79430M@41MBps (emby.example.com)"}
May 25 07:37:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 79840M@41MBps (emby.example.com)"}
May 25 07:37:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 80250M@41MBps (emby.example.com)"}
May 25 07:37:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 80660M@41MBps (emby.example.com)"}
May 25 07:37:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 81070M@41MBps (emby.example.com)"}
May 25 07:37:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 81471M@40MBps (emby.example.com)"}
May 25 07:37:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 81879M@41MBps (emby.example.com)"}
May 25 07:38:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 82293M@41MBps (emby.example.com)"}
May 25 07:38:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 82698M@40MBps (emby.example.com)"}
May 25 07:38:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 83112M@41MBps (emby.example.com)"}
May 25 07:38:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 83527M@41MBps (emby.example.com)"}
May 25 07:38:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 83940M@41MBps (emby.example.com)"}
May 25 07:38:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 84353M@41MBps (emby.example.com)"}
May 25 07:39:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 84763M@41MBps (emby.example.com)"}
May 25 07:39:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 85180M@42MBps (emby.example.com)"}
May 25 07:39:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 85585M@41MBps (emby.example.com)"}
May 25 07:39:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 86005M@42MBps (emby.example.com)"}
May 25 07:39:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 86416M@41MBps (emby.example.com)"}
May 25 07:39:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 86833M@42MBps (emby.example.com)"}
May 25 07:40:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 87242M@41MBps (emby.example.com)"}
May 25 07:40:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 87662M@42MBps (emby.example.com)"}
May 25 07:40:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 88056M@39MBps (emby.example.com)"}
May 25 07:40:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 88466M@41MBps (emby.example.com)"}
May 25 07:40:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 88876M@41MBps (emby.example.com)"}
May 25 07:40:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 89289M@41MBps (emby.example.com)"}
May 25 07:41:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 89692M@40MBps (emby.example.com)"}
May 25 07:41:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 90101M@41MBps (emby.example.com)"}
May 25 07:41:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 90518M@42MBps (emby.example.com)"}
May 25 07:41:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 90931M@41MBps (emby.example.com)"}
May 25 07:41:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 91341M@41MBps (emby.example.com)"}
May 25 07:41:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 91749M@41MBps (emby.example.com)"}
May 25 07:42:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 92143M@39MBps (emby.example.com)"}
May 25 07:42:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 92556M@41MBps (emby.example.com)"}
May 25 07:42:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 92968M@41MBps (emby.example.com)"}
May 25 07:42:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 93382M@41MBps (emby.example.com)"}
May 25 07:42:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 93792M@41MBps (emby.example.com)"}
May 25 07:42:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 94198M@41MBps (emby.example.com)"}
May 25 07:43:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 94617M@42MBps (emby.example.com)"}
May 25 07:43:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 95029M@41MBps (emby.example.com)"}
May 25 07:43:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 95434M@41MBps (emby.example.com)"}
May 25 07:43:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 95846M@41MBps (emby.example.com)"}
May 25 07:43:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 96256M@41MBps (emby.example.com)"}
May 25 07:43:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 96658M@40MBps (emby.example.com)"}
May 25 07:44:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 97074M@42MBps (emby.example.com)"}
May 25 07:44:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 97483M@41MBps (emby.example.com)"}
May 25 07:44:29 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 97892M@41MBps (emby.example.com)"}
May 25 07:44:39 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 98308M@42MBps (emby.example.com)"}
May 25 07:44:49 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 98711M@40MBps (emby.example.com)"}
May 25 07:44:59 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 99131M@42MBps (emby.example.com)"}
May 25 07:45:09 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 99530M@40MBps (emby.example.com)"}
May 25 07:45:19 box:tasks update 2168: {"percent":45.444444444444436,"message":"Uploading backup 99927M@40MBps (emby.example.com)"}
May 25 07:45:21 box:backuptask runBackupUpload: result - {"result":"Error uploading snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471.tar.gz.enc. Message: Part number must be an integer between 1 and 10000, inclusive HTTP Code: InvalidArgument"}
May 25 07:45:21 box:shell backup-snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471 tgz {"localRoot":"/home/yellowtent/appsdata/78974c13-af83-4056-b7ff-218dd7fcb471","layout":[{"localDir":"/mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Emby","remoteDir":"data"}]} errored BoxError: backup-snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471 exited with code 50 signal null
May 25 07:45:21 box:tasks setCompleted - 2168: {"result":null,"error":{"stack":"BoxError: Error uploading snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471.tar.gz.enc. Message: Part number must be an integer between 1 and 10000, inclusive HTTP Code: InvalidArgument\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471.tar.gz.enc. Message: Part number must be an integer between 1 and 10000, inclusive HTTP Code: InvalidArgument"}}
May 25 07:45:21 box:tasks update 2168: {"percent":100,"result":null,"error":{"stack":"BoxError: Error uploading snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471.tar.gz.enc. Message: Part number must be an integer between 1 and 10000, inclusive HTTP Code: InvalidArgument\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471.tar.gz.enc. Message: Part number must be an integer between 1 and 10000, inclusive HTTP Code: InvalidArgument"}}
May 25 07:45:21 box:taskworker Task took 13520.94 seconds
[no timestamp]  Error uploading snapshot/app_78974c13-af83-4056-b7ff-218dd7fcb471.tar.gz.enc. Message: Part number must be an integer between 1 and 10000, inclusive HTTP Code: InvalidArgument
[no timestamp]  }

[necrevistonnezr]

Please use the code tag for the log messages - they’re unreadable without.

[Dont-Worry]

@necrevistonnezr oh, my bad, i'll correct it rn. Thanks

[necrevistonnezr]

I understand correctly: You’re trying a 100+ GB backup via an encrypted tar.gz archive of an Emby instance (which already contains highly compressed movie files, probably, that cannot be compressed further)?

[Dont-Worry]

I have something like 200gb of Emby data (and it's growing every day) you're right about that. On top we have a lot of music and a lot of youtube videos. Some files are in MP3 but the majority are in .WAV and for videos we're on .MP4.

Do you have any idea what I should do to be able to restore the backups system?

[humptydumpty]

I’m also getting backup failed errors much more often lately. No volumes attached though. Backing up to backblaze. Latest fails happened two nights in a row on 5/23 and 5/24, worked fine on 5/25.

2024-05-23T05:06:29.143Z box:shell backup-snapshot/app_56523b4e-70f6-4323-98b9-559b87e74173: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_56523b4e-70f6-4323-98b9-559b87e74173 tgz {"localRoot":"/home/yellowtent/appsdata/56523b4e-70f6-4323-98b9-559b87e74173","layout":[]} errored BoxError: backup-snapshot/app_56523b4e-70f6-4323-98b9-559b87e74173 exited with code 1 signal null
2024-05-23T05:06:29.148Z box:tasks setCompleted - 12664: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n    at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
2024-05-23T05:06:29.148Z box:taskworker Task took 381.54 seconds
2024-05-23T05:06:29.149Z box:tasks update 12664: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n    at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
BoxError: Backuptask crashed

@girish please see last section of the logs above in case there is indeed a bug in the code. However, I believe my issue is network related and differs from yours @Dont-Worry

[girish]

@Dont-Worry It's failing pretty much at 100GB. It seems that the provider may not support > 100GB files. Which backup provider are you using (Is it OVH?) ?

[necrevistonnezr]

@Dont-Worry said in Since I added a storage volume to my infrastructure, I've been getting this backup error message very regularly:

I have something like 200gb of Emby data (and it's growing every day) you're right about that. On top we have a lot of music and a lot of youtube videos. Some files are in MP3 but the majority are in .WAV and for videos we're on .MP4.

Using the encrypted tar backend for this mass of - already compressed - data is asking for trouble, IMHO. As girish pointed out, you end up with a single file of huge proportions, from the docs:

"The tgz format stores all the backup information in a single tarball whereas the rsync format stores all backup information as files inside a directory."

I could be wrong but many B2C providers will probably have difficulties supporting a 200 GB + file?!

[Dont-Worry]

Thank you for your answer @necrevistonnezr
@girish To answer your question, I am using AWS for backups. I tried to use OVH when the problem started but idk why i cant connect OVH Object Storage with my cloudron.

@necrevistonnezr When you say "Using the encrypted tar backend for this mass of - already compressed - data is asking for trouble, IMHO.". I don't think 200gb is a huuuge mass of data (especially for AWS that is not a B2C focused provider). For our Company this is a little bit concerning if we cannot backup anything above 100gb.... Because we use cloudron for Entertainment (Emby), but also for business purposes (Files storages, apps ...) and we planned to use cloudron for long term purposes and forcasted more than 5 to 10 TB of data in a year. There is any way to Backup large amount of data w/cloudron in an automated way, as the system usually worked ?

[necrevistonnezr]

Use the rsync backend?

[Dont-Worry]

The problem has been solved. I have increased the size of the packages which are sent to the S3 and the whole was less than 10,000 packages so it was getting in. I read the documentation and really been over all my Cloudron panel many times, but I really didn't think about reviewing the advanced configurations of my backup, that is the thing I do not often change.

This solution works in the short term, but according to my calculations, once it reaches 10Tb, the maximum cut proposed by Cloudron, namely the cut by 1gb per package, will not be enough. Would it be possible to increase the maximum in the Cloudron application for people who use and consider Cloudron as a long-term mid-scale solution ?

This would not change anything if it was adjustable, but for people like us, who will have to deploy Cloudron on a powerful infrastructure, the double could be supported without any problem (2gb/Package).

[girish]

@Dont-Worry Thanks for the follow-up. Can you tell me which region of OVH Object Storage you are using?

[Dont-Worry]

@girish I am using Graveline for OVH Object Storage. First I choosed Roubaix, but it wasn't in the Cloudron list so I created a new bucket in Graveline. But it never worked. Now it is still working on the old AWS S3 bucket.
Didn't found any way to make the backup to an OVH bucket work.

[humptydumpty]

@girish Can you please expand the doc page to cover what the "advanced settings" in the backups section offers and how they can be best used/optimized for a few use-cases like (general-use, heavy media use, etc.). Thank you!

[girish]

@Dont-Worry thanks, I have added RBX to the region list.

[girish]

@humptydumpty I think the issue is that there are too many providers, each with their own region. Back in the day, we hardcoded this and the advanced setting was not even exposed to the user because we thought it's impossible for the user to guess correct values. We ended up exposing the values because we couldn't figure a way to guess values . Even now, I have no concrete values to suggest for each provider, they keep changing (since upstream providers are also deploying updates) and each network/app/cpu is different. It's a bit of trial and error . But I will discuss internally and add something to the docs to help users.

[Dont-Worry]

@girish Thank you very much !