Backups keep failing

[OrezBodajoy]

My Immich backups have been failing for some time now.
I am now using the rsync backup instead of tar and I have increased the memory of the machine.

Here are the logs: https://upload.disroot.org/r/ndomGdSa#bbIlMwTZpfswm53IwhZ66We9lvxSqjGdHYVMuWzts+I=

Thanks!

[jdaviescoates]

Too many empty directories? If so search the forum, others have hit the same thing.

[nebulon]

Most likely you hit the same as https://forum.cloudron.io/topic/13047/since-update-to-v8-2-1-backups-fail-with-too-many-empty-directories/16?_=1737623616788

[OrezBodajoy]

Please have a quick look at the logs, I see no "too many" directories or executables in the logs.
I have ran the find -type d -empty and found 4000 results though. Could that be the cause still?

[nebulon]

Oh sorry I was mislead by the comment of @jdaviescoates . So looking at the logs it seems the upload to S3 fails with some EPIPE. This indicates that the remote S3 storage terminates the upload connection prematurely or maybe also your system ran out of memory and that killed the pipeline. We have had various issue with backblaze in the past. Can you maybe tweak the backup task memory limit as well as the part-size to see if that makes a difference?

[OrezBodajoy]

I am indeed using backblaze.
I have increased the memory limit from the default 1G to 5G.
I have also increased the part-size from the default 10MB to 128MB (is that what you mean by tweak? Or should I reduce it instead?).
I will let you know how it goes. As it's a big library of photos (100+GB?), it's taking a while.

[OrezBodajoy]

I am getting these errors now:

Jan 27 20:47:44 descriptor
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Error while closing socket [Errno 9] Bad file
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Shutting down
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Shutting down due to inactivity.
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Waiting for application shutdown.
Jan 27 20:47:48 [01/27/25 19:47:48] ERROR Worker (pid:772) was sent SIGINT!
Jan 27 20:47:48 [01/27/25 19:47:48] INFO Application shutdown complete.
Jan 27 20:47:48 [01/27/25 19:47:48] INFO Booting worker with pid: 907
Jan 27 20:47:48 [01/27/25 19:47:48] INFO Finished server process [772]
Jan 27 20:47:57 of inactivity.
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Application startup complete.
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Created in-memory cache with unloading after 300s
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Initialized request thread pool with 4 threads.
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Started server process [907]
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Waiting for application startup.
Jan 28 01:00:00 2025-01-28T00:00:00Z
Jan 28 01:00:00 2025-01-28T00:00:00Z
Jan 28 01:00:00 2025-01-28T00:00:00Z
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/66/b4/66b49043-0cc2-48a9-b163-c138a82362b8.HEIC
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/e4/e6/e4e69bc7-abab-4fb2-9b16-d01f28d3819d.jpg
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: VipsJpeg: Premature end of input file
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object:
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object:
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object:
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/66/b4/66b49043-0cc2-48a9-b163-c138a82362b8.HEIC
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/e4/e6/e4e69bc7-abab-4fb2-9b16-d01f28d3819d.jpg
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: VipsJpeg: Premature end of input file
Jan 28 01:00:00 "id": "60e8f369-c07d-4e55-b54a-904f455f8366"
Jan 28 01:00:00 "id": "78bd4060-73b1-4687-ad99-719856cfed7f"

and

Jan 28 01:00:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Jan 28 01:00:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 28 01:00:00 {
Jan 28 01:00:00 {
Jan 28 01:00:00 {
Jan 28 01:00:00 }
Jan 28 01:00:00 }
Jan 28 01:00:00 }
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:Backup failed with code 1] Database Backup Failure
Jan 28 03:00:00 R [Microservices:BackupService] Backup failed with code 1
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:BackupService] Gzip exited with code 0 but pgdump exited with 1
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:BackupService] pg_dumpall: error: query failed: ERROR: permission denied for table pg_authid
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM LOG [Microservices:BackupService] Database Backup Starting. Database Version: 14
Jan 28 03:00:00
Jan 28 03:00:00 pg_dumpall: error: query was: SELECT rolname FROM pg_authid WHERE rolname !~ '^pg_' ORDER BY 1
Jan 28 16:22:36 [Nest] 47 - 01/28/2025, 3:22:36 PM LOG [Api:EventRepository] Websocket Connect: LBAslGNTdfqXxBm4AAAL
Jan 28 16:23:39 [Nest] 47 - 01/28/2025, 3:23:39 PM LOG [Api:EventRepository] Websocket Disconnect: LBAslGNTdfqXxBm4AAAL
Jan 28 18:31:37 ['CPUExecutionProvider'], in descending order of
Jan 28 18:31:37 preference
Jan 28 18:31:37 [01/28/25 17:31:37] INFO Loading visual model 'ViT-B-32__openai' to memory
Jan 28 18:31:37 [01/28/25 17:31:37] INFO Setting execution providers to
Jan 28 18:31:39 ['CPUExecutionProvider'], in descending order of
Jan 28 18:31:39 preference
Jan 28 18:31:39 [01/28/25 17:31:39] INFO Loading detection model 'buffalo_l' to memory
Jan 28 18:31:39 [01/28/25 17:31:39] INFO Setting execution providers to
Jan 28 18:31:40 ['CPUExecutionProvider'], in descending order of
Jan 28 18:31:40 preference
Jan 28 18:31:40 [01/28/25 17:31:40] INFO Loading recognition model 'buffalo_l' to memory
Jan 28 18:31:40 [01/28/25 17:31:40] INFO Setting execution providers to
Jan 28 18:36:39 descriptor
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Error while closing socket [Errno 9] Bad file
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Shutting down
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Shutting down due to inactivity.
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Waiting for application shutdown.
Jan 28 18:36:40 [01/28/25 17:36:40] ERROR Worker (pid:907) was sent SIGINT!
Jan 28 18:36:40 [01/28/25 17:36:40] INFO Application shutdown complete.
Jan 28 18:36:40 [01/28/25 17:36:40] INFO Booting worker with pid: 946
Jan 28 18:36:40 [01/28/25 17:36:40] INFO Finished server process [907]
Jan 28 18:36:47 of inactivity.

I am not sure how to handle that...

[nebulon]

Those are immich errors, not related to any Cloudron backup. Looks like immich added its own database backup job by now which will not work on Cloudron and we have to disable this in the package. But either way you can ignore those. What is the status on your Cloudron backup of the app?

[OrezBodajoy]

No new immich backup. Right now the app is marked as Error and on the backup tab the 'Create backup' button is greyed-out.
In the notifications I also get a backup task crashed notification.
Here are the logs for both immich and that general backup task crashed.

https://upload.disroot.org/r/nuvYocg7#xGyeIOHzUekVeQhAC3Aup1vR7/SZqPepg3/aC10aSW0=
https://upload.disroot.org/r/9MFpVBdn#B9BzBZ8buWuNh0SPq6j605GBQcCqIedo0G/jI2J956g=

Please have a look at the logs.
Thanks!

[nebulon]

For a start, the next immich package version has the built-in database backups disabled, as they are of no use on Cloudron and also try to write into read-only storage.

To get your app up and running again, please hit the "Retry task" in the repair tab of that apps configure section.

[nebulon]

Looking further at the logs, it seems at least that last run still hit the EPIPE error, where the uploading was cancelled while trying to upload a .mp4 file. This could either be a memory issue on your server or also the remote storage terminated the connection prematurely.

[OrezBodajoy]

I have changed a bit the backup parameters in the UI:

• memory limit to 9 GiB
• upload part size 64 MiB
• concurrency items 10

I now have this error:

Feb 04 13:48:29 box:storage/s3 Upload progress: {"loaded":62003204,"total":62003204,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ"}
Feb 04 13:48:32 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4","reason":"new","position":425}
Feb 04 13:48:32 box:storage/s3 Upload finished. {"ETag":"\"96b07711e9e9cd6313ae1d3bf2eab89d\"","VersionId":"4_z30c7662aa70ee25089da0419_f112b8445e8479200_d20250204_m124750_c003_v0312024_t0049_u01738673270973","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc%2B63z%2BGW%2BIIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:32 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 position 425 try 1
Feb 04 13:48:32 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 (photos.mysuper.website)"}
Feb 04 13:48:34 box:storage/s3 Upload progress: {"loaded":12934596,"total":12934596,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw"}
Feb 04 13:48:37 box:storage/s3 Upload progress: {"loaded":24622868,"total":24622868,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA"}
Feb 04 13:48:38 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4","reason":"new","position":426}
Feb 04 13:48:38 box:storage/s3 Upload finished. {"ETag":"\"a63eb3273bd8c6af9c8be8cdb8b3411c\"","VersionId":"4_z30c7662aa70ee25089da0419_f111b63a9bf79370d_d20250204_m124018_c003_v0312024_t0006_u01738672818986","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-%2Bbm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF%2BWvX4eM5wpaPa04Cp9mm0g9jyU%2BWHPQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:38 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 position 426 try 1
Feb 04 13:48:38 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 (photos.mysuper.website)"}
Feb 04 13:48:43 box:storage/s3 Upload finished. {"ETag":"\"e8f727fbe3017a736863acd3e005920f\"","VersionId":"4_z30c7662aa70ee25089da0419_f102a9d6210f3b68d_d20250204_m124644_c003_v0312028_t0032_u01738673204217","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:43 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4 position 427 try 1
Feb 04 13:48:43 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4","reason":"new","position":427}
Feb 04 13:48:45 box:storage/s3 Upload finished. {"ETag":"\"fef4d9e41a7b0487ef85cde77de337c7\"","VersionId":"4_z30c7662aa70ee25089da0419_f105b54dc655def0e_d20250204_m124514_c003_v0312025_t0014_u01738673114140","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE%2BWncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3%2BwvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:45 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4","reason":"new","position":428}
Feb 04 13:48:45 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 position 428 try 1
Feb 04 13:48:45 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 (photos.mysuper.website)"}
Feb 04 13:48:48 box:storage/s3 Upload progress: {"loaded":83903444,"total":83903444,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg"}
Feb 04 13:48:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:48:56 zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83%2BN073h%2BYeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","Bucket":"cloudron-backup-january-february","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","ETag":"\"de497871d785fd1c59918d801ccfa714-2\""}
Feb 04 13:48:56 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4 position 429 try 1
Feb 04 13:48:56 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4","reason":"new","position":429}
Feb 04 13:49:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:11 box:storage/s3 Upload progress: {"loaded":51287460,"total":118396324,"part":2,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/JXzM88Uqx7ZsSnmpRN00J7pjTkogYPeJj9sRGLylY-Y/lCpIzgRkjCJJlzwTqL5scjzLiKZZcSHPgHnseJC+9VU/xzreWsN82y+-8efk7r1FwykGI9qCIdoC1+WVWtIjKuolkKXYiFi4nMAPlEa4HbLqzGkhp4PA7nDfRQZSvpQgNA"}
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.044 * 10 changes in 300 seconds. Saving...
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.046 * Background saving started by pid 248
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.146 * Background saving terminated with success
Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.129 * DB saved on disk
Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.130 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 13:49:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:12 box:storage/s3 Upload progress: {"loaded":79710836,"total":146819700,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/LQs0pg3EACKpxLfIoyhm3-pwf2CjUC0Qmr45cpZ0XtY/zRON9tC4HPaxKx9vNEkZkXGSyTtIBEVsHwJTKqt6EAI/zSGMZ1+pEr5om7CkzelvYADDA4IPqQHHBv18fQYz-AzjLcjdhD8N9ce0c7KaGaYbYwyPXwq5IcGX07SjbJQWzw"}
Feb 04 13:50:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:36 box:storage/s3 Upload progress: {"loaded":21172484,"total":21172484,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q"}
Feb 04 13:50:39 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4 position 430 try 1
Feb 04 13:50:39 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4","reason":"new","position":430}
Feb 04 13:50:39 box:storage/s3 Upload finished. {"ETag":"\"e8db66fe56ced58e8bccbf20014a6b92\"","VersionId":"4_z30c7662aa70ee25089da0419_f1064034b6c53492b_d20250204_m124857_c003_v0312028_t0047_u01738673337237","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0%2BgncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy%2BIEnSGqo17KBKZjlQcIp7Q","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Bucket":"cloudron-backup-january-february"}
Feb 04 13:50:45 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Feb 04 13:50:46 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Feb 04 13:50:46 box:apptask run: app error for state pending_backup: BoxError: Backuptask crashed at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5) at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12) at async backup (/home/yellowtent/box/src/apptask.js:400:22) { reason: 'Internal Error', details: {} }
Feb 04 13:50:46 box:taskworker Task took 12148.089 seconds
Feb 04 13:50:46 box:tasks setCompleted - 3016: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Feb 04 13:50:46 box:tasks update 3016: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.044 * 10 changes in 300 seconds. Saving...
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.046 * Background saving started by pid 249
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.147 * Background saving terminated with success
Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.137 * DB saved on disk
Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.139 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.020 * 10 changes in 300 seconds. Saving...
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.022 * Background saving started by pid 250
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.223 * Background saving terminated with success
Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.124 * DB saved on disk
Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.125 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.094 * 10 changes in 300 seconds. Saving...
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.097 * Background saving started by pid 251
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.198 * Background saving terminated with success
Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.188 * DB saved on disk
Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.189 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.081 * 10 changes in 300 seconds. Saving...
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.082 * Background saving started by pid 252
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.285 * Background saving terminated with success
Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.184 * DB saved on disk
Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.185 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.059 * 10 changes in 300 seconds. Saving...
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.061 * Background saving started by pid 253
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.262 * Background saving terminated with success
Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.160 * DB saved on disk
Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.162 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.096 * 10 changes in 300 seconds. Saving...
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.098 * Background saving started by pid 254
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.198 * Background saving terminated with success
Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.175 * DB saved on disk
Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.177 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.048 * 10 changes in 300 seconds. Saving...
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.049 * Background saving started by pid 255
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.149 * Background saving terminated with success
Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.142 * DB saved on disk
Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.144 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.087 * 10 changes in 300 seconds. Saving...
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.089 * Background saving started by pid 256
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.190 * Background saving terminated with success
Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.163 * DB saved on disk
Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.165 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.052 * 10 changes in 300 seconds. Saving...
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.053 * Background saving started by pid 257
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.154 * Background saving terminated with success
Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.134 * DB saved on disk
Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.136 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:34:21 408: null
Feb 04 14:34:21 BoxError: Backuptask crashed
Feb 04 14:34:21 Node.js v20.18.0
Feb 04 14:34:21 }
Feb 04 14:34:21 }
Feb 04 14:34:21 }

Thanks!

[joseph]

It's the dreaded 408 error - request time out . But I also notice @0Mbps which I think was fixed in 8.2 .

@OrezBodajoy which version of Cloudron are you on?

[OrezBodajoy]

Platform versionv8.2.3 (Ubuntu 22.04.4 LTS)
I will try updating to 8.2.4

Would it work better if I use another backup method than backblaze?

[joseph]

@OrezBodajoy 8.2.4 has nothing backup related, so I think that won't resolve this issue. If you can use something other than B2, it will help atleast to isolate if this is a backblaze issue.

There's been many reports on this forum already about B2 having network issues. Maybe they throttle traffic after some upload sizes and maybe Cloudron code has to be smarter somehow.

[OrezBodajoy]

I am trying now with AWS.
I didn't get an epipe error but the "Task 3061 timed out" after 1h exactly.
My photo album is quite big and I don't expect it to be uploaded within an hour. Is there a way to increase this timeout limit for the backup task? I don't see anything about that in the advanced section of the backup configuration.

Thanks

[nebulon]

Can you share the logs for that also? There shouldn't be any 1h cutoff timeout set by Cloudron itself, so likely this is some other root cause.

[OrezBodajoy]

Quick update, sorry for the delay.
After migrating from backblaze to aws s3 it seems I no longer have the EPIPE errors.

Here is what I have on the latest failed backup logs:

Mar 04 01:53:23 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:29 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/2f/1d/2f1d57dd-f397-4715-a214-96838b38d205.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:37 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/30/f8/30f835f2-08c2-4366-9915-fd4a92064959.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:43 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70
[no timestamp]  new Error('Socket timed out without establishing a connection'),
[no timestamp]  ^
[no timestamp] 
[no timestamp]  Error [TimeoutError]: Socket timed out without establishing a connection
[no timestamp]  at Timeout.connectTimeout [as _onTimeout] (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70:15)
[no timestamp]  at listOnTimeout (node:internal/timers:581:17)
[no timestamp]  at process.processTimers (node:internal/timers:519:7) {
[no timestamp]  code: 'TimeoutError',
[no timestamp]  time: 2025-03-04T00:53:48.827Z,
[no timestamp]  region: 'eu-central-1',
[no timestamp]  hostname: 'impressive-dangly-smut.s3.eu-central-1.amazonaws.com',
[no timestamp]  retryable: true,
[no timestamp]  statusCode: 200
[no timestamp]  }
[no timestamp] 
[no timestamp]  Node.js v20.18.0
Mar 04 01:54:07 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask fullBackup: app photos.mysuper.website backup finished. Took 4302.722 seconds
Mar 04 01:54:07 box:locks write: current locks: {"backup_task":null}
Mar 04 01:54:07 box:locks release: app_14dd2bbe-1d61-4f3d-91d3-4af16e366076
Mar 04 01:54:07 box:taskworker Task took 6845.852 seconds
Mar 04 01:54:07 box:tasks setCompleted - 3297: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 box:tasks update 3297: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 BoxError: Backuptask crashed
Mar 04 01:54:07 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)
Mar 04 01:54:07 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Mar 04 01:54:07 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)
Mar 04 01:54:07 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)

[mdc773]

use idrive e2 my friend and my advise if your server is working proper right now. remove the backup config completely and setup a new bucket and then try that seems to fix my issue all the time.

[nebulon]

@OrezBodajoy so instead of an EPIPE error you are now seeing upload stalling (repeated @0MBps) until you get the timeout?