Backups keep failing

Reply to Backups keep failing on Mon, 19 May 2025 17:26:12 GMT

nebulon — Mon, 19 May 2025 17:26:12 GMT

given that ruby is mentioned there, it maybe that the system itself runs out of memory causing things to become unstable. At least the backup task would run via nodejs not ruby. Immich also isn't using ruby so this comes from some other app then.

Reply to Backups keep failing on Sun, 18 May 2025 15:37:56 GMT

OrezBodajoy — Sun, 18 May 2025 15:37:56 GMT

Thanks @jdaviescoates ! I will try that!

@nebulon I just checked the system logs and there are a bunch of out of memory error logs:
Out of memory: Killed process XXXXX (ruby2.3 - and sometimea node) total-vm: 1704896kB, anon-rss: 891324kB, file-rss: 0kB, shmem-rss: 0kB, UID: XXXX pgtables: 2072kB, ooom_score_adj: 0

I will check to see if there are still some backup memory settings I need to increase.

Reply to Backups keep failing on Sat, 17 May 2025 15:00:29 GMT

jdaviescoates — Sat, 17 May 2025 15:00:29 GMT

@OrezBodajoy said in Backups keep failing:

@nebulon is there a way to update the app without doing the backup first?

Yes, just check the Skip backup checkbox

Reply to Backups keep failing on Sat, 17 May 2025 14:25:10 GMT

nebulon — Sat, 17 May 2025 14:25:10 GMT

@OrezBodajoy there don't seem to be any info on a backup task in those logs. Have you tried to create a manual backup from the app and then download the logs?

Reply to Backups keep failing on Sat, 17 May 2025 10:22:02 GMT

scooke — Sat, 17 May 2025 10:22:02 GMT

Nah, no thanks. good luck.

Reply to Backups keep failing on Sat, 17 May 2025 10:04:06 GMT

Dont-Worry — Sat, 17 May 2025 10:04:06 GMT

@scooke You can use SquareX extension to dl and open unknown files safely.

Reply to Backups keep failing on Sat, 17 May 2025 09:38:53 GMT

scooke — Sat, 17 May 2025 09:38:53 GMT

Is there no other way to share the logs? I don't feel like downloading an unknown file from a forum.

Reply to Backups keep failing on Sat, 17 May 2025 09:23:31 GMT

OrezBodajoy — Sat, 17 May 2025 09:23:31 GMT

@nebulon is there a way to update the app without doing the backup first?
I am now running a 6+ month old version without backups and this is really getting annoying now.

Note: switching to AWS S3 and changing the backup config has improved the situation but I still get an error during backup after a few hours.

Here is my log for reference. I only have a "Task 4059 crashed with code 1", I don't know what caused it.
https://upload.disroot.org/r/B82DZPjw#3JC3Iw05uQQVfACMKuVNPBtDjYha2BfXiKaSC82w3+E=

Is there no way to know what's causing the issue?

Reply to Backups keep failing on Thu, 06 Mar 2025 17:57:37 GMT

OrezBodajoy — Thu, 06 Mar 2025 17:57:37 GMT

@mdc773 you mean https://www.idrive.com/s3-storage-e2/ ? I will give it a try, thanks for the suggestion.

@nebulon yes. I am using a vpn, maybe it's linked to that.

Reply to Backups keep failing on Wed, 05 Mar 2025 06:20:21 GMT

nebulon — Wed, 05 Mar 2025 06:20:21 GMT

@OrezBodajoy so instead of an EPIPE error you are now seeing upload stalling (repeated @0MBps) until you get the timeout?

Reply to Backups keep failing on Tue, 04 Mar 2025 21:18:43 GMT

mdc773 — Tue, 04 Mar 2025 21:18:43 GMT

use idrive e2 my friend and my advise if your server is working proper right now. remove the backup config completely and setup a new bucket and then try that seems to fix my issue all the time.

Reply to Backups keep failing on Tue, 04 Mar 2025 14:44:40 GMT

OrezBodajoy — Tue, 04 Mar 2025 14:44:40 GMT

Quick update, sorry for the delay.
After migrating from backblaze to aws s3 it seems I no longer have the EPIPE errors.

Here is what I have on the latest failed backup logs:

Mar 04 01:53:23 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:29 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/2f/1d/2f1d57dd-f397-4715-a214-96838b38d205.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:37 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/30/f8/30f835f2-08c2-4366-9915-fd4a92064959.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:43 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70
[no timestamp]  new Error('Socket timed out without establishing a connection'),
[no timestamp]  ^
[no timestamp] 
[no timestamp]  Error [TimeoutError]: Socket timed out without establishing a connection
[no timestamp]  at Timeout.connectTimeout [as _onTimeout] (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70:15)
[no timestamp]  at listOnTimeout (node:internal/timers:581:17)
[no timestamp]  at process.processTimers (node:internal/timers:519:7) {
[no timestamp]  code: 'TimeoutError',
[no timestamp]  time: 2025-03-04T00:53:48.827Z,
[no timestamp]  region: 'eu-central-1',
[no timestamp]  hostname: 'impressive-dangly-smut.s3.eu-central-1.amazonaws.com',
[no timestamp]  retryable: true,
[no timestamp]  statusCode: 200
[no timestamp]  }
[no timestamp] 
[no timestamp]  Node.js v20.18.0
Mar 04 01:54:07 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess. (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess. (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask fullBackup: app photos.mysuper.website backup finished. Took 4302.722 seconds
Mar 04 01:54:07 box:locks write: current locks: {"backup_task":null}
Mar 04 01:54:07 box:locks release: app_14dd2bbe-1d61-4f3d-91d3-4af16e366076
Mar 04 01:54:07 box:taskworker Task took 6845.852 seconds
Mar 04 01:54:07 box:tasks setCompleted - 3297: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 box:tasks update 3297: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 BoxError: Backuptask crashed
Mar 04 01:54:07 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)
Mar 04 01:54:07 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Mar 04 01:54:07 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)
Mar 04 01:54:07 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)

Reply to Backups keep failing on Tue, 11 Feb 2025 13:38:44 GMT

nebulon — Tue, 11 Feb 2025 13:38:44 GMT

Can you share the logs for that also? There shouldn't be any 1h cutoff timeout set by Cloudron itself, so likely this is some other root cause.

Reply to Backups keep failing on Mon, 10 Feb 2025 17:38:59 GMT

OrezBodajoy — Mon, 10 Feb 2025 17:38:59 GMT

I am trying now with AWS.
I didn't get an epipe error but the "Task 3061 timed out" after 1h exactly.
My photo album is quite big and I don't expect it to be uploaded within an hour. Is there a way to increase this timeout limit for the backup task? I don't see anything about that in the advanced section of the backup configuration.

Thanks

Reply to Backups keep failing on Wed, 05 Feb 2025 11:26:47 GMT

joseph — Wed, 05 Feb 2025 11:26:47 GMT

@OrezBodajoy 8.2.4 has nothing backup related, so I think that won't resolve this issue. If you can use something other than B2, it will help atleast to isolate if this is a backblaze issue.

There's been many reports on this forum already about B2 having network issues. Maybe they throttle traffic after some upload sizes and maybe Cloudron code has to be smarter somehow.

Reply to Backups keep failing on Tue, 04 Feb 2025 18:01:57 GMT

OrezBodajoy — Tue, 04 Feb 2025 18:01:57 GMT

Platform versionv8.2.3 (Ubuntu 22.04.4 LTS)
I will try updating to 8.2.4

Would it work better if I use another backup method than backblaze?

Reply to Backups keep failing on Tue, 04 Feb 2025 16:30:38 GMT

joseph — Tue, 04 Feb 2025 16:30:38 GMT

It's the dreaded 408 error - request time out . But I also notice @0Mbps which I think was fixed in 8.2 .

@OrezBodajoy which version of Cloudron are you on?

Reply to Backups keep failing on Tue, 04 Feb 2025 13:45:08 GMT

OrezBodajoy — Tue, 04 Feb 2025 13:45:08 GMT

I have changed a bit the backup parameters in the UI:

memory limit to 9 GiB
upload part size 64 MiB
concurrency items 10

I now have this error:

Feb 04 13:48:29 box:storage/s3 Upload progress: {"loaded":62003204,"total":62003204,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ"}
Feb 04 13:48:32 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4","reason":"new","position":425}
Feb 04 13:48:32 box:storage/s3 Upload finished. {"ETag":"\"96b07711e9e9cd6313ae1d3bf2eab89d\"","VersionId":"4_z30c7662aa70ee25089da0419_f112b8445e8479200_d20250204_m124750_c003_v0312024_t0049_u01738673270973","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc%2B63z%2BGW%2BIIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:32 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 position 425 try 1
Feb 04 13:48:32 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 (photos.mysuper.website)"}
Feb 04 13:48:34 box:storage/s3 Upload progress: {"loaded":12934596,"total":12934596,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw"}
Feb 04 13:48:37 box:storage/s3 Upload progress: {"loaded":24622868,"total":24622868,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA"}
Feb 04 13:48:38 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4","reason":"new","position":426}
Feb 04 13:48:38 box:storage/s3 Upload finished. {"ETag":"\"a63eb3273bd8c6af9c8be8cdb8b3411c\"","VersionId":"4_z30c7662aa70ee25089da0419_f111b63a9bf79370d_d20250204_m124018_c003_v0312024_t0006_u01738672818986","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-%2Bbm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF%2BWvX4eM5wpaPa04Cp9mm0g9jyU%2BWHPQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:38 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 position 426 try 1
Feb 04 13:48:38 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 (photos.mysuper.website)"}
Feb 04 13:48:43 box:storage/s3 Upload finished. {"ETag":"\"e8f727fbe3017a736863acd3e005920f\"","VersionId":"4_z30c7662aa70ee25089da0419_f102a9d6210f3b68d_d20250204_m124644_c003_v0312028_t0032_u01738673204217","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:43 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4 position 427 try 1
Feb 04 13:48:43 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4","reason":"new","position":427}
Feb 04 13:48:45 box:storage/s3 Upload finished. {"ETag":"\"fef4d9e41a7b0487ef85cde77de337c7\"","VersionId":"4_z30c7662aa70ee25089da0419_f105b54dc655def0e_d20250204_m124514_c003_v0312025_t0014_u01738673114140","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE%2BWncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3%2BwvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:45 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4","reason":"new","position":428}
Feb 04 13:48:45 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 position 428 try 1
Feb 04 13:48:45 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 (photos.mysuper.website)"}
Feb 04 13:48:48 box:storage/s3 Upload progress: {"loaded":83903444,"total":83903444,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg"}
Feb 04 13:48:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:48:56 zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83%2BN073h%2BYeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","Bucket":"cloudron-backup-january-february","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","ETag":"\"de497871d785fd1c59918d801ccfa714-2\""}
Feb 04 13:48:56 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4 position 429 try 1
Feb 04 13:48:56 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4","reason":"new","position":429}
Feb 04 13:49:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:11 box:storage/s3 Upload progress: {"loaded":51287460,"total":118396324,"part":2,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/JXzM88Uqx7ZsSnmpRN00J7pjTkogYPeJj9sRGLylY-Y/lCpIzgRkjCJJlzwTqL5scjzLiKZZcSHPgHnseJC+9VU/xzreWsN82y+-8efk7r1FwykGI9qCIdoC1+WVWtIjKuolkKXYiFi4nMAPlEa4HbLqzGkhp4PA7nDfRQZSvpQgNA"}
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.044 * 10 changes in 300 seconds. Saving...
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.046 * Background saving started by pid 248
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.146 * Background saving terminated with success
Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.129 * DB saved on disk
Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.130 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 13:49:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:12 box:storage/s3 Upload progress: {"loaded":79710836,"total":146819700,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/LQs0pg3EACKpxLfIoyhm3-pwf2CjUC0Qmr45cpZ0XtY/zRON9tC4HPaxKx9vNEkZkXGSyTtIBEVsHwJTKqt6EAI/zSGMZ1+pEr5om7CkzelvYADDA4IPqQHHBv18fQYz-AzjLcjdhD8N9ce0c7KaGaYbYwyPXwq5IcGX07SjbJQWzw"}
Feb 04 13:50:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:36 box:storage/s3 Upload progress: {"loaded":21172484,"total":21172484,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q"}
Feb 04 13:50:39 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4 position 430 try 1
Feb 04 13:50:39 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4","reason":"new","position":430}
Feb 04 13:50:39 box:storage/s3 Upload finished. {"ETag":"\"e8db66fe56ced58e8bccbf20014a6b92\"","VersionId":"4_z30c7662aa70ee25089da0419_f1064034b6c53492b_d20250204_m124857_c003_v0312028_t0047_u01738673337237","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0%2BgncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy%2BIEnSGqo17KBKZjlQcIp7Q","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Bucket":"cloudron-backup-january-february"}
Feb 04 13:50:45 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Feb 04 13:50:46 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Feb 04 13:50:46 box:apptask run: app error for state pending_backup: BoxError: Backuptask crashed at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5) at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12) at async backup (/home/yellowtent/box/src/apptask.js:400:22) { reason: 'Internal Error', details: {} }
Feb 04 13:50:46 box:taskworker Task took 12148.089 seconds
Feb 04 13:50:46 box:tasks setCompleted - 3016: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Feb 04 13:50:46 box:tasks update 3016: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.044 * 10 changes in 300 seconds. Saving...
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.046 * Background saving started by pid 249
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.147 * Background saving terminated with success
Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.137 * DB saved on disk
Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.139 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.020 * 10 changes in 300 seconds. Saving...
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.022 * Background saving started by pid 250
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.223 * Background saving terminated with success
Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.124 * DB saved on disk
Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.125 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.094 * 10 changes in 300 seconds. Saving...
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.097 * Background saving started by pid 251
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.198 * Background saving terminated with success
Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.188 * DB saved on disk
Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.189 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.081 * 10 changes in 300 seconds. Saving...
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.082 * Background saving started by pid 252
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.285 * Background saving terminated with success
Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.184 * DB saved on disk
Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.185 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.059 * 10 changes in 300 seconds. Saving...
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.061 * Background saving started by pid 253
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.262 * Background saving terminated with success
Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.160 * DB saved on disk
Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.162 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.096 * 10 changes in 300 seconds. Saving...
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.098 * Background saving started by pid 254
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.198 * Background saving terminated with success
Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.175 * DB saved on disk
Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.177 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.048 * 10 changes in 300 seconds. Saving...
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.049 * Background saving started by pid 255
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.149 * Background saving terminated with success
Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.142 * DB saved on disk
Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.144 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.087 * 10 changes in 300 seconds. Saving...
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.089 * Background saving started by pid 256
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.190 * Background saving terminated with success
Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.163 * DB saved on disk
Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.165 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.052 * 10 changes in 300 seconds. Saving...
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.053 * Background saving started by pid 257
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.154 * Background saving terminated with success
Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.134 * DB saved on disk
Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.136 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:34:21 408: null
Feb 04 14:34:21 BoxError: Backuptask crashed
Feb 04 14:34:21 Node.js v20.18.0
Feb 04 14:34:21 }
Feb 04 14:34:21 }
Feb 04 14:34:21 }

Thanks!

Reply to Backups keep failing on Fri, 31 Jan 2025 07:57:08 GMT

nebulon — Fri, 31 Jan 2025 07:57:08 GMT

Looking further at the logs, it seems at least that last run still hit the EPIPE error, where the uploading was cancelled while trying to upload a .mp4 file. This could either be a memory issue on your server or also the remote storage terminated the connection prematurely.

Reply to Backups keep failing on Fri, 31 Jan 2025 07:50:43 GMT

nebulon — Fri, 31 Jan 2025 07:50:43 GMT

For a start, the next immich package version has the built-in database backups disabled, as they are of no use on Cloudron and also try to write into read-only storage.

To get your app up and running again, please hit the "Retry task" in the repair tab of that apps configure section.

Reply to Backups keep failing on Thu, 30 Jan 2025 20:51:54 GMT

OrezBodajoy — Thu, 30 Jan 2025 20:51:54 GMT

No new immich backup. Right now the app is marked as Error and on the backup tab the 'Create backup' button is greyed-out.
In the notifications I also get a backup task crashed notification.
Here are the logs for both immich and that general backup task crashed.

https://upload.disroot.org/r/nuvYocg7#xGyeIOHzUekVeQhAC3Aup1vR7/SZqPepg3/aC10aSW0=
https://upload.disroot.org/r/9MFpVBdn#B9BzBZ8buWuNh0SPq6j605GBQcCqIedo0G/jI2J956g=

Please have a look at the logs.
Thanks!

Reply to Backups keep failing on Wed, 29 Jan 2025 08:21:57 GMT

nebulon — Wed, 29 Jan 2025 08:21:57 GMT

Those are immich errors, not related to any Cloudron backup. Looks like immich added its own database backup job by now which will not work on Cloudron and we have to disable this in the package. But either way you can ignore those. What is the status on your Cloudron backup of the app?

Reply to Backups keep failing on Tue, 28 Jan 2025 19:32:30 GMT

OrezBodajoy — Tue, 28 Jan 2025 19:32:30 GMT

I am getting these errors now:

Jan 27 20:47:44 descriptor
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Error while closing socket [Errno 9] Bad file
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Shutting down
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Shutting down due to inactivity.
Jan 27 20:47:44 [01/27/25 19:47:44] INFO Waiting for application shutdown.
Jan 27 20:47:48 [01/27/25 19:47:48] ERROR Worker (pid:772) was sent SIGINT!
Jan 27 20:47:48 [01/27/25 19:47:48] INFO Application shutdown complete.
Jan 27 20:47:48 [01/27/25 19:47:48] INFO Booting worker with pid: 907
Jan 27 20:47:48 [01/27/25 19:47:48] INFO Finished server process [772]
Jan 27 20:47:57 of inactivity.
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Application startup complete.
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Created in-memory cache with unloading after 300s
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Initialized request thread pool with 4 threads.
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Started server process [907]
Jan 27 20:47:57 [01/27/25 19:47:57] INFO Waiting for application startup.
Jan 28 01:00:00 2025-01-28T00:00:00Z
Jan 28 01:00:00 2025-01-28T00:00:00Z
Jan 28 01:00:00 2025-01-28T00:00:00Z
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/66/b4/66b49043-0cc2-48a9-b163-c138a82362b8.HEIC
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/e4/e6/e4e69bc7-abab-4fb2-9b16-d01f28d3819d.jpg
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: VipsJpeg: Premature end of input file
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object:
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object:
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object:
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/66/b4/66b49043-0cc2-48a9-b163-c138a82362b8.HEIC
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/e4/e6/e4e69bc7-abab-4fb2-9b16-d01f28d3819d.jpg
Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: VipsJpeg: Premature end of input file
Jan 28 01:00:00 "id": "60e8f369-c07d-4e55-b54a-904f455f8366"
Jan 28 01:00:00 "id": "78bd4060-73b1-4687-ad99-719856cfed7f"

and

Jan 28 01:00:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Jan 28 01:00:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jan 28 01:00:00 {
Jan 28 01:00:00 {
Jan 28 01:00:00 {
Jan 28 01:00:00 }
Jan 28 01:00:00 }
Jan 28 01:00:00 }
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:Backup failed with code 1] Database Backup Failure
Jan 28 03:00:00 R [Microservices:BackupService] Backup failed with code 1
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:BackupService] Gzip exited with code 0 but pgdump exited with 1
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:BackupService] pg_dumpall: error: query failed: ERROR: permission denied for table pg_authid
Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM LOG [Microservices:BackupService] Database Backup Starting. Database Version: 14
Jan 28 03:00:00
Jan 28 03:00:00 pg_dumpall: error: query was: SELECT rolname FROM pg_authid WHERE rolname !~ '^pg_' ORDER BY 1
Jan 28 16:22:36 [Nest] 47 - 01/28/2025, 3:22:36 PM LOG [Api:EventRepository] Websocket Connect: LBAslGNTdfqXxBm4AAAL
Jan 28 16:23:39 [Nest] 47 - 01/28/2025, 3:23:39 PM LOG [Api:EventRepository] Websocket Disconnect: LBAslGNTdfqXxBm4AAAL
Jan 28 18:31:37 ['CPUExecutionProvider'], in descending order of
Jan 28 18:31:37 preference
Jan 28 18:31:37 [01/28/25 17:31:37] INFO Loading visual model 'ViT-B-32__openai' to memory
Jan 28 18:31:37 [01/28/25 17:31:37] INFO Setting execution providers to
Jan 28 18:31:39 ['CPUExecutionProvider'], in descending order of
Jan 28 18:31:39 preference
Jan 28 18:31:39 [01/28/25 17:31:39] INFO Loading detection model 'buffalo_l' to memory
Jan 28 18:31:39 [01/28/25 17:31:39] INFO Setting execution providers to
Jan 28 18:31:40 ['CPUExecutionProvider'], in descending order of
Jan 28 18:31:40 preference
Jan 28 18:31:40 [01/28/25 17:31:40] INFO Loading recognition model 'buffalo_l' to memory
Jan 28 18:31:40 [01/28/25 17:31:40] INFO Setting execution providers to
Jan 28 18:36:39 descriptor
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Error while closing socket [Errno 9] Bad file
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Shutting down
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Shutting down due to inactivity.
Jan 28 18:36:39 [01/28/25 17:36:39] INFO Waiting for application shutdown.
Jan 28 18:36:40 [01/28/25 17:36:40] ERROR Worker (pid:907) was sent SIGINT!
Jan 28 18:36:40 [01/28/25 17:36:40] INFO Application shutdown complete.
Jan 28 18:36:40 [01/28/25 17:36:40] INFO Booting worker with pid: 946
Jan 28 18:36:40 [01/28/25 17:36:40] INFO Finished server process [907]
Jan 28 18:36:47 of inactivity.

I am not sure how to handle that...

Reply to Backups keep failing on Sun, 26 Jan 2025 11:47:14 GMT

OrezBodajoy — Sun, 26 Jan 2025 11:47:14 GMT

I am indeed using backblaze.
I have increased the memory limit from the default 1G to 5G.
I have also increased the part-size from the default 10MB to 128MB (is that what you mean by tweak? Or should I reduce it instead?).
I will let you know how it goes. As it's a big library of photos (100+GB?), it's taking a while.