Backups keep failing

[OrezBodajoy]

I have changed a bit the backup parameters in the UI:

• memory limit to 9 GiB
• upload part size 64 MiB
• concurrency items 10

I now have this error:

Feb 04 13:48:29 box:storage/s3 Upload progress: {"loaded":62003204,"total":62003204,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ"}
Feb 04 13:48:32 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4","reason":"new","position":425}
Feb 04 13:48:32 box:storage/s3 Upload finished. {"ETag":"\"96b07711e9e9cd6313ae1d3bf2eab89d\"","VersionId":"4_z30c7662aa70ee25089da0419_f112b8445e8479200_d20250204_m124750_c003_v0312024_t0049_u01738673270973","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc%2B63z%2BGW%2BIIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:32 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 position 425 try 1
Feb 04 13:48:32 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 (photos.mysuper.website)"}
Feb 04 13:48:34 box:storage/s3 Upload progress: {"loaded":12934596,"total":12934596,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw"}
Feb 04 13:48:37 box:storage/s3 Upload progress: {"loaded":24622868,"total":24622868,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA"}
Feb 04 13:48:38 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4","reason":"new","position":426}
Feb 04 13:48:38 box:storage/s3 Upload finished. {"ETag":"\"a63eb3273bd8c6af9c8be8cdb8b3411c\"","VersionId":"4_z30c7662aa70ee25089da0419_f111b63a9bf79370d_d20250204_m124018_c003_v0312024_t0006_u01738672818986","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-%2Bbm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF%2BWvX4eM5wpaPa04Cp9mm0g9jyU%2BWHPQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:38 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 position 426 try 1
Feb 04 13:48:38 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 (photos.mysuper.website)"}
Feb 04 13:48:43 box:storage/s3 Upload finished. {"ETag":"\"e8f727fbe3017a736863acd3e005920f\"","VersionId":"4_z30c7662aa70ee25089da0419_f102a9d6210f3b68d_d20250204_m124644_c003_v0312028_t0032_u01738673204217","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:43 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4 position 427 try 1
Feb 04 13:48:43 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4","reason":"new","position":427}
Feb 04 13:48:45 box:storage/s3 Upload finished. {"ETag":"\"fef4d9e41a7b0487ef85cde77de337c7\"","VersionId":"4_z30c7662aa70ee25089da0419_f105b54dc655def0e_d20250204_m124514_c003_v0312025_t0014_u01738673114140","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE%2BWncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3%2BwvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Bucket":"cloudron-backup-january-february"}
Feb 04 13:48:45 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4","reason":"new","position":428}
Feb 04 13:48:45 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 position 428 try 1
Feb 04 13:48:45 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 (photos.mysuper.website)"}
Feb 04 13:48:48 box:storage/s3 Upload progress: {"loaded":83903444,"total":83903444,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg"}
Feb 04 13:48:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:48:56 zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83%2BN073h%2BYeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","Bucket":"cloudron-backup-january-february","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","ETag":"\"de497871d785fd1c59918d801ccfa714-2\""}
Feb 04 13:48:56 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4 position 429 try 1
Feb 04 13:48:56 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4","reason":"new","position":429}
Feb 04 13:49:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:11 box:storage/s3 Upload progress: {"loaded":51287460,"total":118396324,"part":2,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/JXzM88Uqx7ZsSnmpRN00J7pjTkogYPeJj9sRGLylY-Y/lCpIzgRkjCJJlzwTqL5scjzLiKZZcSHPgHnseJC+9VU/xzreWsN82y+-8efk7r1FwykGI9qCIdoC1+WVWtIjKuolkKXYiFi4nMAPlEa4HbLqzGkhp4PA7nDfRQZSvpQgNA"}
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.044 * 10 changes in 300 seconds. Saving...
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.046 * Background saving started by pid 248
Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.146 * Background saving terminated with success
Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.129 * DB saved on disk
Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.130 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 13:49:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:49:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:12 box:storage/s3 Upload progress: {"loaded":79710836,"total":146819700,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/LQs0pg3EACKpxLfIoyhm3-pwf2CjUC0Qmr45cpZ0XtY/zRON9tC4HPaxKx9vNEkZkXGSyTtIBEVsHwJTKqt6EAI/zSGMZ1+pEr5om7CkzelvYADDA4IPqQHHBv18fQYz-AzjLcjdhD8N9ce0c7KaGaYbYwyPXwq5IcGX07SjbJQWzw"}
Feb 04 13:50:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:36 box:storage/s3 Upload progress: {"loaded":21172484,"total":21172484,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q"}
Feb 04 13:50:39 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4 position 430 try 1
Feb 04 13:50:39 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4","reason":"new","position":430}
Feb 04 13:50:39 box:storage/s3 Upload finished. {"ETag":"\"e8db66fe56ced58e8bccbf20014a6b92\"","VersionId":"4_z30c7662aa70ee25089da0419_f1064034b6c53492b_d20250204_m124857_c003_v0312028_t0047_u01738673337237","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0%2BgncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy%2BIEnSGqo17KBKZjlQcIp7Q","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Bucket":"cloudron-backup-january-february"}
Feb 04 13:50:45 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"}
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null
Feb 04 13:50:46 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Feb 04 13:50:46 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Feb 04 13:50:46 box:apptask run: app error for state pending_backup: BoxError: Backuptask crashed at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5) at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12) at async backup (/home/yellowtent/box/src/apptask.js:400:22) { reason: 'Internal Error', details: {} }
Feb 04 13:50:46 box:taskworker Task took 12148.089 seconds
Feb 04 13:50:46 box:tasks setCompleted - 3016: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Feb 04 13:50:46 box:tasks update 3016: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.044 * 10 changes in 300 seconds. Saving...
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.046 * Background saving started by pid 249
Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.147 * Background saving terminated with success
Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.137 * DB saved on disk
Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.139 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.020 * 10 changes in 300 seconds. Saving...
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.022 * Background saving started by pid 250
Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.223 * Background saving terminated with success
Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.124 * DB saved on disk
Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.125 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.094 * 10 changes in 300 seconds. Saving...
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.097 * Background saving started by pid 251
Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.198 * Background saving terminated with success
Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.188 * DB saved on disk
Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.189 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.081 * 10 changes in 300 seconds. Saving...
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.082 * Background saving started by pid 252
Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.285 * Background saving terminated with success
Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.184 * DB saved on disk
Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.185 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.059 * 10 changes in 300 seconds. Saving...
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.061 * Background saving started by pid 253
Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.262 * Background saving terminated with success
Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.160 * DB saved on disk
Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.162 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.096 * 10 changes in 300 seconds. Saving...
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.098 * Background saving started by pid 254
Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.198 * Background saving terminated with success
Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.175 * DB saved on disk
Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.177 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.048 * 10 changes in 300 seconds. Saving...
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.049 * Background saving started by pid 255
Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.149 * Background saving terminated with success
Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.142 * DB saved on disk
Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.144 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.087 * 10 changes in 300 seconds. Saving...
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.089 * Background saving started by pid 256
Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.190 * Background saving terminated with success
Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.163 * DB saved on disk
Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.165 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.052 * 10 changes in 300 seconds. Saving...
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.053 * Background saving started by pid 257
Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.154 * Background saving terminated with success
Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.134 * DB saved on disk
Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.136 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
Feb 04 14:34:21 408: null
Feb 04 14:34:21 BoxError: Backuptask crashed
Feb 04 14:34:21 Node.js v20.18.0
Feb 04 14:34:21 }
Feb 04 14:34:21 }
Feb 04 14:34:21 }

Thanks!

[joseph]

It's the dreaded 408 error - request time out . But I also notice @0Mbps which I think was fixed in 8.2 .

@OrezBodajoy which version of Cloudron are you on?

[OrezBodajoy]

Platform versionv8.2.3 (Ubuntu 22.04.4 LTS)
I will try updating to 8.2.4

Would it work better if I use another backup method than backblaze?

[joseph]

@OrezBodajoy 8.2.4 has nothing backup related, so I think that won't resolve this issue. If you can use something other than B2, it will help atleast to isolate if this is a backblaze issue.

There's been many reports on this forum already about B2 having network issues. Maybe they throttle traffic after some upload sizes and maybe Cloudron code has to be smarter somehow.

[OrezBodajoy]

I am trying now with AWS.
I didn't get an epipe error but the "Task 3061 timed out" after 1h exactly.
My photo album is quite big and I don't expect it to be uploaded within an hour. Is there a way to increase this timeout limit for the backup task? I don't see anything about that in the advanced section of the backup configuration.

Thanks

[nebulon]

Can you share the logs for that also? There shouldn't be any 1h cutoff timeout set by Cloudron itself, so likely this is some other root cause.

[OrezBodajoy]

Quick update, sorry for the delay.
After migrating from backblaze to aws s3 it seems I no longer have the EPIPE errors.

Here is what I have on the latest failed backup logs:

Mar 04 01:53:23 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:29 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/2f/1d/2f1d57dd-f397-4715-a214-96838b38d205.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:37 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/30/f8/30f835f2-08c2-4366-9915-fd4a92064959.mp4: 46M@0MBps (photos.mysuper.website)"}
Mar 04 01:53:43 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"}
[no timestamp]  /home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70
[no timestamp]  new Error('Socket timed out without establishing a connection'),
[no timestamp]  ^
[no timestamp] 
[no timestamp]  Error [TimeoutError]: Socket timed out without establishing a connection
[no timestamp]  at Timeout.connectTimeout [as _onTimeout] (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70:15)
[no timestamp]  at listOnTimeout (node:internal/timers:581:17)
[no timestamp]  at process.processTimers (node:internal/timers:519:7) {
[no timestamp]  code: 'TimeoutError',
[no timestamp]  time: 2025-03-04T00:53:48.827Z,
[no timestamp]  region: 'eu-central-1',
[no timestamp]  hostname: 'impressive-dangly-smut.s3.eu-central-1.amazonaws.com',
[no timestamp]  retryable: true,
[no timestamp]  statusCode: 200
[no timestamp]  }
[no timestamp] 
[no timestamp]  Node.js v20.18.0
Mar 04 01:54:07 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null
Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28)
Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12)
Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Mar 04 01:54:07 reason: 'Shell Error',
Mar 04 01:54:07 details: {},
Mar 04 01:54:07 code: 1,
Mar 04 01:54:07 signal: null
Mar 04 01:54:07 }
Mar 04 01:54:07 box:backuptask fullBackup: app photos.mysuper.website backup finished. Took 4302.722 seconds
Mar 04 01:54:07 box:locks write: current locks: {"backup_task":null}
Mar 04 01:54:07 box:locks release: app_14dd2bbe-1d61-4f3d-91d3-4af16e366076
Mar 04 01:54:07 box:taskworker Task took 6845.852 seconds
Mar 04 01:54:07 box:tasks setCompleted - 3297: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 box:tasks update 3297: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Mar 04 01:54:07 BoxError: Backuptask crashed
Mar 04 01:54:07 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)
Mar 04 01:54:07 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Mar 04 01:54:07 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)
Mar 04 01:54:07 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)

[mdc773]

use idrive e2 my friend and my advise if your server is working proper right now. remove the backup config completely and setup a new bucket and then try that seems to fix my issue all the time.

[nebulon]

@OrezBodajoy so instead of an EPIPE error you are now seeing upload stalling (repeated @0MBps) until you get the timeout?

[OrezBodajoy]

@mdc773 you mean https://www.idrive.com/s3-storage-e2/ ? I will give it a try, thanks for the suggestion.

@nebulon yes. I am using a vpn, maybe it's linked to that.

[OrezBodajoy]

@nebulon is there a way to update the app without doing the backup first?
I am now running a 6+ month old version without backups and this is really getting annoying now.

Note: switching to AWS S3 and changing the backup config has improved the situation but I still get an error during backup after a few hours.

Here is my log for reference. I only have a "Task 4059 crashed with code 1", I don't know what caused it.
https://upload.disroot.org/r/B82DZPjw#3JC3Iw05uQQVfACMKuVNPBtDjYha2BfXiKaSC82w3+E=

Is there no way to know what's causing the issue?

[scooke]

Is there no other way to share the logs? I don't feel like downloading an unknown file from a forum.

[Dont-Worry]

@scooke You can use SquareX extension to dl and open unknown files safely.

[scooke]

Nah, no thanks. good luck.

[nebulon]

@OrezBodajoy there don't seem to be any info on a backup task in those logs. Have you tried to create a manual backup from the app and then download the logs?

[jdaviescoates]

@OrezBodajoy said in Backups keep failing:

@nebulon is there a way to update the app without doing the backup first?

Yes, just check the Skip backup checkbox

[OrezBodajoy]

Thanks @jdaviescoates ! I will try that!

@nebulon I just checked the system logs and there are a bunch of out of memory error logs:
Out of memory: Killed process XXXXX (ruby2.3 - and sometimea node) total-vm: 1704896kB, anon-rss: 891324kB, file-rss: 0kB, shmem-rss: 0kB, UID: XXXX pgtables: 2072kB, ooom_score_adj: 0

I will check to see if there are still some backup memory settings I need to increase.

[nebulon]

given that ruby is mentioned there, it maybe that the system itself runs out of memory causing things to become unstable. At least the backup task would run via nodejs not ruby. Immich also isn't using ruby so this comes from some other app then.