Backups failing for 3 weeks

[adrw]

My server is on Cloudron 6.2.4 and backups have been failing for the past 3 weeks with error logs that don't seem to indicate any of the past bugs I've encountered (and have since been fixed – many thanks to @girish and others). I've tried reducing memory usage of services like Email (turning off the full text Solr search) thinking Cloudron was running out of memory but nothing seems to help the backup proceed.

Any ideas?

Mar 13 14:36:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:36:16.877Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25609544 external: 33118796
Mar 13 14:36:16 box:backupupload v8 heap : used 25609544 total: 27140096 max: 4076863488
Mar 13 14:36:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:36:46.877Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25641704 external: 33118796
Mar 13 14:36:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:36:46.877Z box:backupupload v8 heap : used 25641704 total: 27140096 max: 4076863488
Mar 13 14:37:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:37:16.878Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25674104 external: 33118796
Mar 13 14:37:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:37:16.878Z box:backupupload v8 heap : used 25674104 total: 27140096 max: 4076863488
Mar 13 14:37:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:37:46.878Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25706264 external: 33118796
Mar 13 14:37:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:37:46.878Z box:backupupload v8 heap : used 25706264 total: 27140096 max: 4076863488
Mar 13 14:38:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:38:16.880Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25738424 external: 33118796
Mar 13 14:38:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:38:16.880Z box:backupupload v8 heap : used 25738424 total: 27140096 max: 4076863488
Mar 13 14:38:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:38:46.880Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25770960 external: 33118796
Mar 13 14:38:46 box:backupupload v8 heap : used 25770960 total: 27140096 max: 4076863488
Mar 13 14:39:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:39:16.882Z box:backupupload process: rss: 114876416 heapTotal: 27140096 heapUsed: 25803120 external: 33118796
Mar 13 14:39:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:39:16.882Z box:backupupload v8 heap : used 25803120 total: 27140096 max: 4076863488
Mar 13 14:39:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:39:46.881Z box:backupupload process: rss: 115146752 heapTotal: 27140096 heapUsed: 25835520 external: 33118796
Mar 13 14:39:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:39:46.881Z box:backupupload v8 heap : used 25835520 total: 27140096 max: 4076863488
Mar 13 14:40:16 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:40:16.883Z box:backupupload process: rss: 115146752 heapTotal: 27140096 heapUsed: 25867680 external: 33118796
Mar 13 14:40:16 box:backupupload v8 heap : used 25867680 total: 27140096 max: 4076863488
Mar 13 14:40:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:40:46.883Z box:backupupload process: rss: 115146752 heapTotal: 27140096 heapUsed: 25900048 external: 33118796
Mar 13 14:40:46 box:shell backup-snapshot/app_62c2ebf9-92ea-4038-84db-097e7fe100a7 (stdout): 2021-03-13T19:40:46.883Z box:backupupload v8 heap : used 25900048 total: 27140096 max: 4076863488

[nebulon]

What is the failure you are seeing, like does it just time-out or is there a crash?

The logs you pasted don't seem to contain actual errors to investigate further.

[adrw]

@nebulon There isn't an immediately visible stack trace, it seems to time out. I've uploaded the full logs here

[nebulon]

@adrw hm I see, so the snapshotting is working and then during backup rotation, uploading the tarball it just stops without further information looks like:

2021-03-16T20:59:50.991Z box:tasks 3493: {"percent":64.63636363636364,"message":"Copying part 231 - /my-adrw-xyz/snapshot/app_c4dd3c6d-806e-454b-beba-4cdd3f29865a.tar.gz.enc bytes=246960619520-248034361343 (cloud.alxdr.ca)"}

Which storage provider is this? Maybe you can increase the part-size for s3 in the backup configs (if that applies).

[adrw]

@nebulon I'm using BackblazeB2, thanks for looking into this!

[adrw]

@nebulon I tried increasing backup chunk size from 256mb to 1GB and backups are still failing similarly, any idea or other configuration I should be tweaking?

[nebulon]

@adrw not really, maybe we have to debug this directly to get more information.
If you want you can enable remote support for us and send us a mail with your dashboard domain and reference to this thread at support@cloudron.io

[scooke]

FWIW, I've twice had backup failures when using Cloudron (at previous versions), and both times it was due to some image in the backup. One instance was definitely a Wordpress install, and the other some kind of crm, I think on the LAMP app. I never did figure out what it was exactly, but once I deleted one specific jpg, my backups resumed with no problem. The backup problems was an unexplained and unclear stall/hang, and the other was a failure. But neither, I am quite certain, had any definite pointer to the image as the problem.
As I think about it a bit more, the Wordpress image was actually an image hosted remotely on my cloudinary account. I reckon the backup was trying to pull that image and was stalling.
You never know.

[adrw]

@nebulon I enabled remote debugging and emailed, look forward to hearing back