Backup Failing (Cloudron crashed/stopped)

[arshsahzad]

I'm using Scaleway object storage for backups of my cloudron server from the last 8 months, Now from the last 4 days, I'm getting an error - (Cloudron crashed/stopped). I tried to manual backup and increase the backup memory limit up to 1GB/256MB, but the same problem is occurring again and again.

Aug 29 02:00:02 box:settings initCache: pre-load settings
Aug 29 02:00:02 box:taskworker Starting task 4294. Logs are at /home/yellowtent/platformdata/logs/tasks/4294.log
Aug 29 02:00:02 box:tasks 4294: {"percent":2,"error":null}
Aug 29 02:00:02 box:tasks 4294: {"percent":1,"message":"Backing up gitt.example.com"}
Aug 29 02:00:02 box:backups gitt.example.com Backed up. Took 0.025 seconds
Aug 29 02:00:02 box:tasks 4294: {"percent":8.692307692307693,"message":"Backing up example.com"}
Aug 29 02:00:02 box:tasks 4294: {"percent":16.384615384615387,"message":"Snapshotting app example.com"}
Aug 29 02:00:02 box:services example.com backupAddons
Aug 29 02:00:02 box:services example.com backupAddons: Backing up ["mysql","localstorage","sendmail","ldap","scheduler"]
Aug 29 02:00:02 box:services example.com Backing up mysql
Aug 29 02:00:13 box:backups example.com snapshotApp: took 10.773 seconds
Aug 29 02:00:13 box:tasks 4294: {"percent":16.384615384615387,"message":"Uploading app snapshot example.com"}
Aug 29 02:00:13 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 tgz {"localRoot":"/home/yellowtent/appsdata/3267f633-54a1-47e9-8a96-40b4de678549","layout":[]}
Aug 29 02:00:14 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 (stdout): 2021-08-28T20:30:14.723Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/3267f633-54a1-47e9-8a96-40b4de678549","layout":[]} to snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549
Aug 29 02:00:14 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 (stdout): 2021-08-28T20:30:14.780Z box:settings initCache: pre-load settings
Aug 29 02:00:14 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 (stdout): 2021-08-28T20:30:14.878Z box:backupupload process: rss: 59314176 heapTotal: 56098816 heapUsed: 24669456 external: 2720545
Aug 29 02:00:14 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 (stdout): 2021-08-28T20:30:14.878Z box:backupupload v8 heap : used 24673784 total: 56098816 max: 2114191360
Aug 29 02:00:14 box:backups upload: id snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 format tgz dataLayout {"localRoot":"/home/yellowtent/appsdata/3267f633-54a1-47e9-8a96-40b4de678549","layout":[]}
Aug 29 02:00:25 box:tasks 4294: {"percent":16.384615384615387,"message":"Uploading backup 18M@2MBps (example.com)"}
Aug 29 02:00:51 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 (stdout): 2021-08-28T20:30:51.812Z box:backupupload process: rss: 119881728 heapTotal: 31801344 heapUsed: 26084128 external: 68944741
Aug 29 02:00:51 box:shell backup-snapshot/app_3267f633-54a1-47e9-8a96-40b4de678549 (stdout): 2021-08-28T20:30:51.836Z box:backupupload v8 heap : used 26088456 total: 31801344 max: 2114191360
Aug 29 02:00:51 box:tasks 4294: {"percent":16.384615384615387,"message":"Uploading backup 32M@2MBps (example.com)"}

[nebulon]

@arshsahzad looks like you also sent a support ticket. Will be handled there then and we will update this topic, once we understand the issue.

[arshsahzad]

@nebulon I got the response, It was happening due to the system was running out of memory. Now It's working fine after increasing the memory of the cloudron server.

I have one more question, what will be memory limit and upload part size for idle setup of the backup, what will be criteria for that...

[nebulon]

To conclude this, it was a memory issue. The instance as a whole was a bit overcommited.

If the backup task is idle, it won't consume any memory. Also Cloudron does not reserve memory based on the limits set, neither for backup nor for apps. The limit is just to avoid rouge apps or the backup task to bring kill other apps.