Backup timeout

[arnaudguy]

Hello,

Every day I encounter an issue where Cloudron restarts in the middle of the backup process. This causes the backups to fail. Is it possible to trigger backups via the command line?

Attached are the logs. Thank you in advance for your insights and explanations.

Dec 04 11:58:10 box:apphealthmonitor app health: 20 running / 0 stopped / 0 unresponsive
Dec 04 11:58:30 box:apphealthmonitor setHealth: ece432d9-f98a-4ab4-ad09-bafa1cdaff08 (desk.xx.xx) waiting for 1179.554 to update health
Dec 04 11:58:55 box:locker Lock unreleased full_backup
Dec 04 11:58:55 ServerError [ServiceUnavailableError]: Response timeout
Dec 04 11:58:55 at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
Dec 04 11:58:55 at IncomingMessage.emit (node:events:519:28)
Dec 04 11:58:55 at IncomingMessage.emit (node:domain:488:12)
Dec 04 11:58:55 at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
Dec 04 11:58:55 at listOnTimeout (node:internal/timers:581:17)
Dec 04 11:58:55 at process.processTimers (node:internal/timers:519:7) {
Dec 04 11:58:55 code: 'ETIMEDOUT',
Dec 04 11:58:55 timeout: 60000
Dec 04 11:58:55 }
Dec 04 11:58:55 ServerError [ServiceUnavailableError]: Response timeout
Dec 04 11:58:55 at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
Dec 04 11:58:55 at IncomingMessage.emit (node:events:519:28)
Dec 04 11:58:55 at IncomingMessage.emit (node:domain:488:12)
Dec 04 11:58:55 at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
Dec 04 11:58:55 at listOnTimeout (node:internal/timers:581:17)
Dec 04 11:58:55 at process.processTimers (node:internal/timers:519:7) {
Dec 04 11:58:55 code: 'ETIMEDOUT',
Dec 04 11:58:55 timeout: 60000
Dec 04 11:58:55 }
Dec 04 11:59:20 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
Dec 04 11:59:20 + actual - expected
Dec 04 11:59:20 2024-12-04T10:59:20.702Z
Dec 04 11:59:20 + 'undefined'
Dec 04 11:59:20 - 'string'
Dec 04 11:59:20 at get (/home/yellowtent/box/src/routes/tasks.js:33:12)
Dec 04 11:59:20 at Layer.handle [as handle_request] (/home/yellowtent/box/node_modules/express/lib/router/layer.js:95:5)
Dec 04 11:59:20 at next (/home/yellowtent/box/node_modules/express/lib/router/route.js:149:13)
Dec 04 11:59:20 at load (/home/yellowtent/box/src/routes/tasks.js:29:5)
Dec 04 11:59:20 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Dec 04 11:59:22 box:server ==========================================
Dec 04 11:59:22 box:server Cloudron 8.1.0
Dec 04 11:59:22 box:server ==========================================
Dec 04 11:59:22 box:platform initialize: start platform
Dec 04 11:59:22 box:tasks stopAllTasks: stopping all tasks
Dec 04 11:59:22 box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
Dec 04 11:59:22 box:shell box-task-16616.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 16616 /home/yellowtent/platformdata/logs/tasks/16616.log
Dec 04 11:59:22 box-task-16617.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 16617 /home/yellowtent/platformdata/logs/tasks/16617.log
Dec 04 11:59:22
Dec 04 11:59:22 box:shell All tasks stopped


2024-12-04T10:57:52.611Z box:tasks update 16618: {"percent":18.391304347826086,"message":"Backing up matrix.xx.xx (5/20)"}
2024-12-04T10:57:52.613Z box:tasks update 16618: {"percent":22.73913043478261,"message":"Snapshotting app matrix.xx.xx"}
2024-12-04T10:57:52.614Z box:services backupAddons
2024-12-04T10:57:52.614Z box:services backupAddons: backing up ["localstorage","oidc","postgresql","sendmail","turn"]
2024-12-04T10:57:52.614Z box:services Backing up postgresql
2024-12-04T10:57:53.188Z box:services pipeRequestToFile: connected with status code 200
2024-12-04T10:59:07.681Z box:backuptask snapshotApp: matrix.xx.xx took 75.068 seconds
2024-12-04T10:59:07.681Z box:tasks update 16618: {"percent":22.73913043478261,"message":"Uploading app snapshot matrix.xx.xx"}
2024-12-04T10:59:07.682Z box:backuptask runBackupUpload: adjusting heap size to 8192M
2024-12-04T10:59:07.682Z box:shell backuptask /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_51b362b8-a595-476a-8376-cda3beb8a24c tgz {"localRoot":"/home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c","layout":[]}
2024-12-04T10:59:12.046Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c","layout":[]} to snapshot/app_51b362b8-a595-476a-8376-cda3beb8a24c
2024-12-04T10:59:12.067Z box:backuptask upload: path snapshot/app_51b362b8-a595-476a-8376-cda3beb8a24c format tgz dataLayout {"localRoot":"/home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c","layout":[]}
2024-12-04T10:59:12.078Z box:backuptask checkPreconditions: mount point status is {"state":"active"}&&&&&
2024-12-04T10:59:12.182Z box:backuptask checkPreconditions: getting disk usage of /home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c
2024-12-04T10:59:12.182Z box:shell backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c

[nebulon]

That is a very strange crash at https://git.cloudron.io/platform/box/-/blob/master/src/routes/tasks.js?ref_type=heads#L33 which is only every called via https://git.cloudron.io/platform/box/-/blob/master/src/server.js?ref_type=heads#L141

Can't see how that route handler would be called unless the code is somehow changed. But it really looks like some execution error caused by that intermediate timeout.

[arnaudguy]

Hi, I've see it's always crash during the snapshot (dump postgresql) of gitlab.

[nebulon]

Do you see any further errors in the postgresql service logs during such times? Maybe that runs out of memory for example?

[arnaudguy]

no i don't see that. But I have a server that is very loaded with processes.

[arnaudguy]

[arnaudguy]

Hi,
Yesterday night I'have the same situation. Now i'm sure it's was alaways on backup of gitlab.

Thanks

[nebulon]

We may have to debug this on your instance then directly. Can you send us a mail to support@cloudron.io and enable remote SSH support for us? https://docs.cloudron.io/support/#remote-support

[arnaudguy]

Hi, Thank you, I've send email.

[arnaudguy]

The issue is now resolved. The problem was with the GitLab database, where we found 160 GB of logs. The database dump was failing because there wasn’t enough free disk space. We have purged the database, and backups are now successful.

Thank you for your support.