Backup cleanup task starts but do not finish the job
Unsolved
Support
-
Use case 1: In one instance I see multiple started, but not finished backup cleaner jobs with the same log message.
box:taskworker Starting task 3335. Logs are at /home/yellowtent/platformdata/logs/tasks/3335.log box:backupcleaner run: retention is {"keepWithinSecs":604800} box:shell mounts: mountpoint -q -- /mnt/cloudronbackup box:database Connection 422 error: Packets out of order. Got: 0 Expected: 7 PROTOCOL_PACKETS_OUT_OF_ORDER
Use case 2: In another instance other error, but also started but not finished.
retention is{"keepDaily":3,"keepWeekly":4,"keepMonthly":6}
... 2024-11-29T03:30:29.878Z box:tasks update 21410: {"message":"Removing app backup (b178e0a6-ffe8-4ede-976c-c382f998a50d): app_b178e0a6-ffe8-4ede-976c-c382f998a50d_v3.55.0_97d403e1"} 2024-11-29T03:30:30.451Z box:tasks update 21410: {"message":"2024-11-14-030035-138/app_n8n.example.org_v3.55.0: Removing directory /mnt/cloudronbackup/my_example_org/2024-11-14-030035-138/app_n8n.example.org_v3.55.0"} 2024-11-29T03:30:30.523Z box:tasks update 21410: {"message":"Removing directory /mnt/cloudronbackup/my_example_org/2024-11-14-030035-138/app_n8n.example.org_v3.55.0"} 2024-11-29T03:30:30.524Z box:shell filesystem: rm -rf /mnt/cloudronbackup/my_example_org/2024-11-14-030035-138/app_n8n.example.org_v3.55.0 2024-11-29T03:38:00.508Z box:backupcleaner removeBackup: unable to prune backup directory /mnt/cloudronbackup/my_example_org/2024-11-14-030035-138: ENOTEMPTY: directory not empty, rmdir '/mnt/cloudronbackup/my_example_org/2024-11-14-030035-138' 2024-11-29T03:38:00.521Z box:backupcleaner removeBackup: removed 2024-11-14-030035-138/app_n8n.example.org_v3.55.0 ...
Use case 2: The Jobs are finished now as a error. Some magic or the there is a hidden timeout.
-
Looks like two totally different issues. For case 1 no clue why the database queries would be out of order. Is this easily reproducible?
For case 2, looking at the logs, I wonder why it has mixed messages about the app domain? Is this because you manually replaced those or is this actually mixing domains in the original logs? Why a
rm -rf
wouldn't work on a folder which contains data is strange. Can you check permissions with that folder? -