Backup Failed - App task is currently in progress

[BrutalBirdie]

A Cloudron backup task was running for too long.
Did not cancel it and just waited.

Now the Dashboard displays the following:

and if I check in the system it self there is still the task:

systemctl status box-task-5154.service
● box-task-5154.service - /home/yellowtent/box/src/scripts/../taskworker.js 5154 /home/yellowtent/platformdata/logs/tas>
     Loaded: loaded (/run/systemd/transient/box-task-5154.service; transient)
  Transient: yes
     Active: active (running) since Sun 2023-12-31 00:00:00 UTC; 10h ago
   Main PID: 1759055 (node)
      Tasks: 23 (limit: 4479)
     Memory: 493.8M (max: 800.0M)
     CGroup: /system.slice/box-task-5154.service
             ├─1759055 node /home/yellowtent/box/src/scripts/../taskworker.js 5154 /home/yellowtent/platformdata/logs/t>
             ├─1759590 /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app>
             └─1759591 node /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_182aa3d8-8af7-4870-9699-2af09>

Warning: some journal files were not opened due to insufficient permissions.

This can lead to a support case if the User is not familiar with the Linux system it self and knows how to cancel that job in the system it self.

[robi]

Exposing the error would be a good start.

[BrutalBirdie]

@robi There is no "real" error.
The backup task was still running, the error "timed out" is a timed event from Cloudron it self.
So there is no "real" error and the message "timed out" is correct, but that the task it self did not get canceled in the background is questionable.
Maybe the "timed out" logic can be improved to see if the task is actually doing something or is "stuck".

[robi]

@BrutalBirdie Ah, I thought you were pointing out the permissions issue as to why it was stuck.

[nebulon]

So the backup task was still going on and finishing the backup eventually or are there any specific errors in that tasks logs?

[BrutalBirdie]

@nebulon No the backup never finished, I had to cancel the task with systemctl to be able to run a new backup again.

[girish]

@BrutalBirdie The backup UI is showing task 5146 whereas your systemctl is for task 5154 . These might just be different backup tasks.

Do you see a message like startTask: task 5154 took too long. killing (or 5146) ? If that message is displayed, it kills the systemd task as well.

[BrutalBirdie]

@girish Oh!
On 5154

More logs on 5154:

/home/yellowtent/platformdata/logs/box.log.1:65263:2023-12-31T00:00:00.104Z box:tasks startTask - starting task 5154 with options {"timeout":72000000,"nice":15,"memoryLimit":800}. logs at /home/yellowtent/platformdata/logs/tasks/5154.log
/home/yellowtent/platformdata/logs/box.log.1:65264:2023-12-31T00:00:00.104Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 5154 /home/yellowtent/platformdata/logs/tasks/5154.log 15 800
/home/yellowtent/platformdata/logs/box.log.1:65266:2023-12-31T00:00:00.197Z box:shell startTask (stderr): Running as unit: box-task-5154.service

On 5146:

More logs on 5146:

/home/yellowtent/platformdata/logs/box.log.1:54904:2023-12-29T22:00:00.094Z box:tasks startTask - starting task 5146 with options {"timeout":86400000,"nice":15,"memoryLimit":800}. logs at /home/yellowtent/platformdata/logs/tasks/5146.log
/home/yellowtent/platformdata/logs/box.log.1:54905:2023-12-29T22:00:00.094Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 5146 /home/yellowtent/platformdata/logs/tasks/5146.log 15 800
/home/yellowtent/platformdata/logs/box.log.1:54907:2023-12-29T22:00:00.147Z box:shell startTask (stderr): Running as unit: box-task-5146.service
/home/yellowtent/platformdata/logs/box.log.1:64470:2023-12-30T22:00:00.104Z box:tasks startTask: task 5146 took too long. killing
/home/yellowtent/platformdata/logs/box.log.1:64471:2023-12-30T22:00:00.104Z box:tasks stopTask: stopping task 5146
/home/yellowtent/platformdata/logs/box.log.1:64472:2023-12-30T22:00:00.105Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh 5146
/home/yellowtent/platformdata/logs/box.log.1:64478:2023-12-30T22:00:00.141Z box:shell stopTask (stdout): box-task-5146 stopped
/home/yellowtent/platformdata/logs/box.log.1:64480:2023-12-30T22:00:00.146Z box:shell startTask (stdout): Service box-task-5146 finished with exit code 0
/home/yellowtent/platformdata/logs/box.log.1:64482:2023-12-30T22:00:00.147Z box:tasks startTask: 5146 completed with code 0
/home/yellowtent/platformdata/logs/box.log.1:64483:2023-12-30T22:00:00.148Z box:tasks setCompleted - 5146: {"error":{"message":"Task 5146 timed out","code":"timeout"}}
/home/yellowtent/platformdata/logs/box.log.1:64484:2023-12-30T22:00:00.148Z box:tasks update 5146: {"percent":100,"error":{"message":"Task 5146 timed out","code":"timeout"}}
/home/yellowtent/platformdata/logs/box.log.1:64486:2023-12-30T22:00:00.151Z box:tasks startTask: 5146 done. error: { message: 'Task 5146 timed out', code: 'timeout' }

---

So yes, 5154 was killed correctly but there was another backup task running with the id 5154 which was not visible on the dashboard, which I then killed via systemctl

[girish]

Sounds about right. I think the confusion here is probably because the dashboard was not "refreshed". So, when you killed a task, it's probably killing the old task since we get the task id only when you refresh the dashboard. I know, confusing...

[BrutalBirdie]

@girish
Hmmm, I made sure I refreshed the dashboard multiple times, even with no cache and extra check with new login in incognito mode.
I also checked the running box-task-5154.service PID 1759591 if it had any open file descriptors in /proc/1759591/fd/ which revealed nothing happening for more then 10 minutes. After that I killed it.

odd, but this happened the first time so I am OK to write it off as a "ghost in the shell" bug

[girish]

Fair enough.. let us know if it resurfaces and we should investigate.