Backuptask crashing after 8.0 update

[nichu42]

As mentioned in another thread, the backup on my VPS failed because the CFS backup storage was not mounted after upgrading to v8.0.
It now is, but backups still fail:

  "taskId": "5403",
  "errorMessage": "Backuptask crashed",
  "timedOut": false,
  "backupId": null

I have tried three times, and also rebooted the machine in between. However, the problem persists.

[girish]

What is the crash in the backup logs? The logs are above in the top right of the Backup view .

[nichu42]

@girish

Ah, sorry, I was looking for them but couldn't find them.

[no timestamp]  at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:110:19)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)
[no timestamp]  at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)
[no timestamp]  at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[no timestamp]  at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)
[no timestamp]  code: null,
[no timestamp]  details: {},
[no timestamp]  reason: 'Shell Error',
[no timestamp]  signal: 'SIGABRT'
Jun 27 13:39:52 box:tasks update 5403: {"percent":76,"message":"Uploading backup 3059M@0MBps (blueplanet.social)"}
...
Jun 27 13:53:22 box:tasks update 5403: {"percent":76,"message":"Uploading backup 3061M@0MBps (blueplanet.social)"}
Jun 27 13:53:29 box:shell backup-snapshot/app_c5d30482-9837-4fdb-8c21-04ef653d691c: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_c5d30482-9837-4fdb-8c21-04ef653d691c tgz {"localRoot":"/home/yellowtent/appsdata/c5d30482-9837-4fdb-8c21-04ef653d691c","layout":[]} errored BoxError: backup-snapshot/app_c5d30482-9837-4fdb-8c21-04ef653d691c exited with code null signal SIGABRT
Jun 27 13:53:29 box:taskworker Task took 3219.734 seconds
Jun 27 13:53:29 box:tasks setCompleted - 5403: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Jun 27 13:53:29 box:tasks update 5403: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
[no timestamp]  Backuptask crashed
[no timestamp]  }

[girish]

Never seen an SIGABRT before. Do you see anything in dmesg in the server? Are you also able to normally access the CIFS volume (try copying some files) on the host?

[nichu42]

@girish There are lots of entries in dmesg, but I have no clue what exactly I am looking for.
Yes, I am able to access the CIFS volume. And the backup task actually has uploaded files on every run. It just seems that it could not finish the task.
I am used to the backup stalling (going down to 0 kbit/s), but before the 8.0 upgrade, it was eventually finishing what it had begun.

[nichu42]

I switched from CIFS to SSHFS to check if it makes a difference. Unfortunately, the backup task crashes, too.

[nebulon]

@nichu42 so in both CIFS and SSHFS you consistently get some SIGABRT? Maybe we have to then take a direct look at your instance. If you want, please send an email to support@cloudron.io and enable remote SSH support for us.

[nebulon]

So we have found at least one issue with Ubuntu 24 which may be related to the backup issue. Collectd also crashes with SIGABRT due to a wrong configuration on Ubuntu 24. The fix is https://docs.cloudron.io/guides/upgrade-ubuntu-24/#post-upgrade

In this particular case now the typical backup upload stalling issue seems to happen, so we are still working on this.

[nichu42]

@nebulon
Thanks for looking into this.
Please note that I upgraded to Ubtunu 24.04 after the backup had already crashed for 3 consecutive days. So whatever the reason is was already there with Ubuntu 22.04.

[nichu42]

I did a backup to the local file system before deploying today's Mastodon update. The backup went through without any issues. So I suppose the crash is only related to the upload part (CIFS or SSHFS) of the backup process. I guess you already knew and the result was to be expected, but I'm sharing it anyway just in case.

[girish]

We have a performance fix coming for SSHFS atleast. It cuts down the backup rotation for 3-4 hours to a minute or so. Should be in 8.0.1. But this is still not addressing the issue with your crash though. We are still debugging this.

[nichu42]

Hi @girish - whatever you did with SSHFS, I am happy to report that after upgrading to 8.0.1, I was able to run a full backup incl. Mastodon for the 2nd consecutive day now.

[girish]

Unfortunately, we ended up breaking https://forum.cloudron.io/topic/11993/backuptask-crashing-after-8-0-update . But yeah, overall the fix is good.