backup failed - 3 times in one week period

[humptydumpty]

Backups are failing more often lately. I used to see a fail once a month, bi-weekly, and now it's happened three times in a one week period.

May 27 00:22:08 box:tasks update 12699: {"percent":52.51515151515152,"message":"Uploading backup 691M@1MBps
May 27 00:22:16 box:shell backup-snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6 tgz {"localRoot":"/home/yellowtent/appsdata/a52611d6-c20e-4f40-a1cb-74b94e58eda6","layout":[]} errored BoxError: backup-snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6 exited with code 1 signal null
May 27 00:22:16 box:tasks setCompleted - 12699: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
May 27 00:22:16 box:taskworker Task took 1328.503 seconds
May 27 00:22:16 box:tasks update 12699: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
[no timestamp]  Backuptask crashed
[no timestamp]  }

The app that was being backed up is Freescout with a size of 700 MB. If this is a network issue, how can I tell if it's caused by my VPS (contabo) or backup provider (backblaze)? What should I do?

[subven]

22 minutes backup upload for a 700mb app seems too much even for a known weak performing hoster like Contabo. Upload speed of 1MBps also seems way too low. Does your server have enough RAM to perform the backup task? The error is just an box backup task crash.

Have you benchmarked the server? Is your load, RAM and network speed fine? My guess is that your server lacks ressources or is under high load.

If it's just weak performance from Contabo I would recommend to move to known good hoster like Hetzner or Netcup.

[humptydumpty]

@subven Yes, I have more than enough resources. 8 vCPUs 24GB 32GB RAM and barely use 1/3 of them. I've been eyeing Hetzner, but it's been on the back burner for a while now. I'll try to get it done during the next CR 8 upgrade. Thanks for confirming it's a VPS issue. For now, I'll shoot them an angry message

Edit: It looks like Contabo has changed their plans. I'm on the 32GB RAM plan and they've lowered the new plan to 24GB. I wonder if this is their way of kicking me off of their grandfathered plan

[subven]

@humptydumpty said in backup failed - 3 times in one week period:

8 vCPUs 24GB 32GB RAM and barely use 1/3 of them

Check your network speeds and if you really can use the CPU performance you're paying for. Maybe you have "noisy neighbors" on your VPS that constantly use up all ressources.

[humptydumpty]

@subven I've had issues with Contabo before (slow disk speeds, network issues, etc) but it wasn't this bad and contacting support got me nowhere. I'm definitely not getting what I'm paying for, so I'm spending the day reading up on server resources and comparing that to my actual usage. I'm debating which Hetzner plan(s) to get:

• one beefy or two small VPS'
• shared or dedicated vCPU
• whether to self-host internal-use apps on-premise at work

This whole project has been on hold waiting on CR 8 (multiserver support) and how that's going to work out. It doesn't look like I can delay this any longer though if daily backups keep failing.

I ran into this blog post which seems to suggest I might be overpaying for actual usage: https://runcloud.io/blog/scaling-ram-cpu

BTW, my Contabo disk speeds are pathetic. I'm on their NVME plan and it's slower than a standard 2.5" SSD. This is from Cloudron's system info page:

I dug up some old posts of mine (this and this) about the slow disk speeds and just noticed it's been a year since then. Time sure flies!