Keep Getting Backuptask crashed

[Wenhardt]

Hi everyone,

Whenever I try to back up, I get a Backuptask crashed error. I tried increasing the memory limit and that didn't help. Very rarely, backups succeed. My Cloudron instance is v8.02, but this issue has been happening even before that.

Here is the log:

      ^
      throw er; // Unhandled 'error' event
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
    at ChildProcess.emit (node:domain:488:12)
    at ChildProcess.emit (node:events:518:28)
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
    at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
    at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)
    at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)
  code: 'EPIPE',
  code: 1,
  details: {},
  errno: -32,
  reason: 'Shell Error',
  signal: null
  syscall: 'write'

Has anyone encountered this issue and knows how to fix it?

Thanks!

[joseph]

@Wenhardt EPIPE usually indicates write to a close socket. The remote closed the connection when Cloudron was writing the backup to it. Which backend are you using? Have you , by any chance, tested other providers?

[Wenhardt]

Thanks joseph for the response! I am using backblaze. I've used it for years without issue. I haven't tested another provider though.

[joseph]

@Wenhardt mm, if this happens all the time, can you write to support@cloudron.io and we can debug it on your instance.

[nebulon]

We had another support request recently with socket issues and backblaze, there was no real resolution as backblaze just cancels the connection during upload. Maybe they have some issue currently.

[djxx]

I'm also getting this error message with a backblaze backup.

[Wenhardt]

Thanks for the responses. I'll look into other providers.

[matix131997]

I also had this problem a few weeks ago. During the migration from Contabo, creating a backup was so problematic that I started fiddling with the parameters.

It was only with the help of these parameters that it started to stabilise and now performs the backup correctly.

Storage Format: Tarball
Memory Limit: 4GB
Upload Part Size: 512 MiB

What was interesting was that after restoring from the backup on the new server, the settings reset. I only noticed when I had an error message related to the backup.

[joseph]

If one of you can reach out to support@cloudron.io, maybe we can debug on your server as to what the issue is.

[matix131997]

In my case, it works correctly to create a backup with the application above 100GB after applying the above settings that I sent.

[djxx]

@matix131997 - I noticed the same.

@joseph - in case it helps, I did a few backups (tarball) and noticed a pattern. The backup app configuration indicates:
Multi-part upload part size. Up to 3 parts are uploaded in parallel and requires as much memory.

But in practice I saw that the backup was taking 8x as much memory. I watched it do a backup with 512MB chunks and memory went all the way up to 4GB before the multiple backup threads "backed off" on memory usage, and dropped down to 1.5GB again.

On a 256 MB backup with 2 GB of memory set as the limit I saw it go just above 2 GB (2050 MB) and then the memory usage backed off.

On a 128 MB backup with 1 GB of memory set as the limit I saw it go just above 1 GB (1032 MB) and then the memory usage backed off.

I wonder if there was a change introduced recently that caused the memory usage to go from 3x to 8x, and servers that worked fine before are now undersized.

I've set my backup to 128 MB chunk size and 1 GB RAM and have run it twice now manually without issue. We'll see if the automatic job succeeds as well.

[timconsidine]

@Wenhardt try Scaleway - not perfect but works generally