CR 8.0.1 - backup failed (home server & backblaze)

[humptydumpty]

Backup failed again and Contabo is out of the picture so they're not to blame.

The full logs show speeds of 1-5 Mbps.

Jul 22 21:03:21 box:backupformat/tgz Attempt 1 failed. Will retry: Size mismatch
Jul 22 21:03:21 box:backupformat/tgz Error: Size mismatch
Jul 22 21:03:21 box:shell backup-snapshot/app_d78b4352-5fca-4c45-8c66-fe7839bd455f: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_d78b4352-5fca-4c45-8c66-fe7839bd455f tgz {"localRoot":"/home/yellowtent/appsdata/d78b4352-5fca-4c45-8c66-fe7839bd455f","layout":[]} errored BoxError: backup-snapshot/app_d78b4352-5fca-4c45-8c66-fe7839bd455f exited with code 1 signal null
Jul 22 21:03:21 box:tasks setCompleted - 149: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Jul 22 21:03:21 box:tasks update 149: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Jul 22 21:03:21 box:taskworker Task took 201.26 seconds
[no timestamp]  Backuptask crashed
[no timestamp]  Size mismatch
[no timestamp]  v20.12.2
Dec 31 18:09:24 node:internal/process/promises:289
[no timestamp]  }

• What does size mismatch mean?
• Does this have anything to do with the advanced backup settings (upload part size & memory limits)?

TIA!

[girish]

Yes, there is a further fix in 8.0.2 for this. Can you check in that? Latest release is 8.0.3 by now. We will make it stable sometime next week.

[nebulon]

This is a newly introduced check for the tarball creation. This means that a file has changed on disk while the backup tarball is being created. This would result in either a broken tarball or simply stalling the backup process. @girish has a fix for this for next release until that this just means that you have to retry until the backup runs at a time where files don't change so frequently behind its back.

[humptydumpty]

@nebulon Thanks for the quick response. I read something about a backup fix but I thought it was applied in 8.0.1. No worries. I just wanted to report the issue in case it's a different one. Thanks again.

[girish]

Yes, there is a further fix in 8.0.2 for this. Can you check in that? Latest release is 8.0.3 by now. We will make it stable sometime next week.

[humptydumpty]

I updated all servers to 8.0.3. I'll report back if it happens again. Thanks Girish!

[humptydumpty]

@girish Hetzner VPS backing up to Backblaze S3 failed on CR v8.0.3 (Ubuntu 24.04 LTS). Is it a network hiccup or a bug?

2024-08-16T05:15:33.545Z box:tasks update 13536: {"percent":51,"message":"Uploading backup 156M@1MBps (support.mydomain.com)"}
2024-08-16T05:15:35.861Z box:storage/s3 Upload progress: {"loaded":125829120,"part":11,"key":"snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6.tar.gz.enc"}
node:events:496
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
Emitted 'error' event on TLSSocket instance at:
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

Node.js v20.12.2
2024-08-16T05:15:35.883Z box:shell backup-snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6 tgz {"localRoot":"/home/yellowtent/appsdata/a52611d6-c20e-4f40-a1cb-74b94e58eda6","layout":[]} errored BoxError: backup-snapshot/app_a52611d6-c20e-4f40-a1cb-74b94e58eda6 exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess.emit (node:domain:488:12)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
  reason: 'Shell Error',
  details: {},
  code: 1,
  signal: null
}
2024-08-16T05:15:35.885Z box:taskworker Task took 935.316 seconds
2024-08-16T05:15:35.885Z box:tasks setCompleted - 13536: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n    at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
2024-08-16T05:15:35.885Z box:tasks update 13536: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n    at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
BoxError: Backuptask crashed
    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
    at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
    at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)

[matix131997]

I also have a similar problem. I have the same as you, Hetzner server and Backblaze. @girish has already made an improvement and it will be in the new version 8.0.4 next week.

But it's interesting, on the development server where I have an application running that I prepare in LAMP it runs a copy with no problem.

[humptydumpty]

@matix131997 Noted. Thanks for letting me know!

[humptydumpty]

@matix131997 said in CR 8.0.1 - backup failed (home server & backblaze):

application running that I prepare in LAMP it runs a copy with no problem.

Maybe the file size matters? I think my crashes have been happening with larger files mostly. The speed drops to 0-1 mbps after a while.

[matix131997]

@humptydumpty I also think. On the server where the copy is running the size of the application does not exceed 1GB, and on the production server I have Nextcloud where the size exceeds 100GB is not successful.

[humptydumpty]

@matix131997 My freescout app is the one that crashed I think and it's around 1GB only. I have backups disabled for my Nextcloud because I'm pretty sure it would never finish the job within a reasonable time even if it didn't crash.