Restoring Backups fails

[philkunz]

How buggy can something be? Yes... Clicking the Remount Storage button meanwhile works without any error?

[philkunz]

unmounting works after system reboot: To reiterate: The backup that failed, was a backup I created manually 10 minutes earlier. Then I tried earlier backups and I had to go back to 16th October to have a working backup.

[philkunz]

As a consequence I think, two things should be changed:

1. Create a working container first before deleting the old one.
2. Verify backups automatically after creating them.

[philkunz]

Otherwise this can end in a true shit show if someone relies on backups that are not working.

[james]

Hello @philkunz
Really sorry to read that you had such an unpleasant experience with the backup and restore process.
In Cloudron 9 the whole backup and restore process was overhauled with more integrity checks which should also minimize some of the issue you had here.

@philkunz said in Restoring Backups fails:

Why is the cloudron cli required for decrypting backups? It should be standard and documented. Am I missing something?

Here is the documentation: https://docs.cloudron.io/guides/decrypt-backups/

Can you maybe download the older backup that did not work and try to decrypt it locally?

---

Regarding the unmounting exists mount and remounting.
This can also be an issue related to your backup provider.

@philkunz said in Restoring Backups fails:

Clicking the Remount Storage button meanwhile works without any error?

Would have been good to see what the box.log wrote when this happened.

@philkunz said in Restoring Backups fails:

Create a working container first before deleting the old one.

I understand your sentiment, in the meanwhile, if you want to be extra careful, you can clone the app from a backup to ensure everything is working.

@philkunz said in Restoring Backups fails:

Verify backups automatically after creating them.

This is done in Cloudron 9

[necrevistonnezr]

@philkunz said in Restoring Backups fails:

Seems like other people had the same problem before. For me everything runs on Zen 4c infra with ecc, zfs...

Is ZFS supported now?

[philkunz]

zfs is on the host. cloudron uses non-cow inside the vm.

[dylightful]

Experiencing the same issue with CIFS mounts. Creating backups work seamlessly. When you go to restore it attempts to restore a small subset of files 3 times before displaying the error. I noticed when I go to restore the app it doesn't prompt for an encryption password, perhaps an internal issue with storing of the encryption hash?

[p44]

@dylightful Which provider are you using to backup files? What version of Cloudron are you using?

[dylightful]

@p44

CIFS Mount to storage provider - Version 9.0.13

Backups work fine, restoration gives the above error.

[joseph]

@dylightful the verification signature is now saved but there is no UI yet .

Can you check if you have a .backupinfo file next to your .tar.gz ? Inside .backupinfo there will be a sha256 checksum. Can you check if sha256 sum matches with the .tar.gz file ?

[p44]

@dylightfu That is the main problem.... the restoration, not the backup... Can you give us storage provider company name?

[dylightful]

@p44

It is to a self hosted SAN

[joseph]

@dylightful if there's a setup where we can debug, would be great since we are unable to reproduce it .

[p44]

@dylightful ok. it could be useful to follow @joseph advice, to try to reproduce issue...

I had the same problem with Hetzner Storage Box via CIFS mount...

[dylightful]

@p44 tried hetzner now too. Same issue - backups fine, unable to restore (20GB LAMP stack)

[p44]

@dylightful Please share logs or other infos with @joseph so we can finally understand what’s going on... this is a very important problem to understand ASAP. I had this problem too.

[dylightful]

This post is deleted!

[dylightful]

Log file for @joseph

It displays this error 3 times

Dec 16 00:01:24 box:backupformat/tgz Attempt 2 failed. Will retry: tarExtract pipeline error: invalid stored block lengths

After the 3rd time, it times out and fails with

Dec 16 00:01:49 box:apptask run: app error for state pending_restore: BoxError: tarExtract pipeline error: invalid stored block lengths at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:231:26) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async /home/yellowtent/box/src/backupformat/tgz.js:252:9 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:248:5) at async download (/home/yellowtent/box/src/backuptask.js:129:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:169:5) at async installCommand (/home/yellowtent/box/src/apptask.js:350:9) { reason: 'External Error', details: {} }
Dec 16 00:01:49 box:tasks setCompleted - 21001: {"result":null,"error":{"message":"tarExtract pipeline error: invalid stored block lengths","reason":"External Error"},"percent":100}
Dec 16 00:01:49 box:tasks updating task 21001 with: {"completed":true,"result":null,"error":{"message":"tarExtract pipeline error: invalid stored block lengths","reason":"External Error"},"percent":100}
Dec 16 00:01:49 box:taskworker Task took 57.836 seconds
Dec 16 00:01:49 BoxError: tarExtract pipeline error: invalid stored block lengths
Dec 16 00:01:49 at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:231:26)
Dec 16 00:01:49 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Dec 16 00:01:49 at async /home/yellowtent/box/src/backupformat/tgz.js:252:9
Dec 16 00:01:49 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)
Dec 16 00:01:49 at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:248:5)
Dec 16 00:01:49 at async download (/home/yellowtent/box/src/backuptask.js:129:5)
Dec 16 00:01:49 at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:169:5)
Dec 16 00:01:49 at async installCommand (/home/yellowtent/box/src/apptask.js:350:9)
Dec 16 00:01:49 Exiting with code 0

[girish]

@dylightful If we can have access to your setup, can you write to me at support@cloudron.io ? Would love to debug and figure out why it's failing . Is this reproducible easily - meaning if you take a backup and restore, it fails?