Restoring Backups fails

[p44]

@philkunz please check old posts on this forum about this specific topic.

Also, it could be useful to know more about your setup...

[philkunz]

What old topics? Seems to be specific to one app install. Log shows

Oct 21 15:28:15 box:backupformat/tgz tarExtract: ./data/content/themes/tripoli/assets/sass 0 directory to /home/yellowtent/appsdata/988fb337-3525-4e58-b7cb-f98c569d119d/data/content/themes/tripoli/assets/sass
Oct 21 15:28:15 box:backupformat/tgz tarExtract: ./data/content/themes/tripoli/assets/css/style-min-rtl.css 70801 file to /home/yellowtent/appsdata/988fb337-3525-4e58-b7cb-f98c569d119d/data/content/themes/tripoli/assets/css/style-min-rtl.css
Oct 21 15:28:15 box:backupformat/tgz tarExtract: ./data/content/themes/tripoli/assets/css/style-min.css 70777 file to /home/yellowtent/appsdata/988fb337-3525-4e58-b7cb-f98c569d119d/data/content/themes/tripoli/assets/css/style-min.css
Oct 21 15:28:15 box:backupformat/tgz tarExtract: ./data/content/themes/tripoli/assets/js/app.js 3150 file to /home/yellowtent/appsdata/988fb337-3525-4e58-b7cb-f98c569d119d/data/content/themes/tripoli/assets/js/app.js
Oct 21 15:28:15 box:backupformat/tgz tarExtract: ./data/content/themes/tripoli/assets/js/app.min.js 182622 file to /home/yellowtent/appsdata/988fb337-3525-4e58-b7cb-f98c569d119d/data/content/themes/tripoli/assets/js/app.min.js
Oct 21 15:28:15 box:apptask run: app error for state pending_restore: BoxError: tarExtract pipeline error: Invalid password or tampered file (mac mismatch) at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/yellowtent/box/src/backupformat/tgz.js:248:9 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5) at async download (/home/yellowtent/box/src/backuptask.js:104:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5) at async install (/home/yellowtent/box/src/apptask.js:368:9) { reason: 'External Error', details: {} }
Oct 21 15:28:15 box:taskworker Task took 119.385 seconds
Oct 21 15:28:15 box:tasks setCompleted - 13773: {"result":null,"error":{"stack":"BoxError: tarExtract pipeline error: Invalid password or tampered file (mac mismatch)\n at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/backupformat/tgz.js:248:9\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5)\n at async download (/home/yellowtent/box/src/backuptask.js:104:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"External Error","details":{},"message":"tarExtract pipeline error: Invalid password or tampered file (mac mismatch)"}}
Oct 21 15:28:15 box:tasks update 13773: {"percent":100,"result":null,"error":{"stack":"BoxError: tarExtract pipeline error: Invalid password or tampered file (mac mismatch)\n at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/backupformat/tgz.js:248:9\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5)\n at async download (/home/yellowtent/box/src/backuptask.js:104:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"External Error","details":{},"message":"tarExtract pipeline error: Invalid password or tampered file (mac mismatch)"}}
Oct 21 15:28:15 

[philkunz]

Backups should be tested in the background. It is a fail safe that needs to work. If there is a cloudron error across multiple backups of a certain app, that is a cloudron error. Even if the app is corrupted, the backup should still restore the corrupted app. And it for sure should not delete the app before confirming that the restore works!

[philkunz]

Seems like other people had the same problem before. For me everything runs on Zen 4c infra with ecc, zfs... It for sure is not hardware related. Backups worked until a few days ago. Something must have happened on cloudron's die, weird race condition, something. This needs checking, and backups need to be tested every single time automatically by cloudron to find the pattern and cause. It can't be that a tar is corrupted. If the source is corrupted, the tar should still be ok by itself.

[p44]

@philkunz yes, other users had different problems restoring backups (me too), that’s why I suggested to check other topics https://forum.cloudron.io/search?term=tarExtract&in=titlesposts.

Even because It seems that the source cause, in most scenarios, has not been identified, but in 9.x version of Cloudron situation it could be different and more reliable because of different backup health checks.

It could be interesting to know more about your setup so feel free to share here details useful for further analisys.

[joseph]

It looks like decryption is failing. Did you have happen to change the encryption password in between the time backup was made and now?

[philkunz]

no. Also file names are encrypted... Why do I see fileNames then before it aborts?

[philkunz]

Also other apps work. Encryption is a global setting isn't it? All backups since 17th October are bricked for this app.

[philkunz]

Also: When downloading a backup file through the UI: Why is the download not named after the appname and timestamp of the backup, but gernically app-backup + the timestamp of the download process? Why is the cloudron cli required for decrypting backups? It should be standard and documented. Am I missing something?

[philkunz]

Next problem: Trying to change the encryption password fails with this message:

[philkunz]

How buggy can something be? Yes... Clicking the Remount Storage button meanwhile works without any error?

[philkunz]

unmounting works after system reboot: To reiterate: The backup that failed, was a backup I created manually 10 minutes earlier. Then I tried earlier backups and I had to go back to 16th October to have a working backup.

[philkunz]

As a consequence I think, two things should be changed:

1. Create a working container first before deleting the old one.
2. Verify backups automatically after creating them.

[philkunz]

Otherwise this can end in a true shit show if someone relies on backups that are not working.

[james]

Hello @philkunz
Really sorry to read that you had such an unpleasant experience with the backup and restore process.
In Cloudron 9 the whole backup and restore process was overhauled with more integrity checks which should also minimize some of the issue you had here.

@philkunz said in Restoring Backups fails:

Why is the cloudron cli required for decrypting backups? It should be standard and documented. Am I missing something?

Here is the documentation: https://docs.cloudron.io/guides/decrypt-backups/

Can you maybe download the older backup that did not work and try to decrypt it locally?

---

Regarding the unmounting exists mount and remounting.
This can also be an issue related to your backup provider.

@philkunz said in Restoring Backups fails:

Clicking the Remount Storage button meanwhile works without any error?

Would have been good to see what the box.log wrote when this happened.

@philkunz said in Restoring Backups fails:

Create a working container first before deleting the old one.

I understand your sentiment, in the meanwhile, if you want to be extra careful, you can clone the app from a backup to ensure everything is working.

@philkunz said in Restoring Backups fails:

Verify backups automatically after creating them.

This is done in Cloudron 9

[necrevistonnezr]

@philkunz said in Restoring Backups fails:

Seems like other people had the same problem before. For me everything runs on Zen 4c infra with ecc, zfs...

Is ZFS supported now?

[philkunz]

zfs is on the host. cloudron uses non-cow inside the vm.