tarExtract pipeline error: Invalid tar header

[nebulon]

Do you get this error when restoring a newly made backup now without encryption? If so how large is the resulting tarball? I am not aware of any filesystem based problem with CIFS around such fundamental things as block sizes, but this is probably more related to the tar and not the filesystem unless for some reason the CIFS mount gets corrupt.

[nottheend]

Yes, I got the error mentioned before on a newly created backup without encryption. The tarball was around 13 GB.

I was only able to restore backups older than 1 month for the nextcloud app. For all other 12+ apps restoring did work like a charme for the most recent backup (younger than 1 month).

It gives me the impression that the CIFS with Hetzner Storagebox is a risky choice.
Therefore, a hint in the docs might be saving some lives of other folks.

I would like to offer 2 additional feedbacks beyond the CIFS issue and the docs:

1. it seems to be not possible to use the decryption command line cloudron command with a password that includes special characters. I tried to escape with " but it didn't work. Probably sth to document. Maybe there is a way to escape after all.
2. Is there any way to do backup validation? It is probably concerning to see successful backups while they are corrupt.

And thanks a lot for the support @girish and @nebulon for pointing me in the right direction! I resolved it by choosing a different cloud provider and different storage protocol. I used S3 now. Backing up and restoring worked with encryption completely fine!
Thank you!

[girish]

@nottheend said in tarExtract pipeline error: Invalid tar header:

Is there any way to do backup validation? It is probably concerning to see successful backups while they are corrupt.

This is in the immediate roadmap (https://forum.cloudron.io/topic/9975/what-s-coming-in-8-1 ). The "backup integrity". Ideally, we have to store information in the storage and also provide a mechanism to "validate" the integrity from the UI.

[jdaviescoates]

@nottheend said in tarExtract pipeline error: Invalid tar header:

It gives me the impression that the CIFS with Hetzner Storagebox is a risky choice.

I've been using it pretty much without issue for a good couple of years now, including doing a couple of complete restores.

But: I don't use encryption. I've always taken the veiw that issues relating to having troubls decrypting - eg forgetting passphrase, some corruption etc etx - are a bigger risk for me losing data than someone somehow gaining access to my Storage Boxes, in part because I don't have particularly sensitive data stored.

[potemkin_ai]

Hetzner's box, SSHFS with encryption - recovery fails with:

2024-10-19T13:37:34.456Z box:provision setProgress: restore - Downloading backup /mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc
2024-10-19T13:37:34.456Z box:storage/filesystem download: /mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc
2024-10-19T13:37:34.547Z box:backupformat/tgz Attempt 2 failed. Will retry: tarExtract pipeline error: incorrect header check

Checking the file:

file /mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc
/mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc: data

Guess encrypted data can't have tar headers?

[potemkin_ai]

in my specific case - that was due to decryption key error

[p44]

Hello everyone,

restoring Wordpress Developer instance, I had this problem right now:

May 09 13:54:41 box:backupformat/tgz tarExtract: ./data/public/wp-content/uploads/2015/02/filename.jpg 73444 file to /home/yellowtent/appsdata/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/data/public/wp-content/uploads/2015/02/filename.jpg
May 09 13:54:41 box:apptask run: app error for state pending_restore: BoxError: tarExtract pipeline error: invalid stored block lengths at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/yellowtent/box/src/backupformat/tgz.js:248:9 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5) at async download (/home/yellowtent/box/src/backuptask.js:104:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5) at async install (/home/yellowtent/box/src/apptask.js:368:9) { reason: 'External Error', details: {} }
May 09 13:54:41 box:tasks setCompleted - 681: {"result":null,"error":{"stack":"BoxError: tarExtract pipeline error: invalid stored block lengths\n at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/backupformat/tgz.js:248:9\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5)\n at async download (/home/yellowtent/box/src/backuptask.js:104:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"External Error","details":{},"message":"tarExtract pipeline error: invalid stored block lengths"}}
May 09 13:54:41 box:tasks update 681: {"percent":100,"result":null,"error":{"stack":"BoxError: tarExtract pipeline error: invalid stored block lengths\n at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/backupformat/tgz.js:248:9\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5)\n at async download (/home/yellowtent/box/src/backuptask.js:104:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"External Error","details":{},"message":"tarExtract pipeline error: invalid stored block lengths"}}
May 09 13:54:41 box:taskworker Task took 335.441 seconds
May 09 13:54:41 BoxError: tarExtract pipeline error: invalid stored block lengths

Many others file was extracted... but then stopped with this error:

BoxError: tarExtract pipeline error: invalid stored block lengths at tarExtract

and

External Error: tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?

Edit:

I confirm that is something Hetzner Box related (Via CIFS).
I changed to Backblaze and everything is fine.

[nottheend]

Hopefully it will not appear with any storage anymore when this is implemented:

(9.0) Backup integrity - store size and checksum of backups. Also provide a way to "verify" backup integrity in the remote.

Mentioned in
what's coming in cloudron 9

[p44]

@nottheend ok thanks a lot

[p44]

Again, handling a "tarExtract pipeline error: incorrect header check" issue... on a different backup... very very sad for this... I hope you can release soon a solution because is risky to store backups that then cannot be restored @girish

[girish]

@p44 for encrypted backups, not sure what is the expected behaviour. If the backup is corrupt or password is forgotten, one cannot recover. What do you have in mind? What do you think Cloudron could have done? (or was your note about adding periodic integrity checks?)

[p44]

@girish

Periodic integrity checks – for example, verifying the backup archive and reporting issues proactively – could help discover corruption early, before a restore is needed. This would give admins a chance to re-run or fix backups in time.

What do you think about such a feature?