tarExtract pipeline error: Invalid tar header

[nottheend]

After trying to restore cloudron on a new server, I am getting this error while attempting to restore nextcloud:

[no timestamp]  tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?

It worries me, that there is no timestamp.

Until now I tried to

1. reboot the server (multiple times)
2. restore older backups (not only the one from directly before)

Further logs before the log above:

Oct 11 20:09:04 - [POST] /clear
Oct 11 20:12:35 box:backupformat/tgz tarExtract: ./data/appdata_ocofjozos9pc/preview/2/0/9/d/1/9/5/83606/4096-4096-max.png 98592 file to /home/yellowtent/appsdata/39b0914d-1add-4a32-966f-d331fb91f589/data/appdata_ocofjozos9pc/preview/2/0/9/d/1/9/5/83606/4096-4096-max.png

[...]

Oct 11 20:12:35 box:apptask run: app error for state pending_restore: BoxError: tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped? at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/yellowtent/box/src/backupformat/tgz.js:248:9 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5) at async download (/home/yellowtent/box/src/backuptask.js:100:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5) at async install (/home/yellowtent/box/src/apptask.js:368:9) { reason: 'External Error', details: {} }
Oct 11 20:12:35 box:tasks setCompleted - 2703: {"result":null,"error":{"stack":"BoxError: tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?\n at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/backupformat/tgz.js:248:9\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5)\n at async download (/home/yellowtent/box/src/backuptask.js:100:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"External Error","details":{},"message":"tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?"}}
Oct 11 20:12:35 box:tasks update 2703: {"percent":100,"result":null,"error":{"stack":"BoxError: tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?\n at tarExtract (/home/yellowtent/box/src/backupformat/tgz.js:225:26)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/backupformat/tgz.js:248:9\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.download (/home/yellowtent/box/src/backupformat/tgz.js:244:5)\n at async download (/home/yellowtent/box/src/backuptask.js:100:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"External Error","details":{},"message":"tarExtract pipeline error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?"}}
Oct 11 20:12:35 box:taskworker Task took 241.42 seconds
[no timestamp]  2024-10-11T17:53:16Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:C 11 Oct 2024 17:53:16.518 * Configuration loaded
[no timestamp]  2024-10-11T17:53:16Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:C 11 Oct 2024 17:53:16.518 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=12, just started
[no timestamp]  2024-10-11T17:53:16Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:53:16.565 * RDB memory usage when created 0.90 Mb
[no timestamp]  2024-10-11T17:53:16Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:53:16.566 * Done loading RDB, keys loaded: 0, keys expired: 0.
[no timestamp]  2024-10-11T17:53:16Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:53:16.566 * Ready to accept connections tcp
[no timestamp]  2024-10-11T17:53:18Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:53:18,146 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[no timestamp]  2024-10-11T17:55:07Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:55:07.306 * Saving the final RDB snapshot before exiting.
[no timestamp]  2024-10-11T17:55:07Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:55:07.308 * DB saved on disk
[no timestamp]  2024-10-11T17:55:07Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:55:07.308 * Removing the pid file.
[no timestamp]  2024-10-11T17:55:07Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:M 11 Oct 2024 17:55:07.309 # Redis is now ready to exit, bye bye...
[no timestamp]  2024-10-11T17:55:07Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 12:signal-handler (1728669307) Received SIGTERM scheduling shutdown...
[no timestamp]  2024-10-11T17:55:07Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1010 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:55:07,274 INFO waiting for redis, redis-service to die
[no timestamp]  2024-10-11T17:58:47Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:58:47,457 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
[no timestamp]  2024-10-11T17:58:47Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:58:47,457 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
[no timestamp]  2024-10-11T17:58:47Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:58:47,482 CRIT Server 'unix_http_server' running without any HTTP authentication checking
[no timestamp]  2024-10-11T17:58:47Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:58:47,483 INFO supervisord started with pid 1
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:C 11 Oct 2024 17:58:48.694 * Configuration loaded
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:C 11 Oct 2024 17:58:48.694 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=13, just started
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 17:58:48.778 * RDB age 221 seconds
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 17:58:48.778 * RDB memory usage when created 0.83 Mb
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 17:58:48.779 * DB loaded from disk: 0.002 seconds
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 17:58:48.779 * Done loading RDB, keys loaded: 0, keys expired: 0.
[no timestamp]  2024-10-11T17:58:48Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 17:58:48.779 * Ready to accept connections tcp
[no timestamp]  2024-10-11T17:58:50Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 17:58:50,463 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[no timestamp]  2024-10-11T18:07:05Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:05.456 * Saving the final RDB snapshot before exiting.
[no timestamp]  2024-10-11T18:07:05Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:05.458 * Removing the pid file.
[no timestamp]  2024-10-11T18:07:05Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:05.459 # Redis is now ready to exit, bye bye...
[no timestamp]  2024-10-11T18:07:05Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:signal-handler (1728670025) Received SIGTERM scheduling shutdown...
[no timestamp]  2024-10-11T18:07:05Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1009 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 18:07:05,461 INFO stopped: redis (exit status 0)
[no timestamp]  2024-10-11T18:07:33Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 18:07:33,814 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
[no timestamp]  2024-10-11T18:07:33Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 18:07:33,832 INFO RPC interface 'supervisor' initialized
[no timestamp]  2024-10-11T18:07:33Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 18:07:33,833 CRIT Server 'unix_http_server' running without any HTTP authentication checking
[no timestamp]  2024-10-11T18:07:33Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 2024-10-11 18:07:33,833 INFO supervisord started with pid 1
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:C 11 Oct 2024 18:07:34.906 * Configuration loaded
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.909 * monotonic clock: POSIX clock_gettime
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.921 # Failed to write PID file: Permission denied
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.921 * Running mode=standalone, port=6379.
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.922 * Server initialized
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.927 * DB loaded from disk: 0.006 seconds
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.927 * Loading RDB produced by version 7.2.1
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.927 * RDB age 29 seconds
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.927 * RDB memory usage when created 0.90 Mb
[no timestamp]  2024-10-11T18:07:34Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32-966f-d331fb91f589 - 13:M 11 Oct 2024 18:07:34.927 * Ready to accept connections tcp
[no timestamp]  2024-10-11T18:07:35Z username redis-39b0914d-1add-4a32-966f-d331fb91f589 1011 redis-39b0914d-1add-4a32

[nottheend]

All other apps apart from one were successfully restored.

Further, I discovered that the general backup section of cloudron contains this message:

Task was stopped because the server was restarted or crashed

Could be related to a server reboot. I didn't verify the completion of the backup process before rebooting the server.
But if this backup is corrupt, would I need to restore the complete cloudron?

[girish]

@nottheend said in tarExtract pipeline error: Invalid tar header:

It worries me, that there is no timestamp.

I have seen this as well. I think this is some bug in our log viewer. Will investigate.

Task was stopped because the server was restarted or crashed

This can be ignored but will get the message fixed.

But if this backup is corrupt, would I need to restore the complete cloudron?

I don't think the backup is corrupt (atleast based on the two bugs above). Otherwise, nothing will restore. In what way was the one app not restored? Did it give some error message?

[nottheend]

Thanks for your input!

It didn't restore in the sense that the nextcloud app was restoring - as all other Cloudron apps - but ended up with the error message:

Error : External Error - tarExtract pipeline error: incorrect header check

The status of the nextcloud app is "Error".
I tried to restore again the whole cloudron server as well as deinstalling the nextcloud app and import only the backed up version. Neither of them did work.

It sounds to me that the password is wrong or that the algorithm for decrypting would be wrong.

The backuped file is on the server. Will try to decrypt it manually. The nextcloud backup has 12 GB of size, the server has more than 100 GB free

[nottheend]

Downloaded the backuped file from the nextcloud app and tried to encrypt them with commands like this, tried also other algorithms:

openssl enc -d -aes-128-cbc -in app_nextclouddomain.com_v4.22.5.tar.gz.enc -out file.tar.gz

No luck. Any other idea to be able to restore the Nextcloud app?
If the encryption password of the backup weren't correct, the other apps wouldn't be restored successfully too.
Thoughts:

1. Is there any possibility that the passwords for encryption can be different per app?
2. Could a removed volume be an issue? I removed a volume recently before the backup but the app should have been consistent during the backup time

[nebulon]

We have a guide in the docs on how to decrypt backups in case you haven't seen that https://docs.cloudron.io/guides/decrypt-backups/

[nottheend]

Thanks, I was not aware.
When I try to decrypt, it doesn't work, although I use the same password like for the other apps:

Error: Could not decrypt: Invalid password or tampered file (mac mismatch)

I am confused now, because I am not aware of using another password for encrypting a single app (nextcloud) compared to other apps used on the same Cloudron instance.

Is there any other idea?

[nebulon]

If you are sure the password is correct, which it would be if its the same where other backups could be decrypted, then this very much looks like the backup file is corrupt maybe redownload it from the original storage just to rule out a simple downloading error, also if you have multiple backup files maybe try one older one

[nottheend]

Thanks. Luckily I discovered that I didn't delete the old server yet. I spinned it up again and took another backup - the new backup seems to be corrupt again
Not sure what I am missing

[nebulon]

To rule out one step, maybe disable backup encryption and then see if it produces a valid tarball for a start

[nottheend]

Thanks, I am getting a different error:

An error occurred during the import operation: External Error: tarExtract pipeline error: invalid stored block lengths

Is there any limitation in terms of CIFS?

[nottheend]

I just found this hint in the docs:

Hetzner Storage Box

We recommend using SSHFS for Hetzner Storage Box since it is much faster and efficient storage wise compared to CIFS. When using Hetzner Storage Box with CIFS, the Remote Directory is /backup for the main account. For sub accounts, the Remote Directory is /subaccount.

Probably worth expanding to hint that there seem to be some unrealiable component or interaction with this way the storage box is used by Cloudron.

The issue in this thread is related to CIFS with Hetzner Storage box

[nebulon]

Do you get this error when restoring a newly made backup now without encryption? If so how large is the resulting tarball? I am not aware of any filesystem based problem with CIFS around such fundamental things as block sizes, but this is probably more related to the tar and not the filesystem unless for some reason the CIFS mount gets corrupt.

[nottheend]

Yes, I got the error mentioned before on a newly created backup without encryption. The tarball was around 13 GB.

I was only able to restore backups older than 1 month for the nextcloud app. For all other 12+ apps restoring did work like a charme for the most recent backup (younger than 1 month).

It gives me the impression that the CIFS with Hetzner Storagebox is a risky choice.
Therefore, a hint in the docs might be saving some lives of other folks.

I would like to offer 2 additional feedbacks beyond the CIFS issue and the docs:

1. it seems to be not possible to use the decryption command line cloudron command with a password that includes special characters. I tried to escape with " but it didn't work. Probably sth to document. Maybe there is a way to escape after all.
2. Is there any way to do backup validation? It is probably concerning to see successful backups while they are corrupt.

And thanks a lot for the support @girish and @nebulon for pointing me in the right direction! I resolved it by choosing a different cloud provider and different storage protocol. I used S3 now. Backing up and restoring worked with encryption completely fine!
Thank you!

[girish]

@nottheend said in tarExtract pipeline error: Invalid tar header:

Is there any way to do backup validation? It is probably concerning to see successful backups while they are corrupt.

This is in the immediate roadmap (https://forum.cloudron.io/topic/9975/what-s-coming-in-8-1 ). The "backup integrity". Ideally, we have to store information in the storage and also provide a mechanism to "validate" the integrity from the UI.

[jdaviescoates]

@nottheend said in tarExtract pipeline error: Invalid tar header:

It gives me the impression that the CIFS with Hetzner Storagebox is a risky choice.

I've been using it pretty much without issue for a good couple of years now, including doing a couple of complete restores.

But: I don't use encryption. I've always taken the veiw that issues relating to having troubls decrypting - eg forgetting passphrase, some corruption etc etx - are a bigger risk for me losing data than someone somehow gaining access to my Storage Boxes, in part because I don't have particularly sensitive data stored.

[potemkin_ai]

Hetzner's box, SSHFS with encryption - recovery fails with:

2024-10-19T13:37:34.456Z box:provision setProgress: restore - Downloading backup /mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc
2024-10-19T13:37:34.456Z box:storage/filesystem download: /mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc
2024-10-19T13:37:34.547Z box:backupformat/tgz Attempt 2 failed. Will retry: tarExtract pipeline error: incorrect header check

Checking the file:

file /mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc
/mnt/cloudronbackup/my_domain_cloudron/2024-10-19-130519-879/box_v8.0.6.tar.gz.enc: data

Guess encrypted data can't have tar headers?

[potemkin_ai]

in my specific case - that was due to decryption key error