humhub - Broken Backup

[wheez]

Hey Everyone

I was updating the Clear Theme of humhub and it somehow destroyed the Plugin. Since a reinstallation of the Plugin did not fix it I thought I would just restore from my backup since there has been no activity since the backup happend.

Now before restoring I triggered another backup just to be able to back in case something goes wrong.
But to my shock I had to find out, that all my backups are broken. The 5 Backups before the Plugin update and the one after as well. (They all show as completed/working)

They always fail with the same error:

Apr 20 12:52:20 box:tasks update 275: {"percent":65,"message":"Download finished 2025-04-18-230014-680/app_community.HIDDEN.de_v1.8.1/data/php.ini to /home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/php.ini"}
Apr 20 12:52:20 box:tasks update 275: {"percent":65,"message":"Download finished 2025-04-18-230014-680/app_community.HIDDEN.de_v1.8.1/data/modules/fcm-push/vendor/npm-asset/firebase/auth/package.json to /home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/modules/fcm-push/vendor/npm-asset/firebase/auth/package.json"}
Apr 20 12:52:20 box:tasks update 275: {"percent":65,"message":"Download finished 2025-04-18-230014-680/app_community.HIDDEN.de_v1.8.1/fsmetadata.json to /home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/fsmetadata.json"}
Apr 20 12:52:20 box:backupformat/rsync Recreating empty directories in {"localRoot":"/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e","layout":[]}
Apr 20 12:52:20 box:apptask run: app error for state pending_clone: BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file' at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:169:31) at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:259:5) at async download (/home/yellowtent/box/src/backuptask.js:104:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5) at async install (/home/yellowtent/box/src/apptask.js:368:9) { reason: 'FileSystem Error', details: {} }
Apr 20 12:52:20 box:taskworker Task took 685.211 seconds
Apr 20 12:52:20 box:tasks setCompleted - 275: {"result":null,"error":{"stack":"BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file'\n at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:169:31)\n at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:259:5)\n at async download (/home/yellowtent/box/src/backuptask.js:104:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"FileSystem Error","details":{},"message":"unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file'"}}
Apr 20 12:52:20 box:tasks update 275: {"percent":100,"result":null,"error":{"stack":"BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file'\n at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:169:31)\n at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:259:5)\n at async download (/home/yellowtent/box/src/backuptask.js:104:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)\n at async install (/home/yellowtent/box/src/apptask.js:368:9)","name":"BoxError","reason":"FileSystem Error","details":{},"message":"unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file'"}}
Apr 20 12:52:20 BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file'
Apr 20 12:52:20 at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:169:31)
Apr 20 12:52:20 at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:259:5)
Apr 20 12:52:20 at async download (/home/yellowtent/box/src/backuptask.js:104:5)
Apr 20 12:52:20 at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:138:5)
Apr 20 12:52:20 at async install (/home/yellowtent/box/src/apptask.js:368:9)

To me it is weird, that it just says /data. Other files have a proper file name, since I do not encrypt the filenames.

It also seems to be at the firestore plugin. Maybe I could remove that part from the backup and restore from it without that plugin and just install it again afterwards?

This is already a populated instance, so loosing all that data would hurt a lot. I backed the data up to my minio instance running on my synology nas.

Any ideas what I can do?

Thank you for your help!

[nebulon]

Hm so from the stacktrace, the file in question must have been a file with the exec bit set. For a start, do you see that file in your minio? Must be somewhere in the object path of one of the failing backups at /data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file

Do you have any retention settings in your minio config, which may have purged that file?

[wheez]

The buckets have no retention time, so all data is kept. But the path

/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file

is not in my minio. But that also makes sense since that path does not match the folder structure of the backups in general. As you see in the logs I provided a valid path is:

2025-04-18-230014-680/app_community.HIDDEN.de_v1.8.1/data/modules/fcm-push/vendor/npm-asset/firebase/auth/package.json

So it should definitely match path of /date/app/... but here it does not?
Is it maybe where the file needs to be copied to?

Somehow it seems to me to be a bug in the restore process. Since the error is happening on chmod while inside the cloudron docker directory /home/yellowtent/

[wheez]

I also just tried to restore a different app and that works without a problem. So the minio setup does not seem to be the reason of the error, but rather something humhub specific?

[nebulon]

Ah sorry, yes the path prefix ( 2025-04-18-230014-680/app_community.HIDDEN.de_v1.8.1/ ) is the path to the backup in question, so the file path should be appended to that. So something like:

2025-04-18-230014-680/app_community.HIDDEN.de_v1.8.1/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file

[wheez]

Ok. No this does not exist. There is no folder called /data/uploads inside the path. I also checked all existing previous backups of that app and the only folders existing inside /data is assets, config and modules (and then sometimes individual files).
I am still very confused that it is supposed to be just /file ? No extension, no name? All other files have proper filenames.

[wheez]

I guess it is just the folder where the uploads are saved?
https://github.com/humhub/humhub/tree/master/uploads/file

[wheez]

I also used the filesystem on my synology to search the folder. There is definitely no folder "uploads" in the backup. (searching inside the /<date>/<app> structure.

I did however look into the snapshots directory and there is a folder in

snapshot/app_de3a2cf0-dfa9-4ae3-9077-352d141cba17

where there is a folder /data/uploads/file.
There is also files up to the 19th of April in there from when the backup was from. And from the folderstructure in the /data/modules directory I can tell this is humhub.

How can I restore to that? Or is that already what the restore process is using?

[wheez]

I followed the guide here to decrypt the snapshot folder to take a look inside. https://docs.cloudron.io/guides/download-backups/

But I get the following error:

cloudron backup decrypt-dir --password=<MYPASSWORD> app_de3a2cf0-dfa9-4ae3-9077-352d141cba17 decrypt
node:internal/crypto/cipher:121
    this[kHandle].initiv(cipher, credential, iv, authTagLength);
                  ^

TypeError: Invalid initialization vector
    at Decipheriv.createCipherBase (node:internal/crypto/cipher:121:19)
    at Decipheriv.createCipherWithIV (node:internal/crypto/cipher:133:3)
    at new Decipheriv (node:internal/crypto/cipher:278:3)
    at Object.createDecipheriv (node:crypto:147:10)
    at decryptFilePath (/opt/homebrew/lib/node_modules/cloudron/src/backup-tools.js:55:36)
    at Command.decryptDir (/opt/homebrew/lib/node_modules/cloudron/src/backup-tools.js:284:50)
    at Command.listener [as _actionHandler] (/opt/homebrew/lib/node_modules/cloudron/node_modules/commander/lib/command.js:552:17)
    at /opt/homebrew/lib/node_modules/cloudron/node_modules/commander/lib/command.js:1586:14
    at Command._chainOrCall (/opt/homebrew/lib/node_modules/cloudron/node_modules/commander/lib/command.js:1470:12)
    at Command._parseCommand (/opt/homebrew/lib/node_modules/cloudron/node_modules/commander/lib/command.js:1585:27) {
  code: 'ERR_CRYPTO_INVALID_IV'
}

Node.js v23.11.0

Sorry that I spam this thread so much but I really need to get this back to working asap. So just adding the information I gather as I go along.

[wheez]

@wheez

inside of the snapshot folder there is the file

/snapshot/app_de3a2cf0-dfa9-4ae3-9077-352d141cba17/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file

so exactly the file that is erroring from the restore process.

I am still confused about the logs saying

---

Recreating empty directories in {"localRoot":"/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e","layout":[]}

BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/a5fdc9a1-e174-4e16-8826-51bc1025988e/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file'

---

So it seems like it is not failing to find or copy but just to create the empty folders. When I check out the failed restore app inside the File Manager there is the directory /app/data/uploads/file/d and inside are the empty folders 0,1,9,b,f. The folder 2 is missing.

[wheez]

This is the directory. So my current hypothesis is that there is a confusion about what is a directory and what is a file. It seems like humhub is saving the uploads simply as 'file' without a file ending.
But it is clear that this is a image, not a directory since it is 4mb in size and has a preview-image.

And I guess that in the restore process rsync gets confused about if it is a file or directory and errors because of that?

Is there a way to restore the backup without the uploads directory and just upload it after the backup has been restored? I was thinking by removing that part from fsmetadata.json.
I would really need to get the decryption on my machine running to be able to do try that and do further research.

[nebulon]

That is also an option that this folder it tries to chmod is an empty one. The logs are just next to each other as that is the flow of the restore process. That does not mean the errors are related as such).

If you don't see those entries in your backup storage, then they could be empty directories, which cannot be stored in an object store, so ther is a fsmetadata.json which contains those.

Now with your latest screenshot, it seems that failing folder is not an empty one, as it contains objects. So then the question is, why the rotated final backup paths don't have those objects in your minio. You could try to copy them over from the snaphots (that is what the backup code does during rotation) and then trigger a restore, maybe this is really only that one file?

[wheez]

I copied the /uploads folder from the snapshot directory into the backup directory at 2025-04-19-131734-948/app_community.HIDDEN.de_v1.8.1/data. But this does not fix the error. I still get exactly the same error in the end.

[wheez]

@nebulon it is trying to run chmod on

/snapshot/app_de3a2cf0-dfa9-4ae3-9077-352d141cba17/data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file

which is a file, not a directory. So I am really wondering if that is the problem.

[wheez]

@wheez

To look more into it I really need to be able to decrypt the files of my backup on my machine to check out fsmetadata.json. Can you please help me with that?
I have used the guide here: https://docs.cloudron.io/guides/decrypt-backups/ but only get the error described in the previous post. Is there another way to decrypt the directory?

[nebulon]

That guide should be correct, the error you are seeing might indicate that the decryption fails due to a wrong encryption password? Not quite clear to me. But since the folder is not empty and the entries which it complains about are files, we may not need to see the decrypted version of the json file.

Also just to double check, you have copied from the snapshot to the location within the backup you are also restoring? (the paths are easy to get wrong with that timestamp prefix)

[wheez]

Yes I moved it in the right folder, matching the date of the backup I am restoring from.

I also tried now to restore from external backup and give the snapshot directory as the location of the backup. But there I get exactly the same error as before.

Can you help me do the following:
Restore from the backup but change it so that it does not try to restore the uploads folder? I just want to restore and then upload the folder afterwards. That should enable me to get everything running again.

Are you sure it is the wrong encryption key, not some other bug? I have saved my encryption key in my password manager so I am quite sure it is right. Since my cloudron is still running is there a way to check out the encryption key from within cloudron to confirm? It is hidden in the gui but since I don't have to enter the encryption key on restore I guess it is still saved inside cloudron?

[wheez]

ok so I tried to restore to a different cloudron server and I get the same error. But before that error it works without a problem showing me that my encryption password I have is actually correct.
When I put in a different password I get an error that the encryption failed.

[wheez]

I was able to decrypt the fsmetadata.json with the command

cloudron backup decrypt

so something seems to be broken with the cli. I also made sure to update it to the current version.

Now in the decrypted fsmetadata.json I found this part:

    "execFiles": [
        "./data/uploads/file/d/2/d29213ba-4708-4809-b1f8-700f800ac6ba/file",
        "./data/uploads/file/d/3/d3e8ee5e-7ea5-496d-95d6-c6ca4ccf9cc1/file",
        "./data/uploads/file/5/5/55ab31ec-5b88-483f-951f-e9623e8d4ff9/file",
        "./data/uploads/file/5/e/5e1dbb9a-db2c-4ca9-a8a7-a07ad05d19f9/file",
        "./data/uploads/file/5/1/516ea496-f931-45d4-97c9-99ff95c6cf9d/file",
        "./data/uploads/file/5/a/5a981f44-ff10-4b49-842b-cc63616d846c/file",
        "./data/uploads/file/5/3/532d9935-3eec-45c3-9469-b736ec86bdf5/file",
        "./data/uploads/file/2/d/2d582f11-046e-449a-a005-17fdf0a86579/file",
        "./data/uploads/file/2/d/2d4d72fe-87a6-4aef-8fdb-78cff8be20af/file",
        "./data/uploads/file/2/5/255ddaa6-0075-498f-a40f-67de3ef53c56/file",
        "./data/uploads/file/2/e/2e76a400-8220-4fc3-bdf2-683a4aa8afb0/file",
        "./data/uploads/file/2/6/26237f18-cde3-42b2-9ecc-5deca2fb68f2/file",
        "./data/uploads/file/2/6/26274aff-c005-4f4d-81ca-0ebd0920d45e/file",
        "./data/uploads/file/2/6/26c5e394-75c4-416e-8c23-3b6f70e840f5/file",
        "./data/uploads/file/2/6/26cc0a5f-d2a7-40a4-8d28-d53b031fe8a0/file",
        "./data/uploads/file/2/6/26cb69ad-ea82-4f12-8875-f37f3931d95c/file",
        "./data/uploads/file/2/4/243ccdea-b57e-4425-b533-c1e6c67109fb/file",
        "./data/uploads/file/2/9/29580126-7579-42fd-8f97-3892dd631d21/file",
        "./data/uploads/file/2/9/29fdcfbb-33c2-47f2-a13c-6dfae0aa0d64/file",
        "./data/uploads/file/e/2/e2eb9830-15ad-4a52-860b-8e3b10a558a5/file",
        "./data/uploads/file/e/e/ee5320b0-9b3c-4eac-99a1-3faef8a1b020/file",
        "./data/uploads/file/e/6/e60a8456-58c0-48ee-b082-23d7011213dc/file",
        "./data/uploads/file/e/6/e60a8456-58c0-48ee-b082-23d7011213dc/old-5",
        "./data/uploads/file/e/4/e4c62fcf-af7d-48bc-8774-9f81a89740b0/file",
        "./data/uploads/file/e/9/e9a8304b-6ff9-4f97-9d71-864396833475/file",
        "./data/uploads/file/e/9/e94109e2-5863-45e4-8e1b-45ac534ef125/file",
        "./data/uploads/file/e/0/e0c16fc6-2b2c-40db-a919-1912c0a1562d/file",
        "./data/uploads/file/1/6/16aa7a1a-26e6-47c0-aec9-b6af7ffe289d/file",
        "./data/uploads/file/1/b/1b941dca-50fe-46c6-bb72-146ed4c6a0d0/file",
        "./data/uploads/file/1/9/1976db36-ec69-46eb-ba56-e09dcde3d5aa/file",
        "./data/uploads/file/a/e/ae267001-e6c1-4b17-a431-3c9b375603fc/file",
        "./data/uploads/file/6/e/6e3a2379-ffa9-4926-9813-7a030f4e3f6d/file",
        "./data/uploads/file/6/9/69fd301e-26e8-4c6b-896d-f11b636a7cb6/file",
        "./data/uploads/file/6/0/6041fd0a-1aa6-4df2-9035-5b0149f37d19/file",
        "./data/uploads/file/6/7/67fecd3c-c1eb-489a-8bdd-57c78a5a8fb3/file",
        "./data/uploads/file/6/8/68bb454c-8edd-456b-8721-6929bbe95cb5/file",
        "./data/uploads/file/6/8/68d2a785-f6aa-4dd4-80ea-65fb09c474b8/file",
        "./data/uploads/file/b/d/bd36f035-e77b-4a96-9bb7-ddb65a5750b7/file",
        "./data/uploads/file/b/e/beca9a5b-52dc-43ba-a83e-5153e001057b/file",
        "./data/uploads/file/b/e/beefb3f9-13bd-42a4-9635-d6ffbe690cc4/file",
        "./data/uploads/file/b/c/bc6c802c-7937-4f28-9ff0-c081e6d152fb/file",
        "./data/uploads/file/b/9/b91796a9-14a2-4295-bcb0-c936f3c76e77/file",
        "./data/uploads/file/4/a/4a2b4b01-e945-4167-adf2-2338d312bd82/file",
        "./data/uploads/file/4/4/4493e6c2-eec2-48d6-ac39-063ac1a73e36/file",
        "./data/uploads/file/4/f/4fcdd7fe-ac5b-4fbf-9a7b-74e9eb6e8d51/file",
        "./data/uploads/file/f/2/f2027049-ebdb-4711-bc1e-41779f7ee270/file",
        "./data/uploads/file/f/2/f2c92a65-f186-4947-9279-22d0310b2b01/file",
        "./data/uploads/file/f/1/f190bce4-4238-45dd-b139-22a7ea27948d/file",
        "./data/uploads/file/f/4/f4c41dbf-1882-46cb-837b-15a31a8733a9/file",
        "./data/uploads/file/f/0/f0dee544-3b1e-442a-b441-7f8f036b5562/file",
        "./data/uploads/file/c/d/cda56b9a-67a8-4747-a759-55696c719122/file",
        "./data/uploads/file/c/5/c5f5d7ea-3554-4d05-89e7-4f514d55907d/file",
        "./data/uploads/file/c/a/caa0771e-f1fa-4778-b211-2529903fac92/file",
        "./data/uploads/file/c/9/c9f19e18-3ac9-4281-ae1d-915c1ae3ef08/file",
        "./data/uploads/file/9/6/96af3a57-12b9-49bb-96ed-c0a4b78f3137/file",
        "./data/uploads/file/9/6/962112d7-71f9-4023-aadb-25e3b9404297/file",
        "./data/uploads/file/0/d/0dadd978-eda1-43d4-9c45-21a92a5ee8a5/file",
        "./data/uploads/file/0/5/05972e67-fe2d-449b-a4e6-f7de08f9b0d1/file",
        "./data/uploads/file/0/a/0a2d0fa1-42dc-4b41-a5f4-182221ba7209/file",
        "./data/uploads/file/0/3/03a5bbe4-a228-478d-9775-3e32b8fb52b3/file",
        "./data/uploads/file/3/2/326bec47-8e46-424c-bff8-7ed2a9c8fcf5/file",
        "./data/uploads/file/3/e/3e5626f3-8fcd-46f1-9f16-0e3811d12050/file",
        "./data/uploads/file/3/e/3edbf99e-d675-4a00-b540-0b0df1f41051/file",
        "./data/uploads/file/3/f/3f5c1c60-d5f6-4bde-ba53-09bcf56ce738/file",
        "./data/uploads/file/3/c/3c80e130-01de-46c6-9d95-93147b98b145/file",
        "./data/uploads/file/3/c/3c80e130-01de-46c6-9d95-93147b98b145/old-4",
        "./data/uploads/file/7/2/72549e3c-c857-4f49-bc25-08e91804fc21/file",
        "./data/uploads/file/7/1/71dde4c2-261b-4333-82bf-d46e1fed81d6/file",
        "./data/uploads/file/7/c/7c8c02ac-5c38-4b85-9d74-a9e343b29d00/file",
        "./data/uploads/file/8/d/8d2f9876-f045-45a3-9014-b6e751d6fa87/file",
        "./data/uploads/file/8/5/856fc239-2a32-47d4-9734-4a5a21f23cd5/file",
        "./data/uploads/file/8/e/8e0aca86-4abb-458b-a5e7-8e889f5c1a63/file",
        "./data/uploads/file/8/f/8f30270d-7339-42c6-8019-7d2647530c00/file",
        "./data/uploads/file/8/3/83f431b4-15bb-4ccd-81c3-72051d4447ca/file",
        "./data/uploads/file/8/8/881409a4-fb7f-4af7-97bb-4cc144ba61e2/file"
    ],

This seems very strange to me because it sets all these files as executable if I understand this correctly? But these are user uploaded files that do not (and should not) be executed as far as I understand?

[nebulon]

ah very good, then we have to look into the decryption of folders with the cli on the side. Are you running this on linux?

I agree the files should not be executable, maybe humhub stores them as such with 777 permissions. Have to check but probably not at all necessary.

Since those are the only exec files, we can try to skip that step. Are you able to edit that .json file, remove all entries in the execFiles array and reupload the file encrypted, then do a restore? Otherwise we have to skip that in the restore code path. This is happending at https://git.cloudron.io/platform/box/-/blob/master/src/backupformat/rsync.js?ref_type=heads#L178 not sure if you are able to change that on your server. It would be in /home/yellowtent/box/src/backupformat/rsync.js if you want to comment out those lines and then run systemctl restart box to then retrigger the restore. Otherwise send a mail to support@cloudron.io with server info and remote ssh enabled for us.