Backblaze encrypted backup with Rsync fails

[ethanxrosen]

Hi All, I am unable to complete backup of a nextcloud instance (~500gb) with backblaze. I know cloudron backup is not usually recommended for larger apps but I thought I'd give it a shot again now that more backup parameters are tweakable. The error I am seeing seems related to encrypted file name length, here it is below:

Error uploading snapshot/app_<nextcloud app id here>/KQtuToxscu6Wp+TQwes2r+Zt37fqVz9KBKw9r3zlGFo/rLwS54KpQJ5BUWx9Wf57gBjNtYl00DkPTpRIO7DHr6E/TQkmjgv8A-8ri6mQv26IOC3SBV98HjxGQYu7Wj942Bc/uROadiwRY44JGJ7Dx8MZs9Tl43w1dViTXgkUZQELVs4/gfgicTfaoUCSMjUN1MOOBnZV3VIOt-jmyeRQIYHGPgs/yROzi4S0rB1UDLZciqxuJMHauKsCowwWvn14ymMFMCo/zK0oecogUE+CWK3JQQCDlW2mmn2Ny-BzrmGk3QokTgk/jOMbp4bhfQklbq5PlI9VYAxfjWMDqO-kd04br5t9gp4/Bdj3Hxfg03RHtuW1vhWh30hEtbB9hS75XLlvUkpdn5M-orp0303DwHg86Kj0jH+Z/hVvdpoNTbKd5ttdU9ZK5JkxtksDlRByNTlLrKZ0Kd5w/7Z2EkbU6DDooF1el4rNzETh4Hi9hW4gqC1yNKEy2byo/s788Z0aGNJt5k5t4LUbO8bmitJ+Y2lTTeYbNV-aDguk/TyEoW0Ge6PJHTg6BeGnNuN4ATowyTxWeejACY22-tgeQA7KJlA7-LVNN-ANz+xc8/DYluRyzYVRZRwR+OtWyLsND675uSdAq7WOzPwbl7tH4/jLUYu3qxS96Fj5-MwYq0YowSytZATPSJkcoBi-JbVgY/Hnvhas87QQpcrWkXBVoITd-eMTXxw6Gh4xCov9sqVsc/0y6Kh1Lt99UO6RdBTII7kgHHoPVLeSu9r-6HWmzi5MSBa9ioD6J9pkSDgI6bPq6a/7Z2EkbU6DDooF1el4rNzETh4Hi9hW4gqC1yNKEy2byo/Hnvhas87QQpcrWkXBVoITd-eMTXxw6Gh4xCov9sqVsc/1GQ+V1wtz+KCmZU1hEcQb0Wf0ZxKEAZhoejV31Cr6Lg/+tqCfG-K-nTRG3LiFmWOSqRg1HmzpiqY2ytGbHNF8Xxh4xhfznEEAfrMKhJIKDFl. Message: File name in UTF8 must be no more than 1024 bytes HTTP Code: InvalidRequest

[ethanxrosen]

For future reference i managed to clear it up by using a slightly modified version of Girish's search command. It seems like the length limit applies to path length not just the filename length

find . -type f -print|awk '{print length($0), $0}' | sort -n

Went into the web interface and deleted / renamed files longer than 156 bytes and finally cleared trashbin/versioned files

(wipes all deleted / old versions of files - careful):

sudo -u www-data php -f /app/code/occ versions:cleanup
sudo -u www-data php -f /app/code/occ trashbin:cleanup

@robi @girish agreed it would be awesome if there was an automated way to handle this in the future

[d19dotca]

@ethanxrosen The error indicates you have at least one file which is too lengthy in name for Backblaze's requirements. This is a somewhat common side effect of encryption, unfortunately, when it comes to using object storage services. That's not even a BB issue specifically but common to many of those providers. In your case, you'll either need to determine what file is too lengthy to see if you can trim it down a bit so that it won't be too long once encrypted, or abandon encryption to any object storage services all together.

[robi]

one can also change the cypher or other parameters which won't generate the same length cypher text.

this could also be checked by Cloudron and perhaps split into two objects/files.

[girish]

Encryption has filename length limitations of 156 bytes - https://docs.cloudron.io/backups/#encryption

If you run the command listed there, it will tell you what filenames are too long (run this using Web terminal):

cd /app/data
find . -type f -printf "%f\n" | awk '{ print length(), $0 | "sort -rn" }' | less

[robi]

@girish if it's just a renaming issue, that would be an easy filter for the Cloudron backup process to make.. truncate names to 150 bytes.

[girish]

@robi might conflict with an existing filename, if we truncate no? I guess another (dangerous) idea is to simply skip such files.

[robi]

@girish not within the same backup, and not at the remote backup location due to use of directory structures.

[ethanxrosen]

For future reference i managed to clear it up by using a slightly modified version of Girish's search command. It seems like the length limit applies to path length not just the filename length

find . -type f -print|awk '{print length($0), $0}' | sort -n

Went into the web interface and deleted / renamed files longer than 156 bytes and finally cleared trashbin/versioned files

(wipes all deleted / old versions of files - careful):

sudo -u www-data php -f /app/code/occ versions:cleanup
sudo -u www-data php -f /app/code/occ trashbin:cleanup

@robi @girish agreed it would be awesome if there was an automated way to handle this in the future