Backblaze encrypted backup with Rsync fails
-
wrote on Jan 4, 2022, 6:17 PM last edited by girish Jan 7, 2022, 5:56 PM
Hi All, I am unable to complete backup of a nextcloud instance (~500gb) with backblaze. I know cloudron backup is not usually recommended for larger apps but I thought I'd give it a shot again now that more backup parameters are tweakable. The error I am seeing seems related to encrypted file name length, here it is below:
Error uploading snapshot/app_<nextcloud app id here>/KQtuToxscu6Wp+TQwes2r+Zt37fqVz9KBKw9r3zlGFo/rLwS54KpQJ5BUWx9Wf57gBjNtYl00DkPTpRIO7DHr6E/TQkmjgv8A-8ri6mQv26IOC3SBV98HjxGQYu7Wj942Bc/uROadiwRY44JGJ7Dx8MZs9Tl43w1dViTXgkUZQELVs4/gfgicTfaoUCSMjUN1MOOBnZV3VIOt-jmyeRQIYHGPgs/yROzi4S0rB1UDLZciqxuJMHauKsCowwWvn14ymMFMCo/zK0oecogUE+CWK3JQQCDlW2mmn2Ny-BzrmGk3QokTgk/jOMbp4bhfQklbq5PlI9VYAxfjWMDqO-kd04br5t9gp4/Bdj3Hxfg03RHtuW1vhWh30hEtbB9hS75XLlvUkpdn5M-orp0303DwHg86Kj0jH+Z/hVvdpoNTbKd5ttdU9ZK5JkxtksDlRByNTlLrKZ0Kd5w/7Z2EkbU6DDooF1el4rNzETh4Hi9hW4gqC1yNKEy2byo/s788Z0aGNJt5k5t4LUbO8bmitJ+Y2lTTeYbNV-aDguk/TyEoW0Ge6PJHTg6BeGnNuN4ATowyTxWeejACY22-tgeQA7KJlA7-LVNN-ANz+xc8/DYluRyzYVRZRwR+OtWyLsND675uSdAq7WOzPwbl7tH4/jLUYu3qxS96Fj5-MwYq0YowSytZATPSJkcoBi-JbVgY/Hnvhas87QQpcrWkXBVoITd-eMTXxw6Gh4xCov9sqVsc/0y6Kh1Lt99UO6RdBTII7kgHHoPVLeSu9r-6HWmzi5MSBa9ioD6J9pkSDgI6bPq6a/7Z2EkbU6DDooF1el4rNzETh4Hi9hW4gqC1yNKEy2byo/Hnvhas87QQpcrWkXBVoITd-eMTXxw6Gh4xCov9sqVsc/1GQ+V1wtz+KCmZU1hEcQb0Wf0ZxKEAZhoejV31Cr6Lg/+tqCfG-K-nTRG3LiFmWOSqRg1HmzpiqY2ytGbHNF8Xxh4xhfznEEAfrMKhJIKDFl. Message: File name in UTF8 must be no more than 1024 bytes HTTP Code: InvalidRequest
-
wrote on Jan 31, 2022, 7:22 PM last edited by ethanxrosen Jan 31, 2022, 7:29 PM
For future reference i managed to clear it up by using a slightly modified version of Girish's search command. It seems like the length limit applies to path length not just the filename length
find . -type f -print|awk '{print length($0), $0}' | sort -n
Went into the web interface and deleted / renamed files longer than 156 bytes and finally cleared trashbin/versioned files
(wipes all deleted / old versions of files - careful):
sudo -u www-data php -f /app/code/occ versions:cleanup sudo -u www-data php -f /app/code/occ trashbin:cleanup
@robi @girish agreed it would be awesome if there was an automated way to handle this in the future
-
Hi All, I am unable to complete backup of a nextcloud instance (~500gb) with backblaze. I know cloudron backup is not usually recommended for larger apps but I thought I'd give it a shot again now that more backup parameters are tweakable. The error I am seeing seems related to encrypted file name length, here it is below:
Error uploading snapshot/app_<nextcloud app id here>/KQtuToxscu6Wp+TQwes2r+Zt37fqVz9KBKw9r3zlGFo/rLwS54KpQJ5BUWx9Wf57gBjNtYl00DkPTpRIO7DHr6E/TQkmjgv8A-8ri6mQv26IOC3SBV98HjxGQYu7Wj942Bc/uROadiwRY44JGJ7Dx8MZs9Tl43w1dViTXgkUZQELVs4/gfgicTfaoUCSMjUN1MOOBnZV3VIOt-jmyeRQIYHGPgs/yROzi4S0rB1UDLZciqxuJMHauKsCowwWvn14ymMFMCo/zK0oecogUE+CWK3JQQCDlW2mmn2Ny-BzrmGk3QokTgk/jOMbp4bhfQklbq5PlI9VYAxfjWMDqO-kd04br5t9gp4/Bdj3Hxfg03RHtuW1vhWh30hEtbB9hS75XLlvUkpdn5M-orp0303DwHg86Kj0jH+Z/hVvdpoNTbKd5ttdU9ZK5JkxtksDlRByNTlLrKZ0Kd5w/7Z2EkbU6DDooF1el4rNzETh4Hi9hW4gqC1yNKEy2byo/s788Z0aGNJt5k5t4LUbO8bmitJ+Y2lTTeYbNV-aDguk/TyEoW0Ge6PJHTg6BeGnNuN4ATowyTxWeejACY22-tgeQA7KJlA7-LVNN-ANz+xc8/DYluRyzYVRZRwR+OtWyLsND675uSdAq7WOzPwbl7tH4/jLUYu3qxS96Fj5-MwYq0YowSytZATPSJkcoBi-JbVgY/Hnvhas87QQpcrWkXBVoITd-eMTXxw6Gh4xCov9sqVsc/0y6Kh1Lt99UO6RdBTII7kgHHoPVLeSu9r-6HWmzi5MSBa9ioD6J9pkSDgI6bPq6a/7Z2EkbU6DDooF1el4rNzETh4Hi9hW4gqC1yNKEy2byo/Hnvhas87QQpcrWkXBVoITd-eMTXxw6Gh4xCov9sqVsc/1GQ+V1wtz+KCmZU1hEcQb0Wf0ZxKEAZhoejV31Cr6Lg/+tqCfG-K-nTRG3LiFmWOSqRg1HmzpiqY2ytGbHNF8Xxh4xhfznEEAfrMKhJIKDFl. Message: File name in UTF8 must be no more than 1024 bytes HTTP Code: InvalidRequest
wrote on Jan 6, 2022, 12:36 AM last edited by d19dotca Jan 6, 2022, 12:37 AM@ethanxrosen The error indicates you have at least one file which is too lengthy in name for Backblaze's requirements. This is a somewhat common side effect of encryption, unfortunately, when it comes to using object storage services. That's not even a BB issue specifically but common to many of those providers. In your case, you'll either need to determine what file is too lengthy to see if you can trim it down a bit so that it won't be too long once encrypted, or abandon encryption to any object storage services all together.
-
wrote on Jan 6, 2022, 8:54 AM last edited by
one can also change the cypher or other parameters which won't generate the same length cypher text.
this could also be checked by Cloudron and perhaps split into two objects/files.
-
Encryption has filename length limitations of 156 bytes - https://docs.cloudron.io/backups/#encryption
If you run the command listed there, it will tell you what filenames are too long (run this using Web terminal):
cd /app/data find . -type f -printf "%f\n" | awk '{ print length(), $0 | "sort -rn" }' | less
-
Encryption has filename length limitations of 156 bytes - https://docs.cloudron.io/backups/#encryption
If you run the command listed there, it will tell you what filenames are too long (run this using Web terminal):
cd /app/data find . -type f -printf "%f\n" | awk '{ print length(), $0 | "sort -rn" }' | less
-
@girish if it's just a renaming issue, that would be an easy filter for the Cloudron backup process to make.. truncate names to 150 bytes.
-
@robi might conflict with an existing filename, if we truncate no? I guess another (dangerous) idea is to simply skip such files.
-
wrote on Jan 31, 2022, 7:22 PM last edited by ethanxrosen Jan 31, 2022, 7:29 PM
For future reference i managed to clear it up by using a slightly modified version of Girish's search command. It seems like the length limit applies to path length not just the filename length
find . -type f -print|awk '{print length($0), $0}' | sort -n
Went into the web interface and deleted / renamed files longer than 156 bytes and finally cleared trashbin/versioned files
(wipes all deleted / old versions of files - careful):
sudo -u www-data php -f /app/code/occ versions:cleanup sudo -u www-data php -f /app/code/occ trashbin:cleanup
@robi @girish agreed it would be awesome if there was an automated way to handle this in the future