Consider improvements to the backup experience to support long filenames/directory names

[Robin]

Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

(As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

[girish]

If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

[Jan Macenka]

@girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

Any update here? Is there something on the roadmap to solve this?

[girish]

@Jan-Macenka We can look into this for the next release but for the moment the best workaround is to rename the files. Alternately, don't use minio and instead use some sort of filesystem mount (nfs) setup.

[girish]

@Jan-Macenka @Robin Would it be acceptable if filename encryption was optional? That would atleast make it slightly better.

[Jan Macenka]

@girish as a quick fix this would help aslong as the contents still are encrypted

[girish]

@Jan-Macenka ok, that should be easy to do for next release. a checkbox that disables filename encryption.

[marcusquinn]

I wonder if filenames/paths could be hashed into a database lookup table?

I've not looked in greater detail, but I suspect this how blockchain file storage systems like IFPS and Storj might handle such things.

[girish]

@marcusquinn yes, that's the long term plan. The implementation is a bit involved though. This "database" has to be stored as a separate file/object on s3, downloaded and then we have to decrypt filenames. And viceversa for the backup process where we have to upload this database after all files are backed up.

[marcusquinn]

@girish Yeah, that's the fun part, easier said than done eh — especially when you have to consider unexpected interrupts & resumes etc.

[girish]

@Jan-Macenka OK, I have made filename encryption optional in 7.3.