Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Consider improvements to the backup experience to support long filenames/directory names

    Feature Requests
    backups encryption
    4
    11
    427
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      Robin last edited by girish

      Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

      Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

      (As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

      See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

      1 Reply Last reply Reply Quote 4
      • girish
        girish Staff last edited by

        If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

        Jan Macenka 1 Reply Last reply Reply Quote 0
        • Jan Macenka
          Jan Macenka @girish last edited by

          @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

          Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

          Any update here? Is there something on the roadmap to solve this?

          girish 2 Replies Last reply Reply Quote 0
          • girish
            girish Staff @Jan Macenka last edited by

            @Jan-Macenka We can look into this for the next release but for the moment the best workaround is to rename the files. Alternately, don't use minio and instead use some sort of filesystem mount (nfs) setup.

            1 Reply Last reply Reply Quote 0
            • girish
              girish Staff @Jan Macenka last edited by

              @Jan-Macenka @Robin Would it be acceptable if filename encryption was optional? That would atleast make it slightly better.

              Jan Macenka 1 Reply Last reply Reply Quote 0
              • Jan Macenka
                Jan Macenka @girish last edited by

                @girish as a quick fix this would help aslong as the contents still are encrypted 🙂

                girish 1 Reply Last reply Reply Quote 1
                • girish
                  girish Staff @Jan Macenka last edited by

                  @Jan-Macenka ok, that should be easy to do for next release. a checkbox that disables filename encryption.

                  1 Reply Last reply Reply Quote 2
                  • marcusquinn
                    marcusquinn last edited by

                    I wonder if filenames/paths could be hashed into a database lookup table?

                    I've not looked in greater detail, but I suspect this how blockchain file storage systems like IFPS and Storj might handle such things.

                    We're not here for a long time - but we are here for a good time :)
                    Jersey/UK
                    Work & Ecommerce Advice: https://brandlight.org
                    Personal & Software Tips: https://marcusquinn.com

                    girish 1 Reply Last reply Reply Quote 1
                    • girish
                      girish Staff @marcusquinn last edited by

                      @marcusquinn yes, that's the long term plan. The implementation is a bit involved though. This "database" has to be stored as a separate file/object on s3, downloaded and then we have to decrypt filenames. And viceversa for the backup process where we have to upload this database after all files are backed up.

                      marcusquinn 1 Reply Last reply Reply Quote 1
                      • marcusquinn
                        marcusquinn @girish last edited by

                        @girish Yeah, that's the fun part, easier said than done eh — especially when you have to consider unexpected interrupts & resumes etc.

                        We're not here for a long time - but we are here for a good time :)
                        Jersey/UK
                        Work & Ecommerce Advice: https://brandlight.org
                        Personal & Software Tips: https://marcusquinn.com

                        1 Reply Last reply Reply Quote 0
                        • girish
                          girish Staff last edited by

                          @Jan-Macenka OK, I have made filename encryption optional in 7.3.

                          fbb205df-5917-403b-92c5-9084a856c656-image.png

                          1 Reply Last reply Reply Quote 3
                          • First post
                            Last post
                          Powered by NodeBB