Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Feature Requests
  3. Consider improvements to the backup experience to support long filenames/directory names

Consider improvements to the backup experience to support long filenames/directory names

Scheduled Pinned Locked Moved Feature Requests
backupsencryption
11 Posts 4 Posters 1.7k Views 4 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R Offline
    R Offline
    Robin
    wrote on last edited by girish
    #1

    Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

    Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

    (As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

    See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

    1 Reply Last reply
    4
    • girishG Offline
      girishG Offline
      girish
      Staff
      wrote on last edited by
      #2

      If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

      Jan MacenkaJ 1 Reply Last reply
      0
      • girishG girish

        If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

        Jan MacenkaJ Offline
        Jan MacenkaJ Offline
        Jan Macenka
        wrote on last edited by
        #3

        @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

        Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

        Any update here? Is there something on the roadmap to solve this?

        girishG 2 Replies Last reply
        0
        • Jan MacenkaJ Jan Macenka

          @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

          Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

          Any update here? Is there something on the roadmap to solve this?

          girishG Offline
          girishG Offline
          girish
          Staff
          wrote on last edited by
          #4

          @Jan-Macenka We can look into this for the next release but for the moment the best workaround is to rename the files. Alternately, don't use minio and instead use some sort of filesystem mount (nfs) setup.

          1 Reply Last reply
          0
          • Jan MacenkaJ Jan Macenka

            @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

            Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

            Any update here? Is there something on the roadmap to solve this?

            girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #5

            @Jan-Macenka @Robin Would it be acceptable if filename encryption was optional? That would atleast make it slightly better.

            Jan MacenkaJ 1 Reply Last reply
            0
            • girishG girish

              @Jan-Macenka @Robin Would it be acceptable if filename encryption was optional? That would atleast make it slightly better.

              Jan MacenkaJ Offline
              Jan MacenkaJ Offline
              Jan Macenka
              wrote on last edited by
              #6

              @girish as a quick fix this would help aslong as the contents still are encrypted 🙂

              girishG 1 Reply Last reply
              1
              • Jan MacenkaJ Jan Macenka

                @girish as a quick fix this would help aslong as the contents still are encrypted 🙂

                girishG Offline
                girishG Offline
                girish
                Staff
                wrote on last edited by
                #7

                @Jan-Macenka ok, that should be easy to do for next release. a checkbox that disables filename encryption.

                1 Reply Last reply
                2
                • marcusquinnM Offline
                  marcusquinnM Offline
                  marcusquinn
                  wrote on last edited by
                  #8

                  I wonder if filenames/paths could be hashed into a database lookup table?

                  I've not looked in greater detail, but I suspect this how blockchain file storage systems like IFPS and Storj might handle such things.

                  Web Design https://www.evergreen.je
                  Development https://brandlight.org
                  Life https://marcusquinn.com

                  girishG 1 Reply Last reply
                  1
                  • marcusquinnM marcusquinn

                    I wonder if filenames/paths could be hashed into a database lookup table?

                    I've not looked in greater detail, but I suspect this how blockchain file storage systems like IFPS and Storj might handle such things.

                    girishG Offline
                    girishG Offline
                    girish
                    Staff
                    wrote on last edited by
                    #9

                    @marcusquinn yes, that's the long term plan. The implementation is a bit involved though. This "database" has to be stored as a separate file/object on s3, downloaded and then we have to decrypt filenames. And viceversa for the backup process where we have to upload this database after all files are backed up.

                    marcusquinnM 1 Reply Last reply
                    1
                    • girishG girish

                      @marcusquinn yes, that's the long term plan. The implementation is a bit involved though. This "database" has to be stored as a separate file/object on s3, downloaded and then we have to decrypt filenames. And viceversa for the backup process where we have to upload this database after all files are backed up.

                      marcusquinnM Offline
                      marcusquinnM Offline
                      marcusquinn
                      wrote on last edited by
                      #10

                      @girish Yeah, that's the fun part, easier said than done eh — especially when you have to consider unexpected interrupts & resumes etc.

                      Web Design https://www.evergreen.je
                      Development https://brandlight.org
                      Life https://marcusquinn.com

                      1 Reply Last reply
                      0
                      • girishG Offline
                        girishG Offline
                        girish
                        Staff
                        wrote on last edited by
                        #11

                        @Jan-Macenka OK, I have made filename encryption optional in 7.3.

                        fbb205df-5917-403b-92c5-9084a856c656-image.png

                        1 Reply Last reply
                        3
                        Reply
                        • Reply as topic
                        Log in to reply
                        • Oldest to Newest
                        • Newest to Oldest
                        • Most Votes


                        • Login

                        • Don't have an account? Register

                        • Login or register to search.
                        • First post
                          Last post
                        0
                        • Categories
                        • Recent
                        • Tags
                        • Popular
                        • Bookmarks
                        • Search