Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Feature Requests
  3. Consider improvements to the backup experience to support long filenames/directory names

Consider improvements to the backup experience to support long filenames/directory names

Scheduled Pinned Locked Moved Feature Requests
backupsencryption
11 Posts 4 Posters 1.6k Views 4 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      R Offline
      Robin
      wrote on last edited by girish
      #1

      Large filenames cause problems, especially with blob-based storage (e.g. S3-compatible services have a key limit of 1024). This gives problems quite easily with hierarchical storage like syncthing where one might have a set of nested folders that quite easily hit that key limit, especially combined with encryption.

      Disabling encryption altogether is not desirable in many cases, but it could be worthwhile to allow disabling filename encryption, in some cases.

      (As a longer term project, it may also be worth considering ways to circumvent this limit altogether.)

      See https://forum.cloudron.io/topic/3050/backup-feedback-minio/ for the initial discussion leading to this request.

      1 Reply Last reply
      4
      • girishG Offline
        girishG Offline
        girish
        Staff
        wrote on last edited by
        #2

        If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

        Jan MacenkaJ 1 Reply Last reply
        0
        • girishG girish

          If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

          Jan MacenkaJ Offline
          Jan MacenkaJ Offline
          Jan Macenka
          wrote on last edited by
          #3

          @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

          Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

          Any update here? Is there something on the roadmap to solve this?

          girishG 2 Replies Last reply
          0
          • Jan MacenkaJ Jan Macenka

            @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

            Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

            Any update here? Is there something on the roadmap to solve this?

            girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #4

            @Jan-Macenka We can look into this for the next release but for the moment the best workaround is to rename the files. Alternately, don't use minio and instead use some sort of filesystem mount (nfs) setup.

            1 Reply Last reply
            0
            • Jan MacenkaJ Jan Macenka

              @girish is there any progress here? Wanted to implement CR for a business client and ran into this problem.

              Since the NextCloud Instance already has approx 1 TB and increasing, doing daily (request by client) tarball Backups to an offsite storage produces way too much traffic. rsync would be perfect and the Client already has MINIO services running but they encounter the already discussed key-length-error when syncing.

              Any update here? Is there something on the roadmap to solve this?

              girishG Offline
              girishG Offline
              girish
              Staff
              wrote on last edited by
              #5

              @Jan-Macenka @Robin Would it be acceptable if filename encryption was optional? That would atleast make it slightly better.

              Jan MacenkaJ 1 Reply Last reply
              0
              • girishG girish

                @Jan-Macenka @Robin Would it be acceptable if filename encryption was optional? That would atleast make it slightly better.

                Jan MacenkaJ Offline
                Jan MacenkaJ Offline
                Jan Macenka
                wrote on last edited by
                #6

                @girish as a quick fix this would help aslong as the contents still are encrypted 🙂

                girishG 1 Reply Last reply
                1
                • Jan MacenkaJ Jan Macenka

                  @girish as a quick fix this would help aslong as the contents still are encrypted 🙂

                  girishG Offline
                  girishG Offline
                  girish
                  Staff
                  wrote on last edited by
                  #7

                  @Jan-Macenka ok, that should be easy to do for next release. a checkbox that disables filename encryption.

                  1 Reply Last reply
                  2
                  • marcusquinnM Online
                    marcusquinnM Online
                    marcusquinn
                    wrote on last edited by
                    #8

                    I wonder if filenames/paths could be hashed into a database lookup table?

                    I've not looked in greater detail, but I suspect this how blockchain file storage systems like IFPS and Storj might handle such things.

                    Web Design https://www.evergreen.je
                    Development https://brandlight.org
                    Life https://marcusquinn.com

                    girishG 1 Reply Last reply
                    1
                    • marcusquinnM marcusquinn

                      I wonder if filenames/paths could be hashed into a database lookup table?

                      I've not looked in greater detail, but I suspect this how blockchain file storage systems like IFPS and Storj might handle such things.

                      girishG Offline
                      girishG Offline
                      girish
                      Staff
                      wrote on last edited by
                      #9

                      @marcusquinn yes, that's the long term plan. The implementation is a bit involved though. This "database" has to be stored as a separate file/object on s3, downloaded and then we have to decrypt filenames. And viceversa for the backup process where we have to upload this database after all files are backed up.

                      marcusquinnM 1 Reply Last reply
                      1
                      • girishG girish

                        @marcusquinn yes, that's the long term plan. The implementation is a bit involved though. This "database" has to be stored as a separate file/object on s3, downloaded and then we have to decrypt filenames. And viceversa for the backup process where we have to upload this database after all files are backed up.

                        marcusquinnM Online
                        marcusquinnM Online
                        marcusquinn
                        wrote on last edited by
                        #10

                        @girish Yeah, that's the fun part, easier said than done eh — especially when you have to consider unexpected interrupts & resumes etc.

                        Web Design https://www.evergreen.je
                        Development https://brandlight.org
                        Life https://marcusquinn.com

                        1 Reply Last reply
                        0
                        • girishG Offline
                          girishG Offline
                          girish
                          Staff
                          wrote on last edited by
                          #11

                          @Jan-Macenka OK, I have made filename encryption optional in 7.3.

                          fbb205df-5917-403b-92c5-9084a856c656-image.png

                          1 Reply Last reply
                          3
                          Reply
                          • Reply as topic
                          Log in to reply
                          • Oldest to Newest
                          • Newest to Oldest
                          • Most Votes


                            • Login

                            • Don't have an account? Register

                            • Login or register to search.
                            • First post
                              Last post
                            0
                            • Categories
                            • Recent
                            • Tags
                            • Popular
                            • Bookmarks
                            • Search