Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Minio backup fails for no reason

Minio backup fails for no reason

Scheduled Pinned Locked Moved Solved Support
miniobackups
16 Posts 4 Posters 2.9k Views 4 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T Offline
      T Offline
      thibaud
      wrote on last edited by girish
      #1

      Hello,

      I have configured backups with Minio and it worked perfectly for weeks.

      But it does no longer work. I think the problem started after an update, but I am not sure. As far as I understand from the logs (both from Cloudron and Minio sides) Cloudron tries to copy to Minio bucket a multipart archive and it fails. And Minio log says Cloudron tries to rename and/or copy a file that does not exist (yet, I guess).

      Abstract from Cloudron log :

      Unable to backup { BoxError: Error copying crsav_/snapshot/app_f0770611-2970-4e86-9047-0260e0297b79.tar.gz.enc (35961955060 bytes): XMLParserError XMLParserError: Unexpected close tag\nLine: 5\nColumn: 7\nChar: >' }
      

      Abstract from Minio log :

      Error: rename /data/.minio.sys/tmp/057fd8e5-234e-4e99-abe9-4ddb96796b34/e4b6158e-b4f5-4afc-8bb2-08e63228e618.2ff401ac-895f-43f9-aa2a-e40e9b862aef.9 /data/.minio.sys/multipart/caa122a8292342e6d4f7592bf3aed821c15f88ae471d458b70474d192b99be02/e4b6158e-b4f5-4afc-8bb2-08e63228e618/00009.0d55bc9f9c910f5b64049580ba387d34.1073741824: no such file or directory
      
             5: github.com/minio/minio@/cmd/fs-v1-helpers.go:397:cmd.fsSimpleRenameFile()
      
             4: github.com/minio/minio@/cmd/fs-v1-multipart.go:331:cmd.(*FSObjects).PutObjectPart()
      
             3: github.com/minio/minio@/cmd/fs-v1-multipart.go:259:cmd.(*FSObjects).CopyObjectPart()
      
             2: github.com/minio/minio@/cmd/object-handlers.go:1892:cmd.objectAPIHandlers.CopyObjectPartHandler()
      
             1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()
      

      Thank you,
      Thibaud

      1 Reply Last reply
      0
      • girishG Offline
        girishG Offline
        girish
        Staff
        wrote on last edited by girish
        #2

        @thibaud Do you have a guesstimate of what is the size of that file? Also, in the backup configuration, do you minio or s3 compatible as the provider? Can you also post a screenshot of Backups -> configure -> advanced settings (just the bottom part of the advanced settings screenshot).

        1 Reply Last reply
        0
        • T Offline
          T Offline
          thibaud
          wrote on last edited by
          #3

          @girish I have no idea. I understand the backup fails when it comes to Nextcloud. So basically, the file is large.

          Backup conf is on Minio.

          bce3e0d3-9a6b-4162-bab0-3200417d7bf2-image.png

          Are those settings new ? I am pretty sure I have never seen them. I guess they have something to do with my problem...

          1 Reply Last reply
          0
          • girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #4

            @thibaud said in Minio backup fails for no reason:

            So basically, the file is large

            You can check this from the graphs maybe as to how much the app consumes? It's under System view.

            Are those settings new ?

            Yes, new in 5.4 and 5.5

            1 Reply Last reply
            0
            • T Offline
              T Offline
              thibaud
              wrote on last edited by
              #5

              @girish

              You can check this from the graphs maybe as to how much the app consumes? It's under System view.

              Yes. As expected its huge as it is a Nextcloud. The entire app consumes 36 GB.

              Yes, new in 5.4 and 5.5

              And I think my problem appeared after 5.4 update. So basically I guess I need to increase memory limit? Up to... how much?

              robiR girishG 2 Replies Last reply
              0
              • T thibaud

                @girish

                You can check this from the graphs maybe as to how much the app consumes? It's under System view.

                Yes. As expected its huge as it is a Nextcloud. The entire app consumes 36 GB.

                Yes, new in 5.4 and 5.5

                And I think my problem appeared after 5.4 update. So basically I guess I need to increase memory limit? Up to... how much?

                robiR Offline
                robiR Offline
                robi
                wrote on last edited by
                #6

                @thibaud until it works 😕

                Conscious tech

                1 Reply Last reply
                0
                • T thibaud

                  @girish

                  You can check this from the graphs maybe as to how much the app consumes? It's under System view.

                  Yes. As expected its huge as it is a Nextcloud. The entire app consumes 36 GB.

                  Yes, new in 5.4 and 5.5

                  And I think my problem appeared after 5.4 update. So basically I guess I need to increase memory limit? Up to... how much?

                  girishG Offline
                  girishG Offline
                  girish
                  Staff
                  wrote on last edited by
                  #7

                  @thibaud Yes, I would start with say 1GB or so and see if it works. Note that this memory is just the "max" and also only for the duration of the backup itself.

                  1 Reply Last reply
                  0
                  • T Offline
                    T Offline
                    thibaud
                    wrote on last edited by
                    #8

                    Thank you all.

                    @girish I increased the parameters step by step and tried to backup for each (until the max i.e. 16 go). I still have the same error message. Any help?

                    girishG 1 Reply Last reply
                    0
                    • T thibaud

                      Thank you all.

                      @girish I increased the parameters step by step and tried to backup for each (until the max i.e. 16 go). I still have the same error message. Any help?

                      girishG Offline
                      girishG Offline
                      girish
                      Staff
                      wrote on last edited by
                      #9

                      @thibaud Are you able to mail the entire backup log to support@cloudron.io ? Let's see if it's something obvious from the logs.

                      T 1 Reply Last reply
                      0
                      • girishG girish

                        @thibaud Are you able to mail the entire backup log to support@cloudron.io ? Let's see if it's something obvious from the logs.

                        T Offline
                        T Offline
                        thibaud
                        wrote on last edited by
                        #10

                        @girish Done!

                        girishG 1 Reply Last reply
                        0
                        • T thibaud

                          @girish Done!

                          girishG Offline
                          girishG Offline
                          girish
                          Staff
                          wrote on last edited by girish
                          #11

                          @thibaud Thanks. It seems multi-part copies are failing altogether. As in, even the first part is failing. I thought it was failing much later.

                          Retrying (1) multipart copy of crsav_/snapshot/app_f0770611-2970-4e86-9047-0260e0297b79.tar.gz.enc. Error: XMLParserError: Unexpected close tag\nLine: 5\nColumn: 7\nChar: > 405 (
                          

                          Let me quickly test this and see if it's easy to reproduce.

                          1 Reply Last reply
                          0
                          • girishG Offline
                            girishG Offline
                            girish
                            Staff
                            wrote on last edited by
                            #12

                            @thibaud Mmm, I cannot reproduce this. I tried to backup and it succeeded over ~40 parts (https://paste.cloudron.io/ihulamulow.apache). So, to take a step back, it failed in "rename" . Is there something special about the filesystem where minio is located? Is it ext4? What is the underlying filesytem of Cloudron itself?

                            T 1 Reply Last reply
                            0
                            • girishG girish

                              @thibaud Mmm, I cannot reproduce this. I tried to backup and it succeeded over ~40 parts (https://paste.cloudron.io/ihulamulow.apache). So, to take a step back, it failed in "rename" . Is there something special about the filesystem where minio is located? Is it ext4? What is the underlying filesytem of Cloudron itself?

                              T Offline
                              T Offline
                              thibaud
                              wrote on last edited by
                              #13

                              @girish

                              My Minio is self-hosted in a Docker on a Synology NAS which underlying filesystem is proprietary (btfrs). I already checked if there are any known issue reported with Minio. I will double check.

                              My Cloudron is hosted on Ubuntu / ext4.

                              imc67I 1 Reply Last reply
                              0
                              • T thibaud

                                @girish

                                My Minio is self-hosted in a Docker on a Synology NAS which underlying filesystem is proprietary (btfrs). I already checked if there are any known issue reported with Minio. I will double check.

                                My Cloudron is hosted on Ubuntu / ext4.

                                imc67I Offline
                                imc67I Offline
                                imc67
                                translator
                                wrote on last edited by
                                #14

                                @thibaud said in Minio backup fails for no reason:

                                Minio is self-hosted in a Docker on a Synology NAS which underlying filesystem is proprietary (btfrs)

                                I have exactly the same situation on two Cloudrons to Minio's on two NAS's, only difference is that my backups are rsync and not tar. I experience no single issue and with Watchtower in Docker Minio is automatically updated to the latest version within 24 hours.

                                T 1 Reply Last reply
                                0
                                • imc67I imc67

                                  @thibaud said in Minio backup fails for no reason:

                                  Minio is self-hosted in a Docker on a Synology NAS which underlying filesystem is proprietary (btfrs)

                                  I have exactly the same situation on two Cloudrons to Minio's on two NAS's, only difference is that my backups are rsync and not tar. I experience no single issue and with Watchtower in Docker Minio is automatically updated to the latest version within 24 hours.

                                  T Offline
                                  T Offline
                                  thibaud
                                  wrote on last edited by
                                  #15

                                  @imc67 said in Minio backup fails for no reason:

                                  my backups are rsync and not tar

                                  Thanks for the tip. I switched to rsync and I was full of hope. Unfortunately it also fails.

                                  @girish I send you Minio full log by mail.

                                  girishG 1 Reply Last reply
                                  0
                                  • T thibaud

                                    @imc67 said in Minio backup fails for no reason:

                                    my backups are rsync and not tar

                                    Thanks for the tip. I switched to rsync and I was full of hope. Unfortunately it also fails.

                                    @girish I send you Minio full log by mail.

                                    girishG Offline
                                    girishG Offline
                                    girish
                                    Staff
                                    wrote on last edited by
                                    #16

                                    @thibaud I replied to you on support@ but the issue is that there is some long file name. The current rsync+encryption backup has some file name length limitation - https://docs.cloudron.io/backups/#encryption . There is a feature request at https://forum.cloudron.io/topic/3057/consider-improvements-to-the-backup-experience-to-support-long-filenames-directory-names .

                                    Run the following command in /home/yellowtent/appsdata to find the large filenames:

                                    find . -type f -printf "%f\n" | awk '{ print length(), $0 | "sort -rn" }' | less
                                    
                                    1 Reply Last reply
                                    0
                                    Reply
                                    • Reply as topic
                                    Log in to reply
                                    • Oldest to Newest
                                    • Newest to Oldest
                                    • Most Votes


                                      • Login

                                      • Don't have an account? Register

                                      • Login or register to search.
                                      • First post
                                        Last post
                                      0
                                      • Categories
                                      • Recent
                                      • Tags
                                      • Popular
                                      • Bookmarks
                                      • Search