Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    SOLVED DigitalOcean Space Backup Failed - SlowDown: Reduce your request rate.

    Support
    digitalocean sgp1 spaces
    3
    17
    95
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      andirahmat @girish last edited by

      @girish I'm sorry for late reply. I did today, i'll follow up their response soon.

      1 Reply Last reply Reply Quote 0
      • A
        andirahmat @jimcavoli last edited by andirahmat

        @jimcavoli Ah I see. Is there anything we can do on cloudron side? I do not think i use rsycn btw. I've also enable cdn. Or deleting existing backup files will solve with the limitation?

        jimcavoli girish 2 Replies Last reply Reply Quote 0
        • jimcavoli
          jimcavoli App Dev @andirahmat last edited by

          @andirahmat Unfortunately, no, there's not much to be done. The limitation of the service is there all the time. The backup mechanism could be made to back off a little more aggressively in the case of 503 errors specifically, trading speed for success, I suppose @girish could help make the call on whether this has been an issue with other backends as well

          1 Reply Last reply Reply Quote 0
          • girish
            girish Staff @andirahmat last edited by

            @andirahmat It's quite strange that you hit rate limits when using tgz format. Are you able to reach out to us on support@cloudron.io so we can debug this further? We should be far below the request limit. Also, AFAIK, the rate limits are per account/token. Could there be other things other than Cloudron using the same DO Space?

            A 2 Replies Last reply Reply Quote 0
            • A
              andirahmat @girish last edited by

              @girish said in DigitalOcean Space Backup Failed - SlowDown: Reduce your request rate.:

              ere be other things oth

              Hi. I only have 1 spaces and only use cloudron for it.
              Thank you Girish i'm opening ticket on cloudron support.

              And also here is the reponse from DO support :

              Thank you for your cooperation.

              Can you please confirm exactly how you are trying to upload the content? Is it trough control panel? If yes, then can you please try with the s3cmd tool?

              Please look at below article which will help you to understand how to upload files using s3cmd.

              https://www.digitalocean.com/docs/spaces/resources/s3cmd-usage/#upload-files-to-a-space

              1 Reply Last reply Reply Quote 0
              • A
                andirahmat @girish last edited by

                @girish I've send emails to support@cloudron.io

                There is follow up response from do support, i tried again but still failed :

                Thanks for reaching out to us. We have escalated this issue to our engineering team. They mentioned that there was a dip in API availability around the time of this issue. Just to confirm, are you still experiencing these issues? If so, we will need you to try to upload using s3cmd, and we will need to see the output of any errors that occur:

                https://www.digitalocean.com/docs/spaces/resources/s3cmd/
                https://www.digitalocean.com/docs/spaces/resources/s3cmd-usage/

                jimcavoli 1 Reply Last reply Reply Quote 0
                • jimcavoli
                  jimcavoli App Dev @andirahmat last edited by

                  @andirahmat Are you still seeing the rate limit error regularly?

                  girish 1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff @jimcavoli last edited by

                    @jimcavoli He has reached out to us on support, so I am looking into this later today.

                    1 Reply Last reply Reply Quote 1
                    • girish
                      girish Staff last edited by girish

                      This turned out of be a DO SGP region issue. If you try to multi-part upload with a part size > 15MB, it fails (!). I was able to reproduce this with the s3cmd tool.

                      root@localhost:~# s3cmd --access_key=key --secret_key=secret --host=sgp1.digitaloceanspaces.com --host-bucket=bucket --multipart-chunk-size-mb=20 put 500MB.bin s3://bucket/500MB.bin
                      upload: '500MB.bin' -> 's3://bgsmd/500MB.bin'  [part 1 of 25, 20MB] [1 of 1]
                       20971520 of 20971520   100% in    0s    42.81 MB/s  done
                      WARNING: Upload failed: /500MB.bin?partNumber=1&uploadId=2~mT5ZeflCYTchMXHNZc5EZUyc9ABNpT- (500 (UnknownError))
                      WARNING: Waiting 3 sec...
                      upload: '500MB.bin' -> 's3://bgsmd/500MB.bin'  [part 1 of 25, 20MB] [1 of 1]
                       20971520 of 20971520   100% in    5s     3.82 MB/s  done
                      WARNING: Upload failed: /500MB.bin?partNumber=1&uploadId=2~mT5ZeflCYTchMXHNZc5EZUyc9ABNpT- (500 (UnknownError))
                      WARNING: Waiting 6 sec...
                      

                      Reduce chunk size all the way down to 15MB, makes it work.

                      1 Reply Last reply Reply Quote 0
                      • girish
                        girish Staff last edited by

                        @andirahmat if you open a support ticket with them, let us know the outcome. I cannot open a ticket from our account because SGP1 Spaces is disabled for new spaces creation. Looks like they hit some capacity problems there and are thus limiting uploads.

                        1 Reply Last reply Reply Quote 0
                        • girish
                          girish Staff last edited by

                          Will mark this as solved since this is a DO issue.

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post