Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Feature Requests
  3. Improve DNS updates to avoid rate limits

Improve DNS updates to avoid rate limits

Scheduled Pinned Locked Moved Solved Feature Requests
dns
18 Posts 4 Posters 2.7k Views 4 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by girish
    #1

    Currently adding multiple aliases or redirections does them all one at a time, and if 3 more are added to a list of lets say an existing 20 that are not changing, 23 new changes are done with each update which can trigger rate limits at the upstream DNS provider (Route53).

    Another approach would be to compare the existing list with the new list and only apply the 3 that are new or different.

    I'd prefer this to slowing things down rate wise.

    Conscious tech

    1 Reply Last reply
    0
    • girishG Offline
      girishG Offline
      girish
      Staff
      wrote on last edited by
      #2

      DNS requests are not done in parallel, they are only done one by one. But you can still hit rate limits of course.

      robiR 1 Reply Last reply
      0
      • girishG girish

        DNS requests are not done in parallel, they are only done one by one. But you can still hit rate limits of course.

        robiR Offline
        robiR Offline
        robi
        wrote on last edited by robi
        #3

        @girish Yes, I see that in the log before the "External Error, rate limit reached" message comes up.

        It's more than one per second and worse if there's IPv6 as it doubles the number of records it sets, even when no IP changes are needed.

        If triggered, it leaves the app in an Error state (does it need to be?), and if you retry it starts the whole thing from the beginning, not just the addresses that weren't set yet.
        Hence failure loops and extra downtime.

        So why set things that don't need to be set?

        Conscious tech

        girishG 1 Reply Last reply
        0
        • robiR robi

          @girish Yes, I see that in the log before the "External Error, rate limit reached" message comes up.

          It's more than one per second and worse if there's IPv6 as it doubles the number of records it sets, even when no IP changes are needed.

          If triggered, it leaves the app in an Error state (does it need to be?), and if you retry it starts the whole thing from the beginning, not just the addresses that weren't set yet.
          Hence failure loops and extra downtime.

          So why set things that don't need to be set?

          girishG Offline
          girishG Offline
          girish
          Staff
          wrote on last edited by
          #4

          @robi One still needs to query route53 to find out if it's set correctly. Do you know if that counts in rate limits?

          robiR 1 Reply Last reply
          0
          • girishG girish

            @robi One still needs to query route53 to find out if it's set correctly. Do you know if that counts in rate limits?

            robiR Offline
            robiR Offline
            robi
            wrote on last edited by
            #5

            @girish I would think that only needs to happen for the primary subdomain, not all the aliases/redirects.
            The app should still work.

            And yes, since it's an API, all queries count towards the rate limit.

            Even if there are 100+ aliases, it needs to have a way to succeed.

            Perhaps in a async way with eventual consistency.

            One would think there would also be a "bulk" API that can deal with lists. Or at least the ability to validate all existing ones.

            Conscious tech

            1 Reply Last reply
            0
            • robiR Offline
              robiR Offline
              robi
              wrote on last edited by
              #6

              What has also come up is the inability to uninstall an app due to the same rate limits.

              Uninstalling should work for the primary domain and the aliases could be done async with some +1 sec backoff if rate limits are hit.

              Now app is stuck in Error state, uninstall can't be cancelled as it errors before the task stop (red x) appears.

              App could be running, but isn't.

              Conscious tech

              1 Reply Last reply
              0
              • nebulonN Offline
                nebulonN Offline
                nebulon
                Staff
                wrote on last edited by
                #7

                The code does things in sequence and also retries. I am not sure what the solution is if rate-limits are so low (I mean we only talk about max tens of requests within a short time-frame, if I understand the situation correctly).

                Seems like there is something else at play in your case?

                As a quick solution, you can set the DNS provider temporarily to no-op for the uninstallation.

                robiR 1 Reply Last reply
                0
                • nebulonN nebulon

                  The code does things in sequence and also retries. I am not sure what the solution is if rate-limits are so low (I mean we only talk about max tens of requests within a short time-frame, if I understand the situation correctly).

                  Seems like there is something else at play in your case?

                  As a quick solution, you can set the DNS provider temporarily to no-op for the uninstallation.

                  robiR Offline
                  robiR Offline
                  robi
                  wrote on last edited by
                  #8

                  @nebulon said in Improve DNS updates to avoid rate limits:

                  The code does things in sequence and also retries.

                  Not seeing that in the logs. It mentions the error is one that can be continued, however that does not happen either.

                  IMO the error should not be fatal, so it can continue (not from the beginning) from where it left off and eventually complete all requests.

                  I am not sure what the solution is if rate-limits are so low

                  That burst seems to be the problem, if rate-limit is hit and handled in the code, it can be slowed to 1/sec to be ok and complete. A more universal solution for all providers.

                  As I already mentioned, optimizing the retry logic so it doesn't start from scratch and make it worse.
                  You can assume all previous API requests -1 succeeded, right?

                  Seems like there is something else at play in your case?

                  No, happening on the demo server as well.

                  As a quick solution, you can set the DNS provider temporarily to no-op for the uninstallation.

                  Yes, plus cleanup of the DNS manually.
                  Or the retry dialog which gives a checklist of the aliases.

                  Conscious tech

                  1 Reply Last reply
                  0
                  • L Offline
                    L Offline
                    LoudLemur
                    wrote on last edited by
                    #9

                    Route 53 (from Amazon) and Bunny.net are mentioned in the Cloudron blog on running a Cloudron at home.
                    https://blog.cloudron.io/installing-cloudron-on-a-home-server/

                    If you visit eg Bunny, they don't seem to sell domain names, in the way that other companies do, eg. Epik. How would you get a domain name from Bunny or route 53?

                    girishG 1 Reply Last reply
                    0
                    • L LoudLemur

                      Route 53 (from Amazon) and Bunny.net are mentioned in the Cloudron blog on running a Cloudron at home.
                      https://blog.cloudron.io/installing-cloudron-on-a-home-server/

                      If you visit eg Bunny, they don't seem to sell domain names, in the way that other companies do, eg. Epik. How would you get a domain name from Bunny or route 53?

                      girishG Offline
                      girishG Offline
                      girish
                      Staff
                      wrote on last edited by
                      #10

                      @LoudLemur I guess you are posting on the wrong thread?

                      L 1 Reply Last reply
                      0
                      • girishG girish

                        @LoudLemur I guess you are posting on the wrong thread?

                        L Offline
                        L Offline
                        LoudLemur
                        wrote on last edited by
                        #11

                        @girish said in Improve DNS updates to avoid rate limits:

                        @LoudLemur I guess you are posting on the wrong thread?

                        I thought I would throw the question about Bunny and Route 53 in here, since people seem to be familiar with the services and they are up for discussion. Maybe it would be better to create another thread. ...

                        1 Reply Last reply
                        0
                        • girishG Offline
                          girishG Offline
                          girish
                          Staff
                          wrote on last edited by
                          #12

                          With a bit of hunting, I found https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html#limits-api-requests . It says "Five requests per second per AWS account per Region" . Looks like they recommend implementing a backoff as the solution. I will try to add this in our route53 backend.

                          1 Reply Last reply
                          2
                          • girishG Offline
                            girishG Offline
                            girish
                            Staff
                            wrote on last edited by
                            #13

                            Retry added with https://git.cloudron.io/cloudron/box/-/commit/37ed87f9c18c8d4f37a86e5503e6cc2870b215e4

                            1 Reply Last reply
                            2
                            • girishG girish marked this topic as a question on
                            • girishG girish has marked this topic as solved on
                            • robiR Offline
                              robiR Offline
                              robi
                              wrote on last edited by
                              #14

                              Okay, thanks for the retry delay patch, can you apply it to the demo server?

                              This, however, does not solve the topic though, for which the list compare approach makes a large list with one change just do one change as an atomic subset, not the entire list as a global re-set.

                              In Multi Cloudron this could potentially create API storms.
                              We're getting close to this with the large netblock lists etc.

                              Please think ahead on this one.

                              Conscious tech

                              girishG 1 Reply Last reply
                              0
                              • robiR robi

                                Okay, thanks for the retry delay patch, can you apply it to the demo server?

                                This, however, does not solve the topic though, for which the list compare approach makes a large list with one change just do one change as an atomic subset, not the entire list as a global re-set.

                                In Multi Cloudron this could potentially create API storms.
                                We're getting close to this with the large netblock lists etc.

                                Please think ahead on this one.

                                girishG Offline
                                girishG Offline
                                girish
                                Staff
                                wrote on last edited by girish
                                #15

                                @robi I have patched the demo server.

                                As for the optimization: Cloudron checks if the record is up to date first and only updates it if it is not up to date. It's not a blind update each time. This is the best we can do without complex state tracking.

                                To write code such as "continue from where it last errored" and "update whatever you can right now" and "only updated what is minimally required" etc , requires Cloudron to track state . In practically all cases, this state tracking is not needed. The most common case (by which is mean 99.9999%) is that there are say < 5 domains for an app. So, all this code and state tracking to reduce some API requests? It's not worth the complexity and it's not like we are generating some 1000 requests. Also, think about all the UI buttons and error messages that now need to be created just for this. You need a button to say "continue" , messages that show what exactly went wrong and what domains did not update and why exactly and so on.

                                robiR 1 Reply Last reply
                                0
                                • girishG girish

                                  @robi I have patched the demo server.

                                  As for the optimization: Cloudron checks if the record is up to date first and only updates it if it is not up to date. It's not a blind update each time. This is the best we can do without complex state tracking.

                                  To write code such as "continue from where it last errored" and "update whatever you can right now" and "only updated what is minimally required" etc , requires Cloudron to track state . In practically all cases, this state tracking is not needed. The most common case (by which is mean 99.9999%) is that there are say < 5 domains for an app. So, all this code and state tracking to reduce some API requests? It's not worth the complexity and it's not like we are generating some 1000 requests. Also, think about all the UI buttons and error messages that now need to be created just for this. You need a button to say "continue" , messages that show what exactly went wrong and what domains did not update and why exactly and so on.

                                  robiR Offline
                                  robiR Offline
                                  robi
                                  wrote on last edited by
                                  #16

                                  @girish Thanks! Will test ;-]

                                  Q: Why not trust the last state?

                                  You already have a button to re-sync DNS in case that becomes an issue.

                                  Q: Why do you need any new UI buttons or error messages when it's "handled" in the background transparently?

                                  Isn't it iterative retry to eventual completion even if failures occur?

                                  Conscious tech

                                  girishG 1 Reply Last reply
                                  0
                                  • robiR robi

                                    @girish Thanks! Will test ;-]

                                    Q: Why not trust the last state?

                                    You already have a button to re-sync DNS in case that becomes an issue.

                                    Q: Why do you need any new UI buttons or error messages when it's "handled" in the background transparently?

                                    Isn't it iterative retry to eventual completion even if failures occur?

                                    girishG Offline
                                    girishG Offline
                                    girish
                                    Staff
                                    wrote on last edited by
                                    #17

                                    @robi said in Improve DNS updates to avoid rate limits:

                                    Q: Why not trust the last state?

                                    You already have a button to re-sync DNS in case that becomes an issue.

                                    Maybe I am not completely understanding. Let's say you have 5 domains right now and say 2 failed . The app now says 'Running', but I also assume it has show some message or error somewhere that 2 failed. Otherwise, people will say it didn't really succeed. To track that 2 failed, we have to maintain state. But ok, let's say we ignore this. User adds a domain. Now, we only try this new domain. The 2 failed are silently ignored. And maybe user adds more and more failures add up all silently ignored. There's no way to know what has succeeded and what has failed unless we maintain state.

                                    robiR 1 Reply Last reply
                                    0
                                    • girishG girish

                                      @robi said in Improve DNS updates to avoid rate limits:

                                      Q: Why not trust the last state?

                                      You already have a button to re-sync DNS in case that becomes an issue.

                                      Maybe I am not completely understanding. Let's say you have 5 domains right now and say 2 failed . The app now says 'Running', but I also assume it has show some message or error somewhere that 2 failed. Otherwise, people will say it didn't really succeed. To track that 2 failed, we have to maintain state. But ok, let's say we ignore this. User adds a domain. Now, we only try this new domain. The 2 failed are silently ignored. And maybe user adds more and more failures add up all silently ignored. There's no way to know what has succeeded and what has failed unless we maintain state.

                                      robiR Offline
                                      robiR Offline
                                      robi
                                      wrote on last edited by
                                      #18

                                      @girish You're right, I think not.

                                      Currently the app is NOT in a 'Running' state if any failures occur.

                                      An alternate method would be to bring up the app anyway (as long as the primary subdomain is up, and continue to resolve the failed API calls until some 'threshold' (then alert) or it succeeds and all is well.

                                      No need to ignore the failed ones, they just try again async as with your API rate backoff code, or at the next addition of a new sub if you want to be less proactive and more lazy. (The non working subs, if used will draw attention back to the app anyway.)

                                      Conscious tech

                                      1 Reply Last reply
                                      0
                                      Reply
                                      • Reply as topic
                                      Log in to reply
                                      • Oldest to Newest
                                      • Newest to Oldest
                                      • Most Votes


                                      • Login

                                      • Don't have an account? Register

                                      • Login or register to search.
                                      • First post
                                        Last post
                                      0
                                      • Categories
                                      • Recent
                                      • Tags
                                      • Popular
                                      • Bookmarks
                                      • Search