DNS - DigitalOcean - ERROR 429 - Too Many Requests

BrutalBirdie

2023-07-24T10:18:01.235Z box:settings initCache: pre-load settings
2023-07-24T10:18:01.249Z box:taskworker Starting task 738. Logs are at /home/yellowtent/platformdata/logs/tasks/738.log
2023-07-24T10:18:01.258Z box:tasks update 738: {"percent":1,"message":"Updating DNS of DOMAIN.TLD"}
2023-07-24T10:18:01.259Z box:dns registerLocations: Will register [{"subdomain":"my","domain":"DOMAIN.TLD"},{"subdomain":"mail","domain":"DOMAIN.TLD"},{"subdomain":"sogo","domain":"DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"mail","certificate":null,"fqdn":"mail.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"webmail","certificate":null,"fqdn":"webmail.DOMAIN.TLD"},{"subdomain":"short","domain":"DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"k","certificate":null,"fqdn":"k.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"kurz","certificate":null,"fqdn":"kurz.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"kutt","certificate":null,"fqdn":"kutt.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"s","certificate":null,"fqdn":"s.DOMAIN.TLD"},{"subdomain":"vault","domain":"DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"pass","certificate":null,"fqdn":"pass.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"passwoerter","certificate":null,"fqdn":"passwoerter.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"password","certificate":null,"fqdn":"password.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"passwords","certificate":null,"fqdn":"passwords.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"passwort","certificate":null,"fqdn":"passwort.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"pw","certificate":null,"fqdn":"pw.DOMAIN.TLD"},{"subdomain":"collabora","domain":"DOMAIN.TLD"},{"subdomain":"roundcube","domain":"DOMAIN.TLD"},{"subdomain":"nextcloud","domain":"DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"cloud","certificate":null,"fqdn":"cloud.DOMAIN.TLD"},{"subdomain":"meet","domain":"DOMAIN.TLD"},{"subdomain":"stats","domain":"DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"matomo","certificate":null,"fqdn":"matomo.DOMAIN.TLD"},{"subdomain":"oc-test1","domain":"DOMAIN.TLD"},{"subdomain":"hedgedoc","domain":"DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"pad","certificate":null,"fqdn":"pad.DOMAIN.TLD"},{"domain":"DOMAIN.TLD","subdomain":"text","certificate":null,"fqdn":"text.DOMAIN.TLD"},{"subdomain":"chat","domain":"DOMAIN.TLD"}] with options {"overwriteDns":true}
2023-07-24T10:18:01.262Z box:sysinfo/generic getServerIPv4: querying ipv4.api.cloudron.io to get server IPv4
2023-07-24T10:18:01.665Z box:sysinfo/generic getServerIPv6: querying ipv6.api.cloudron.io to get server IPv6
2023-07-24T10:18:02.018Z box:tasks update 738: {"message":"Registering location: my.DOMAIN.TLD"}
2023-07-24T10:18:02.023Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with my and type CNAME
2023-07-24T10:18:02.846Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with my and type A
2023-07-24T10:18:03.516Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with my and type AAAA
2023-07-24T10:18:04.164Z box:tasks update 738: {"message":"Registering location: mail.DOMAIN.TLD"}
2023-07-24T10:18:04.165Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with mail and type CNAME
2023-07-24T10:18:04.837Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with mail and type A
2023-07-24T10:18:05.566Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with mail and type AAAA
2023-07-24T10:18:06.269Z box:tasks update 738: {"message":"Registering location: sogo.DOMAIN.TLD"}
2023-07-24T10:18:06.271Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with sogo and type CNAME
2023-07-24T10:18:06.929Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with sogo and type A
2023-07-24T10:18:07.574Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with sogo and type AAAA
2023-07-24T10:18:08.247Z box:tasks update 738: {"message":"Registering location: mail.DOMAIN.TLD"}
2023-07-24T10:18:08.249Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with mail and type CNAME
2023-07-24T10:18:08.895Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with mail and type A
2023-07-24T10:18:09.786Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with mail and type AAAA
2023-07-24T10:18:10.691Z box:tasks update 738: {"message":"Registering location: webmail.DOMAIN.TLD"}
2023-07-24T10:18:10.692Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with webmail and type CNAME
2023-07-24T10:18:11.358Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with webmail and type A
2023-07-24T10:18:12.268Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with webmail and type AAAA
2023-07-24T10:18:13.007Z box:tasks update 738: {"message":"Registering location: short.DOMAIN.TLD"}
2023-07-24T10:18:13.008Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with short and type CNAME
2023-07-24T10:18:13.851Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with short and type A
2023-07-24T10:18:14.502Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with short and type AAAA
2023-07-24T10:18:15.129Z box:tasks update 738: {"message":"Registering location: k.DOMAIN.TLD"}
2023-07-24T10:18:15.130Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with k and type CNAME
2023-07-24T10:18:15.812Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with k and type A
2023-07-24T10:18:16.486Z box:dns/digitalocean getInternal: getting dns records of DOMAIN.TLD with k and type AAAA
2023-07-24T10:18:17.534Z box:dns registerLocation: Get error. retryable: true. DigitalOcean DNS error 429 {"id":"Too Many Requests","message":"Too many requests"}
2023-07-24T10:18:17.535Z box:tasks setCompleted - 738: {"result":{"errors":[{"domain":"DOMAIN.TLD","message":"DigitalOcean DNS error 429 {\"id\":\"Too Many Requests\",\"message\":\"Too many requests\"}"}]},"error":null}
2023-07-24T10:18:17.535Z box:tasks update 738: {"percent":100,"result":{"errors":[{"domain":"DOMAIN.TLD","message":"DigitalOcean DNS error 429 {\"id\":\"Too Many Requests\",\"message\":\"Too many requests\"}"}]},"error":null}
2023-07-24T10:18:17.535Z box:taskworker Task took 16.346 seconds

I noticed this on multiple Cloudron instances that use DO DNS when syncing DNS records.
This got noticed after a customer fiddled with the SPF record for some reason and me wanting to re-sync the DNS.

nebulon

how many domains or rather DNS records are we roughly talking about? Maybe DO has changed their api rate-limits recently

BrutalBirdie

15x apps / subdomains so 30x Domain Records since A and AAAA records.
To that coming txt records for mail and all.

nebulon

The rate-limit seem a lot higher than this https://docs.digitalocean.com/reference/api/api-reference/#section/Introduction/Rate-Limit

When did this start and since you mentioned multiple Cloudrons, are those using maybe the same api token?

BrutalBirdie

The setups have a mixed setup of domains actually.
3-4 Domains with manual DNS config and 1x domain with DigitalOcean.
And the processed records in the top log are from the main Cloudron domain which is manual DNS.
So its even more strange that he fails there with DO when its not even used yet.

MISTAKE! Was locking at the wrong tab

So the instance has one 1x Domain with DigitalOcean.
The DO Team uses nothing but the DNS feature.
Only one token active.

So this really does not make much sense.

System Information:
Cloudron: v7.4.3
Ubuntu 22.04

BrutalBirdie

Opened a support ticket with DO since this makes no sense at all.

BrutalBirdie

The support simply shrugged it of as "check on your side that you don't do more then 250 requests a minute and then it should work".

I am drilling a bit.

BrutalBirdie

Update from DO > @staff :

Hello Elias,

Thank you for your patience.

We reviewed the backend logs and noticed that when the requests are sent to "/v2/domains/DOMAIN.TLD/records" and rate limit message shows up "rate_limiter_burst_blocked", however, I am not able to exactly verify the number of requests your API token is making per minute.

I am checking this with our internal team, but in the meantime, I would request you to set up exponential back-off or delay between requests to avoid hitting these limits and flooding the API with requests. If your requests can be spaced out over multiple minutes, you shouldn't hit the limit.

For more information and insights, check out this article here:

https://dzone.com/articles/understanding-retry-pattern-with-exponential-back

Here is an example that lists a few scripts on how you can implement an exponential backoff:

https://cloud.google.com/iot/docs/how-tos/exponential-backoff

In the meantime, if you have any additional questions, please do not hesitate to ask. We're always happy to help whenever we can.

girish

@BrutalBirdie how many subdomains/records are there in that domain ?

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

DNS - DigitalOcean - ERROR 429 - Too Many Requests