Throttling issue (Error 429) with DynDNS auto update and desec.io - Avoidable?

[Teiluj]

One of my Cloudron servers using a ISP with no fix public IP (The public IP provided by the ISP rotates fairly regularly.)
The Cloudron server is set to use dynamic DNS and the domain name is configured with desec.io.

The issue comes from the fact that the overnight DNS update attempt by cloudron @ desec.io is partially throttled Returned error 429), resulting in some apps being accessible, some other not, including sometime the cloudron dashboard (my.).

Here is the anonymized log entry from the server:

Task 1061
Aug 21 04:03:01 taskworker: Starting task 1061. Logs are at /home/yellowtent/platformdata/logs/tasks/1061.log
Aug 21 04:03:01 taskworker: Running task of type syncDyndns
Aug 21 04:03:01 tasks: updating task 1061 with: {"percent":5,"message":"Updating dashboard location my.url-of-cloudron-server.here"}
Aug 21 04:03:01 dns: upsertDnsRecords: subdomain:my domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:05 tasks: updating task 1061 with: {"percent":15,"message":"Updating mail location my.url-of-cloudron-server.here"}
Aug 21 04:03:05 tasks: updating task 1061 with: {"percent":22,"message":"Updating app app01.url-of-cloudron-server.here"}
Aug 21 04:03:05 dns: upsertDnsRecords: subdomain:media domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:07 tasks: updating task 1061 with: {"percent":29,"message":"Updating app app02.url-of-cloudron-server.here"}
Aug 21 04:03:07 dns: upsertDnsRecords: subdomain:cron domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:09 tasks: updating task 1061 with: {"percent":36,"message":"Updating app app03.url-of-cloudron-server.here"}
Aug 21 04:03:09 dns: upsertDnsRecords: subdomain:change domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:11 tasks: updating task 1061 with: {"percent":43,"message":"Updating app app04.url-of-cloudron-server.here"}
Aug 21 04:03:11 dns: upsertDnsRecords: subdomain:timeline domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:13 tasks: updating task 1061 with: {"percent":50,"message":"Updating app app05.url-of-cloudron-server.here"}
Aug 21 04:03:13 dns: upsertDnsRecords: subdomain:bookmarks domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:16 tasks: updating task 1061 with: {"percent":57,"message":"Updating app app06.url-of-cloudron-server.here"}
Aug 21 04:03:16 dns: upsertDnsRecords: subdomain:vpn domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:18 tasks: updating task 1061 with: {"percent":64,"message":"Updating app app07.url-of-cloudron-server.here"}
Aug 21 04:03:18 dns: upsertDnsRecords: subdomain:paperless domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:20 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 44 seconds."}
    at Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:108:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:20 tasks: updating task 1061 with: {"percent":71,"message":"Updating app app08.url-of-cloudron-server.here"}
Aug 21 04:03:20 dns: upsertDnsRecords: subdomain:app08 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:21 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 43 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:21 dns: upsertDnsRecords: subdomain:app08alias01 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:22 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 42 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:22 tasks: updating task 1061 with: {"percent":78,"message":"Updating app app09.url-of-cloudron-server.here"}
Aug 21 04:03:22 dns: upsertDnsRecords: subdomain:app09 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:23 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 41 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:23 dns: upsertDnsRecords: subdomain:app09alias01 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:24 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 40 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:24 dns: upsertDnsRecords: subdomain:app09alias02 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:25 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 39 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:25 tasks: updating task 1061 with: {"percent":85,"message":"Updating app app10.url-of-cloudron-server.here"}
Aug 21 04:03:25 dns: upsertDnsRecords: subdomain:app10 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:26 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 38 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:26 dns: upsertDnsRecords: subdomain:app10alias01 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:27 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 37 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:27 tasks: updating task 1061 with: {"percent":92,"message":"Updating app app11.url-of-cloudron-server.here"}
Aug 21 04:03:27 dns: upsertDnsRecords: subdomain:app11 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:28 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 36 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:28 dns: upsertDnsRecords: subdomain:app11alias01 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:29 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 35 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:29 tasks: updating task 1061 with: {"percent":99,"message":"Updating app app12.url-of-cloudron-server.here"}
Aug 21 04:03:29 dns: upsertDnsRecords: subdomain:app12 domain:url-of-cloudron-server.here type:A values:["AAA.BBB.CCC.DDD"]
Aug 21 04:03:30 dyndns: BoxError: deSEC DNS error [429] {"detail":"Request was throttled. Expected available in 34 seconds."}
    at del (file:///home/yellowtent/box/src/dns/desec.js:76:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async Object.upsert (file:///home/yellowtent/box/src/dns/desec.js:89:5)
    at async Object.upsertDnsRecords (file:///home/yellowtent/box/src/dns.js:146:5) {
  reason: 'External Error',
  details: {}
}
Aug 21 04:03:30 tasks: updating task 1061 with: {"percent":100,"message":"Done"}
Aug 21 04:03:30 tasks: updating task 1061 with: {"completed":true,"result":null,"error":null,"percent":100}
Aug 21 04:03:30 tasks: setCompleted - 1061: {"result":null,"error":null,"percent":100}
Aug 21 04:03:30 taskworker: Task took 29.632 seconds
Aug 21 04:03:30 Exiting with code 0

This is not a massive number of domain name records to update, yet it is still throttled.

Could this be due to Cloudron treating each subdomain as an independent DNS record and calling deSEC once per record, thus going over deSEC's rate limit ?

Could the DNS update from cloudron by pool into one API request? this would counts the whole batch as one rate-limited HTTP call, which would solve the issue of throttling.

To note:

• the same also happens when syncing DNS manually (pressing the sync DNS button from the domain page)

Additionally:

• according to the log, a returned error 429 does not count as an error for cloudron attempt (error":null,"percent":100)
• it would be great if, when a throttling answer is sent back, Cloudron could attempt another try after the throttling delay.
• when using DynDNS, It seems as cloudron only check for change of public IP once a day and does not take into account whether the update of records is successful or not and thus does not attempt further attempt to update in case of failure. It would be great if this wasn't the case and Cloudron would persist (a bit, more than once) in updating the records.

At the moment, this leaves the server regularly in an half limbo state with some apps reachable and some other not. So I am just trying to find a way around this while continuing supporting and using deSEC initiative

[milian.hackradt]

I'm afraid it's unavoidable. I remember that we wanted to use deSEC as our primary DNS provider, but ran into their rate limits fairly quickly a while ago when we were restoring a whole Cloudron instance. They are a non-profit registered association and are fully dependent on donations, so their limits are pretty low-ish. They won't adapt them for specific users (which is why we stopped using deSEC for SLA-critical domains), but you can ask them nicely and they can reset your limits - that helped us back then but that obviously isn't a feasible solution in your case.

[Teiluj]

Thanks @milian.hackradt

@milian.hackradt said:

They are a non-profit registered association and are fully dependent on donations, so their limits are pretty low-ish. They won't adapt them for specific users (which is why we stopped using deSEC for SLA-critical domains), but you can ask them nicely and they can reset your limits - that helped us back then but that obviously isn't a feasible solution in your case.

Yes - I was in touch with them too with regards to other situations, and Cloudron was also able to improve things to make deSEC record update more reliable.
I am hoping that this is the case also here.

@milian.hackradt said:

I'm afraid it's unavoidable.

I am not so sure of this - or at least in this case/scenario. There are a couple of things here which could be improved:

• Cloudron treat Error 429 as the error that it is (or at least not report is has a successful update) and make limited number of subsequent attempt(s) accordingly
• Cloudron could poll all DNS update to deSec in one HTTP API call (one per root domain) since deSEC's DynDNS update API accepts a comma-separated list of hostnames in a single request, and the whole batch counts as one rate-limited call. This will directly limit the number of necessary update request against deSEC servers making it a win-win situation.

So for now, I remain hopeful and I am looking forward to hearing from Cloudron staff on this.

[girish]

I have added a 429 check and waiting for "retry-after" in e532401510dc64570f5c0d3d9972429da6268e0f . I guess that's the best we can do on our side .

[Teiluj]

@girish - thanks for this. I am looking forward to try these changes.

Can I ask: has polling/batching DNS updates been considered or is this simply not possible in this situation?

Thanks again,

[Teiluj]

@girish said:

I have added a 429 check and waiting for "retry-after" in e532401510dc64570f5c0d3d9972429da6268e0f . I guess that's the best we can do on our side .

Hi @girish
Thanks for this - I appreciate your attempt in trying to help with this.
The related server experiencing the issue has been updated to v10.0.2 recently and it seems that this change was part of the update.

However, the patch has made the situation worse actually.

Since the update, I have been locked out of the related server because the "my." record did not successfully update (or only sporadically, before being scrapped from the DeSEC domain name record table entries shortly after)
Attempts to manually add the record to DeSEC entry were also impossible because the DeSEC account was also locked out behind never-ending permanently-resetting with growing threshold waiting times.

Not too sure how the Cloudron side is handling this though:

• It seems like the Cloudron DNS update requests are still batched. e.g. when one record needs updating, all records are updated, no matter if they actually need to.
• Still seem like Cloudron does not take into account whether or not a record was successfully update. e.g. do DNS record udpate -> if it works great; if it does not, carry on with the next record (thus leaving previous record in limbo)
• Update task starts at H=x and the next relevant entry is at approx H=x+8 (I suppose this could be due to the threshold retry waiting time)

At the moment, the errors on the Cloudron server log oscillate between:

• reason: 'External Error'
• reason: 'Database Error' (-> sqlMessage: 'Too many connections',)
• HTML: 429 too many request
• HTML: 502 Bad gateway

So currently, I feel this is a bit of a mess and I am sorry that this has come down to this.
Any idea how this could/should be dealt with?

I remain convinced that a more individual approach to DNS record update as well as switching DNS update method to batch HTTP API call would alleviate part of this/the whole issue i.e. only update records that needs updating, prioritze record update ("my." top prio)
But I also acknowledge that this seems very specific to DeSEC and might not fit in Cloudron's development plan

I hope that this makes sense.

Otherwise we will be obliged to move on from DeSEC.
@milian.hackradt May I ask what DN server are you using instead of deSEC?