Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Invalid response code when fetching directory : 429

Scheduled Pinned Locked Moved Solved Support
23 Posts 6 Posters 766 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by d19dotca
    #1

    I receive the subject error when Cloudron is trying to renew certificates.

    Invalid response code when fetching directory : 429

    I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

    --
    Dustin Dauncey
    www.d19.ca

    girishG P 2 Replies Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    replied to d19dotca on last edited by
    #2

    @d19dotca can you paste the full logs? Usually it says what the response code was.

    d19dotcaD 1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by d19dotca
    #3

    @girish Yes, sorry, I meant to do that but filed it from my phone earlier so couldn't easily do that. 😛 I'm on my computer now and have found the following task logs for renewing this one particular certificate (all others are successful, only this one fails):

    2021-03-12T12:00:01.717Z box:tasks 8971: {"percent":7,"message":"Renewing certs of www.staging.<subdomain>.<domain>.<tld>"}
    2021-03-12T12:00:01.719Z box:domains Unable to read fallback certificates of <domain>.<tld> from disk
    2021-03-12T12:00:01.724Z box:reverseproxy ensureCertificate: www.staging.<subdomain>.<domain>.<tld> certificate already exists at /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.key
    2021-03-12T12:00:01.742Z box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.cert Certificate will expire 1
    2021-03-12T12:00:01.742Z box:reverseproxy ensureCertificate: www.staging.<subdomain>.<domain>.<tld> cert requires renewal
    2021-03-12T12:00:01.742Z box:reverseproxy ensureCertificate: getting certificate for www.staging.<subdomain>.<domain>.<tld> with options {"prod":true,"performHttpAuthorization":true,"wildcard":false,"email":"<emailAddress>"}
    2021-03-12T12:00:01.743Z box:cert/acme2 getCertificate: attempt 1
    2021-03-12T12:00:01.743Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
    2021-03-12T12:00:02.783Z box:cert/acme2 getCertificate: using existing acme account key
    2021-03-12T12:00:02.891Z box:cert/acme2 registerUser: registering user
    2021-03-12T12:00:04.181Z box:cert/acme2 sendSignedRequest: using nonce 0003bACthgA3dch1bIZAplagmGDezb3NMnkqqOYbUeTlw8o for url https://acme-v02.api.letsencrypt.org/acme/new-acct
    2021-03-12T12:00:04.914Z box:cert/acme2 registerUser: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/59537731
    2021-03-12T12:00:04.914Z box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/59537731 email: <emailAddress>
    2021-03-12T12:00:05.188Z box:cert/acme2 getCertificate: attempt 2
    2021-03-12T12:00:05.188Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
    2021-03-12T12:00:05.460Z box:cert/acme2 getCertificate: attempt 3
    2021-03-12T12:00:05.460Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
    2021-03-12T12:00:05.713Z box:reverseproxy ensureCertificate: error: Invalid response code when fetching directory : 429 cert: null
    2021-03-12T12:00:05.740Z box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.cert Certificate will not expire 0
    2021-03-12T12:00:05.740Z box:reverseproxy ensureCertificate: continue using existing bundle since renewal failed
    

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #4

    This is probably a temporary error. Do you see this all the time?

    d19dotcaD 1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by
    #5

    @girish I thought it might be too but it’s been having for about the last 36 hours (I’ve had three failures on it so far, with it trying every 12 hours).

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • M Offline
    M Offline
    msbt App Dev
    wrote on last edited by
    #6

    just got the same error the first time on 2 different cloudrons (one is still a v6.0.0, the other a v6.2.4)

    imc67I 1 Reply Last reply
    0
  • imc67I Offline
    imc67I Offline
    imc67 translator
    replied to msbt on last edited by
    #7

    @msbt @girish just received an email from one of my 4 Cloudron Premiums:

    Dear Cloudron Admin,

    The certificate for my.domain.tld could not be renewed.

    The Cloudron will attempt to renew the certificate every 12 hours
    until the certificate expires (at which point it will switch to
    using the fallback certificate).

    See https://docs.cloudron.io/troubleshooting/#certificates to
    double check if your server is configured correctly to obtain certificates
    via Let's Encrypt.

    The error was:


    Invalid response code when fetching nonce : 429


    1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #8

    Do you all still see the errors still? If so, can you please write to support@ and give me access to check what might be happenning ?

    I tried to debug this on one other customer's server but the problem seems to have gone away atleast for their domain. It also looks like these errors come from using Wildcard/Manual DNS (and thus http based authorization + non-wildcard certs). Is that the case for you all as well?

    When debugging, I noticed that the "Renew all certs" button is br0ken 😕 Guess, we will put a fix into the next patch release.

    d19dotcaD imc67I 2 Replies Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by
    #9

    @girish It seems like it recently just resolved itself too in my system, the latest renewal logs seem to indicate it was successful now and I don’t have any failure notifications today. Maybe it was a Let’s Encrypt issue then? Seems weird though.

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • imc67I Offline
    imc67I Offline
    imc67 translator
    replied to girish on last edited by
    #10

    @girish said in Invalid response code when fetching directory : 429:

    Do you all still see the errors still?

    It seems it solved itself, the error email was almost 24 hours ago and I don't see errors in the latest log records.

    1 Reply Last reply
    0
  • P Offline
    P Offline
    p44 translator
    replied to d19dotca on last edited by
    #11

    @d19dotca said in Invalid response code when fetching directory : 429:

    I receive the subject error when Cloudron is trying to renew certificates.

    Invalid response code when fetching directory : 429

    I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

    I'm having same issue right now

    1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by d19dotca
    #12

    I just had the same issue again on a different domain. Twice in a row 12 hours apart. Can’t tell if this is a Cloudron issue or a Let’s Encrypt issue. I see one domain with the original error, and a second domain with a different error (that I believe I also saw posted recently elsewhere in this forum).

    C28E8C54-7529-4042-893A-2A13C9F89F4D.jpeg

    PS - I see that the text is wrong too… I think it should read “failed to renew” not “failed to new”.


    Edit: The one that was just "status: invalid" seems to have been resolved. Still receiving the original 429 error though on the other domain. It doesn't help that I can't use the UI to renew certs in 6.2.4, so I can't manually force it to get fresher logs, I am stuck waiting for 12 hours each time. 😞

    --
    Dustin Dauncey
    www.d19.ca

    girishG 1 Reply Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    replied to d19dotca on last edited by
    #13

    @d19dotca I fixed the typo. As for the error itself, from what I have seen this seems to go away magically.

    Looks like we have to fix our notification to not be so aggressive. Maybe we should inform user of this error only if it happens say 10 days up to expiry or something. Cloudron starts renewing certs a month in advance and it seems to make people panic...

    d19dotcaD 2 Replies Last reply
    1
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by
    #14

    @girish Yeah totally fair enough. It's a Let's Encrypt issue that we're comfortable will resolve itself (and it seems that's the behaviour we've seen over time too) then maybe the only change needed in Cloudron is to make it less aggressive. Perhaps start renewing 30 days in advance as it does already, but don't notify of any problems until maybe 15 or 20 days in advance? That way presumably it'd be an issue that's occurring for roughly 10 days prior repeatedly that wasn't yet resolved if we see any notifications like that? Hopefully that makes sense.

    I like your idea, I think less aggressive notifications for issues we know tend to resolve themselves over time would be helpful. And more importantly it'll save a bit of time on your end in less posts 😉 haha

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by d19dotca
    #15

    @girish Just found this too: https://bobcares.com/blog/lets-encrypt-new-auth-status-429-error/ - Seems like one possible cause of this is too many subdomains in Cloudron for a single domain, and Let's Encrypt limiting how many are given out for the domain each week. In which case less aggressive notifications is probably a great change to be made. 🙂

    I wonder if a second improvement here could be to show the actual Let's Encrypt response in its totality, rather than stripped down in Cloudron? That may be helpful so people at least know if the issue is in Cloudron or coming from Let's Encrypt's side.

    --
    Dustin Dauncey
    www.d19.ca

    girishG 1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    replied to d19dotca on last edited by
    #16

    @d19dotca On Cloudron, this seems to happen when querying the public URL - https://acme-v02.api.letsencrypt.org/directory . That links provides a directory map of URLs (instead of hardcoding the URLs it in the code base). For some reason that returns a 429. So, it's not related to cert limits or account limits.

    1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #17

    If i see https://tools.ietf.org/html/rfc8555#page-23, there is no 429 response code.

    1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by
    #18

    I am getting this message on a few domains as well.

    Life of sky tech

    1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #19

    Does curl https://acme-v02.api.letsencrypt.org/directory return an error on the server?

    robiR 1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    replied to girish on last edited by robi
    #20

    @girish no, all come back with data.

    What about setting up a time to run these curls around the same time cloudron does (and log it), which may be when LE does something on their systems.

    Also shifting the time cloudron does it may be good. There's hundreds of cloudrons hammering them at midnight for even more domains 😉

    Life of sky tech

    1 Reply Last reply
    0

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.