Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Invalid response code when fetching directory : 429

    Support
    6
    23
    704
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • d19dotca
      d19dotca last edited by d19dotca

      I receive the subject error when Cloudron is trying to renew certificates.

      Invalid response code when fetching directory : 429

      I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

      --
      Dustin Dauncey
      www.d19.ca

      girish P 2 Replies Last reply Reply Quote 1
      • girish
        girish Staff @d19dotca last edited by

        @d19dotca can you paste the full logs? Usually it says what the response code was.

        d19dotca 1 Reply Last reply Reply Quote 0
        • d19dotca
          d19dotca @girish last edited by d19dotca

          @girish Yes, sorry, I meant to do that but filed it from my phone earlier so couldn't easily do that. 😛 I'm on my computer now and have found the following task logs for renewing this one particular certificate (all others are successful, only this one fails):

          2021-03-12T12:00:01.717Z box:tasks 8971: {"percent":7,"message":"Renewing certs of www.staging.<subdomain>.<domain>.<tld>"}
          2021-03-12T12:00:01.719Z box:domains Unable to read fallback certificates of <domain>.<tld> from disk
          2021-03-12T12:00:01.724Z box:reverseproxy ensureCertificate: www.staging.<subdomain>.<domain>.<tld> certificate already exists at /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.key
          2021-03-12T12:00:01.742Z box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.cert Certificate will expire 1
          2021-03-12T12:00:01.742Z box:reverseproxy ensureCertificate: www.staging.<subdomain>.<domain>.<tld> cert requires renewal
          2021-03-12T12:00:01.742Z box:reverseproxy ensureCertificate: getting certificate for www.staging.<subdomain>.<domain>.<tld> with options {"prod":true,"performHttpAuthorization":true,"wildcard":false,"email":"<emailAddress>"}
          2021-03-12T12:00:01.743Z box:cert/acme2 getCertificate: attempt 1
          2021-03-12T12:00:01.743Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
          2021-03-12T12:00:02.783Z box:cert/acme2 getCertificate: using existing acme account key
          2021-03-12T12:00:02.891Z box:cert/acme2 registerUser: registering user
          2021-03-12T12:00:04.181Z box:cert/acme2 sendSignedRequest: using nonce 0003bACthgA3dch1bIZAplagmGDezb3NMnkqqOYbUeTlw8o for url https://acme-v02.api.letsencrypt.org/acme/new-acct
          2021-03-12T12:00:04.914Z box:cert/acme2 registerUser: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/59537731
          2021-03-12T12:00:04.914Z box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/59537731 email: <emailAddress>
          2021-03-12T12:00:05.188Z box:cert/acme2 getCertificate: attempt 2
          2021-03-12T12:00:05.188Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
          2021-03-12T12:00:05.460Z box:cert/acme2 getCertificate: attempt 3
          2021-03-12T12:00:05.460Z box:cert/acme2 getCertificate: start acme flow for www.staging.<subdomain>.<domain>.<tld> from https://acme-v02.api.letsencrypt.org/directory
          2021-03-12T12:00:05.713Z box:reverseproxy ensureCertificate: error: Invalid response code when fetching directory : 429 cert: null
          2021-03-12T12:00:05.740Z box:reverseproxy isExpiringSync: /home/yellowtent/boxdata/certs/www.staging.<subdomain>.<domain>.<tld>.cert Certificate will not expire 0
          2021-03-12T12:00:05.740Z box:reverseproxy ensureCertificate: continue using existing bundle since renewal failed
          

          --
          Dustin Dauncey
          www.d19.ca

          1 Reply Last reply Reply Quote 0
          • girish
            girish Staff last edited by

            This is probably a temporary error. Do you see this all the time?

            d19dotca 1 Reply Last reply Reply Quote 0
            • d19dotca
              d19dotca @girish last edited by

              @girish I thought it might be too but it’s been having for about the last 36 hours (I’ve had three failures on it so far, with it trying every 12 hours).

              --
              Dustin Dauncey
              www.d19.ca

              1 Reply Last reply Reply Quote 0
              • M
                msbt App Dev last edited by

                just got the same error the first time on 2 different cloudrons (one is still a v6.0.0, the other a v6.2.4)

                imc67 1 Reply Last reply Reply Quote 0
                • imc67
                  imc67 translator @msbt last edited by

                  @msbt @girish just received an email from one of my 4 Cloudron Premiums:

                  Dear Cloudron Admin,

                  The certificate for my.domain.tld could not be renewed.

                  The Cloudron will attempt to renew the certificate every 12 hours
                  until the certificate expires (at which point it will switch to
                  using the fallback certificate).

                  See https://docs.cloudron.io/troubleshooting/#certificates to
                  double check if your server is configured correctly to obtain certificates
                  via Let's Encrypt.

                  The error was:


                  Invalid response code when fetching nonce : 429


                  1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff last edited by

                    Do you all still see the errors still? If so, can you please write to support@ and give me access to check what might be happenning ?

                    I tried to debug this on one other customer's server but the problem seems to have gone away atleast for their domain. It also looks like these errors come from using Wildcard/Manual DNS (and thus http based authorization + non-wildcard certs). Is that the case for you all as well?

                    When debugging, I noticed that the "Renew all certs" button is br0ken 😕 Guess, we will put a fix into the next patch release.

                    d19dotca imc67 2 Replies Last reply Reply Quote 0
                    • d19dotca
                      d19dotca @girish last edited by

                      @girish It seems like it recently just resolved itself too in my system, the latest renewal logs seem to indicate it was successful now and I don’t have any failure notifications today. Maybe it was a Let’s Encrypt issue then? Seems weird though.

                      --
                      Dustin Dauncey
                      www.d19.ca

                      1 Reply Last reply Reply Quote 0
                      • imc67
                        imc67 translator @girish last edited by

                        @girish said in Invalid response code when fetching directory : 429:

                        Do you all still see the errors still?

                        It seems it solved itself, the error email was almost 24 hours ago and I don't see errors in the latest log records.

                        1 Reply Last reply Reply Quote 0
                        • P
                          p44 translator @d19dotca last edited by

                          @d19dotca said in Invalid response code when fetching directory : 429:

                          I receive the subject error when Cloudron is trying to renew certificates.

                          Invalid response code when fetching directory : 429

                          I’ve searched but found nothing on this so far. Any ideas what is going on with this behaviour? It’s only happening on one particular app/sub-domain so far.

                          I'm having same issue right now

                          1 Reply Last reply Reply Quote 0
                          • d19dotca
                            d19dotca last edited by d19dotca

                            I just had the same issue again on a different domain. Twice in a row 12 hours apart. Can’t tell if this is a Cloudron issue or a Let’s Encrypt issue. I see one domain with the original error, and a second domain with a different error (that I believe I also saw posted recently elsewhere in this forum).

                            C28E8C54-7529-4042-893A-2A13C9F89F4D.jpeg

                            PS - I see that the text is wrong too… I think it should read “failed to renew” not “failed to new”.


                            Edit: The one that was just "status: invalid" seems to have been resolved. Still receiving the original 429 error though on the other domain. It doesn't help that I can't use the UI to renew certs in 6.2.4, so I can't manually force it to get fresher logs, I am stuck waiting for 12 hours each time. 😞

                            --
                            Dustin Dauncey
                            www.d19.ca

                            girish 1 Reply Last reply Reply Quote 1
                            • girish
                              girish Staff @d19dotca last edited by

                              @d19dotca I fixed the typo. As for the error itself, from what I have seen this seems to go away magically.

                              Looks like we have to fix our notification to not be so aggressive. Maybe we should inform user of this error only if it happens say 10 days up to expiry or something. Cloudron starts renewing certs a month in advance and it seems to make people panic...

                              d19dotca 2 Replies Last reply Reply Quote 1
                              • d19dotca
                                d19dotca @girish last edited by

                                @girish Yeah totally fair enough. It's a Let's Encrypt issue that we're comfortable will resolve itself (and it seems that's the behaviour we've seen over time too) then maybe the only change needed in Cloudron is to make it less aggressive. Perhaps start renewing 30 days in advance as it does already, but don't notify of any problems until maybe 15 or 20 days in advance? That way presumably it'd be an issue that's occurring for roughly 10 days prior repeatedly that wasn't yet resolved if we see any notifications like that? Hopefully that makes sense.

                                I like your idea, I think less aggressive notifications for issues we know tend to resolve themselves over time would be helpful. And more importantly it'll save a bit of time on your end in less posts 😉 haha

                                --
                                Dustin Dauncey
                                www.d19.ca

                                1 Reply Last reply Reply Quote 0
                                • d19dotca
                                  d19dotca @girish last edited by d19dotca

                                  @girish Just found this too: https://bobcares.com/blog/lets-encrypt-new-auth-status-429-error/ - Seems like one possible cause of this is too many subdomains in Cloudron for a single domain, and Let's Encrypt limiting how many are given out for the domain each week. In which case less aggressive notifications is probably a great change to be made. 🙂

                                  I wonder if a second improvement here could be to show the actual Let's Encrypt response in its totality, rather than stripped down in Cloudron? That may be helpful so people at least know if the issue is in Cloudron or coming from Let's Encrypt's side.

                                  --
                                  Dustin Dauncey
                                  www.d19.ca

                                  girish 1 Reply Last reply Reply Quote 0
                                  • girish
                                    girish Staff @d19dotca last edited by

                                    @d19dotca On Cloudron, this seems to happen when querying the public URL - https://acme-v02.api.letsencrypt.org/directory . That links provides a directory map of URLs (instead of hardcoding the URLs it in the code base). For some reason that returns a 429. So, it's not related to cert limits or account limits.

                                    1 Reply Last reply Reply Quote 0
                                    • girish
                                      girish Staff last edited by

                                      If i see https://tools.ietf.org/html/rfc8555#page-23, there is no 429 response code.

                                      1 Reply Last reply Reply Quote 0
                                      • robi
                                        robi last edited by

                                        I am getting this message on a few domains as well.

                                        Life of Advanced Technology

                                        1 Reply Last reply Reply Quote 0
                                        • girish
                                          girish Staff last edited by

                                          Does curl https://acme-v02.api.letsencrypt.org/directory return an error on the server?

                                          robi 1 Reply Last reply Reply Quote 0
                                          • robi
                                            robi @girish last edited by robi

                                            @girish no, all come back with data.

                                            What about setting up a time to run these curls around the same time cloudron does (and log it), which may be when LE does something on their systems.

                                            Also shifting the time cloudron does it may be good. There's hundreds of cloudrons hammering them at midnight for even more domains 😉

                                            Life of Advanced Technology

                                            1 Reply Last reply Reply Quote 0
                                            • girish
                                              girish Staff last edited by

                                              For lack of ideas, I have added a retry now when fetching directory.

                                              imc67 1 Reply Last reply Reply Quote 1
                                              • imc67
                                                imc67 translator @girish last edited by

                                                @girish on one Cloudron I already get for the third time in a row (so each time after 12 hours) these below errors, usually it was only one time. Since this moring (so after the second and before the third) I updated Cloudron to 6.2.7:

                                                The error was:
                                                
                                                -------------------------------------
                                                
                                                Invalid response code when fetching nonce : 429
                                                
                                                -------------------------------------
                                                
                                                
                                                Powered by https://cloudron.io
                                                
                                                Sent at: Mon, 29 Mar 2021 12:00:07 GMT
                                                
                                                The error was:
                                                
                                                -------------------------------------
                                                
                                                Unexpected status: invalid
                                                
                                                -------------------------------------
                                                
                                                
                                                Powered by https://cloudron.io
                                                
                                                Sent at: Tue, 30 Mar 2021 00:05:38 GMT
                                                
                                                The error was:
                                                
                                                -------------------------------------
                                                
                                                Invalid response code when fetching nonce : 429
                                                
                                                -------------------------------------
                                                
                                                
                                                Powered by https://cloudron.io
                                                
                                                Sent at: Tue, 30 Mar 2021 12:00:09 GMT
                                                
                                                girish 1 Reply Last reply Reply Quote 0
                                                • girish
                                                  girish Staff @imc67 last edited by

                                                  @imc67 in the next release, we will suppress those notifications. it seems to not add much value. in any case, the solution is just to wait since cloudron will try again later. it's not a Cloudron issue, the LE service just goes up/down by nature.

                                                  1 Reply Last reply Reply Quote 0
                                                  • First post
                                                    Last post
                                                  Powered by NodeBB