Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. App Packaging & Development
  3. Could pull, but now cannot (private reg)

Could pull, but now cannot (private reg)

Scheduled Pinned Locked Moved Solved App Packaging & Development
15 Posts 3 Posters 1.3k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jadudmJ jadudm

      @robi I do.

      I think the error is somewhere in Cloudron.

      Feb 06 10:21:35 box:docker downloadImage: pulling image. attempt 10
      Feb 06 10:21:35 box:docker pullImage: will pull docker.host.us/jadudm/grocy:1.0.0. auth: yes
      

      I'll claim that the better behavior would be for Cloudron to fail when it cannot pull an image. Instead, it is trying to set up the extensions and run the container when the container doesn't exist.

      Given that it loops through 10 (failed) attempts to pull the image, there's something going wrong with the image pull. There's nothing in the existing logs that gives me enough information to figure out why I'm getting a 404 on the image. (I can pull the image from another machine, and both the dev machine and the Cloudron host are effectively on the same network.)

      I'll see if there's a way I can... I don't know. I feel like I'm missing something obvious, but I'd like a bit more info from the Cloudron logs in order to debug this.

      EDIT: It looks like

      https://git.cloudron.io/cloudron/box/-/blob/master/src/apptask.js#L354
      

      is where downloadImage is called, but I'm unclear where errors that are generated from that process are handled. It looks like a failed image download does just fall through to a continued attempt to run the container.

      jadudmJ Offline
      jadudmJ Offline
      jadudm
      wrote on last edited by jadudm
      #4

      When I attempt to update my Docker registry credentials in the Settings, I get the following error:

      (HTTP code 500) server error - Get https://docker.host.us/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 
      

      From the command line on my dev machine, I can

      curl -X GET -u jadudm:ASDFASDF https://docker.host.us/v2/_catalog
      

      and get a list of the images on the host. Hence why I am confused as to what may have changed under the hood that my private registry no longer works from Cloudron as an image installation source (when it did not that long ago).

      EDIT: In docker.js,

      https://git.cloudron.io/cloudron/box/-/blob/master/src/docker.js#L60
      

      The comment says what it says:

          const [error] = await safe(gConnection.checkAuth(config)); // this returns a 500 even for auth errors
      

      I have the correct username/password in my Registry conf. It never changed... but it stopped working at some point in Cloudron. It works just fine elsewhere. So, I don't know if this is a network issue, or an auth issue, or what... because it looks like checkAuth doesn't give me a lot of info to debug with?

      I use Cloudron on a Dell 7040 I bought on eBay.

      girishG 1 Reply Last reply
      0
      • jadudmJ jadudm

        When I attempt to update my Docker registry credentials in the Settings, I get the following error:

        (HTTP code 500) server error - Get https://docker.host.us/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 
        

        From the command line on my dev machine, I can

        curl -X GET -u jadudm:ASDFASDF https://docker.host.us/v2/_catalog
        

        and get a list of the images on the host. Hence why I am confused as to what may have changed under the hood that my private registry no longer works from Cloudron as an image installation source (when it did not that long ago).

        EDIT: In docker.js,

        https://git.cloudron.io/cloudron/box/-/blob/master/src/docker.js#L60
        

        The comment says what it says:

            const [error] = await safe(gConnection.checkAuth(config)); // this returns a 500 even for auth errors
        

        I have the correct username/password in my Registry conf. It never changed... but it stopped working at some point in Cloudron. It works just fine elsewhere. So, I don't know if this is a network issue, or an auth issue, or what... because it looks like checkAuth doesn't give me a lot of info to debug with?

        girishG Offline
        girishG Offline
        girish
        Staff
        wrote on last edited by
        #5

        @jadudm Missed this one somehow, did you get this sorted out?

        jadudmJ 2 Replies Last reply
        0
        • girishG girish

          @jadudm Missed this one somehow, did you get this sorted out?

          jadudmJ Offline
          jadudmJ Offline
          jadudm
          wrote on last edited by
          #6

          @girish I started using the DockerHub repo, and will circle back around to try and re-install the local Docker Registry. So, the answer is "sorta no." But I found a path to keep moving.

          I'm not sure why, given no changes on my part, that the local registry decided to stop working for app install by Cloudron. (All of these services are on the same host...) There were not disk/RAM issues that I could see, so something was going on, but I couldn't get enough info to debug beyond the fact that things retried-and-failed.

          I use Cloudron on a Dell 7040 I bought on eBay.

          1 Reply Last reply
          1
          • girishG girish

            @jadudm Missed this one somehow, did you get this sorted out?

            jadudmJ Offline
            jadudmJ Offline
            jadudm
            wrote on last edited by
            #7

            @girish, Even with a clean re-install of the docker registry, I can't convince my Cloudron instance to authenticate against it. I get 500 errors from it consistently (but can hit the v2 URL from a browser just fine?).

            I'm happy to do whatever I can to help with debugging. Perhaps this is one of those "it works everywhere else, but not on my machine" kind of things. 🤷

            I use Cloudron on a Dell 7040 I bought on eBay.

            jadudmJ 1 Reply Last reply
            0
            • jadudmJ jadudm

              @girish, Even with a clean re-install of the docker registry, I can't convince my Cloudron instance to authenticate against it. I get 500 errors from it consistently (but can hit the v2 URL from a browser just fine?).

              I'm happy to do whatever I can to help with debugging. Perhaps this is one of those "it works everywhere else, but not on my machine" kind of things. 🤷

              jadudmJ Offline
              jadudmJ Offline
              jadudm
              wrote on last edited by
              #8

              @jadudm Looking at the timeline on this thread, I'm wondering if I have the same problem.

              Updating to 7.1.something and reinstalling the image registry to see if that fixes things. Will update this thread when I discover the outcome...

              I use Cloudron on a Dell 7040 I bought on eBay.

              jadudmJ 1 Reply Last reply
              1
              • jadudmJ jadudm

                @jadudm Looking at the timeline on this thread, I'm wondering if I have the same problem.

                Updating to 7.1.something and reinstalling the image registry to see if that fixes things. Will update this thread when I discover the outcome...

                jadudmJ Offline
                jadudmJ Offline
                jadudm
                wrote on last edited by
                #9

                @jadudm This might be an aggressive firewall issue.

                I'll update when I know for sure. However, I suspect I'm not letting :5000 through, which would allow web clients to browse the registry, but not allow me to autheticate/push from the command line.

                I use Cloudron on a Dell 7040 I bought on eBay.

                jadudmJ 1 Reply Last reply
                0
                • jadudmJ jadudm

                  @jadudm This might be an aggressive firewall issue.

                  I'll update when I know for sure. However, I suspect I'm not letting :5000 through, which would allow web clients to browse the registry, but not allow me to autheticate/push from the command line.

                  jadudmJ Offline
                  jadudmJ Offline
                  jadudm
                  wrote on last edited by
                  #10

                  @jadudm Sorry, @girish . This was "user error." I do not know how this local registry ever worked, because I would swear the ports were not open to allow it to work.

                  However, this is a case where 5000 was not open to me. So, I could hit the /v2/ endpoint via HTTPS, and could locally store my login credentials via docker login, and I could browse the registry... but push and pull would fail.

                  Sill... I don't know how this worked in the first place, given that I don't think I had firewall rules open for it to succeed.

                  🤷

                  If this was a ticket, I'd mark it closed.

                  I use Cloudron on a Dell 7040 I bought on eBay.

                  jadudmJ 1 Reply Last reply
                  0
                  • jadudmJ jadudm

                    @jadudm Sorry, @girish . This was "user error." I do not know how this local registry ever worked, because I would swear the ports were not open to allow it to work.

                    However, this is a case where 5000 was not open to me. So, I could hit the /v2/ endpoint via HTTPS, and could locally store my login credentials via docker login, and I could browse the registry... but push and pull would fail.

                    Sill... I don't know how this worked in the first place, given that I don't think I had firewall rules open for it to succeed.

                    🤷

                    If this was a ticket, I'd mark it closed.

                    jadudmJ Offline
                    jadudmJ Offline
                    jadudm
                    wrote on last edited by
                    #11

                    @jadudm Nope.

                    If the machine I'm working on is dev, and the registry is hosted on my cloudron as registry.cloudron.example

                    I can now:

                    1. Browse to my private registry.
                    2. Authenticate and push to it from the command line from dev.
                    3. I cannot, via the Cloudron admin panel, set up registry.cloudron.example as my private registry.

                    I consistently get 500 errors for the registry.cloudron.example/v2/ endpoint. However, if I visit it in a browser on dev, it is easily reachable.

                    I am up to date on updates, I have restarted my cloudron recently, I have restarted the registry... I've even torn down and rebuilt the registry. I'm unclear why I can't set up my private registry.

                    Is there an invisible config file somewhere deep in the heart of Cloudron that I could go inspect the old fashioned way, and make sure some old data isn't being held on to that's... bad?

                    I use Cloudron on a Dell 7040 I bought on eBay.

                    jadudmJ 1 Reply Last reply
                    0
                    • jadudmJ jadudm

                      @jadudm Nope.

                      If the machine I'm working on is dev, and the registry is hosted on my cloudron as registry.cloudron.example

                      I can now:

                      1. Browse to my private registry.
                      2. Authenticate and push to it from the command line from dev.
                      3. I cannot, via the Cloudron admin panel, set up registry.cloudron.example as my private registry.

                      I consistently get 500 errors for the registry.cloudron.example/v2/ endpoint. However, if I visit it in a browser on dev, it is easily reachable.

                      I am up to date on updates, I have restarted my cloudron recently, I have restarted the registry... I've even torn down and rebuilt the registry. I'm unclear why I can't set up my private registry.

                      Is there an invisible config file somewhere deep in the heart of Cloudron that I could go inspect the old fashioned way, and make sure some old data isn't being held on to that's... bad?

                      jadudmJ Offline
                      jadudmJ Offline
                      jadudm
                      wrote on last edited by
                      #12

                      @jadudm

                      And, for myself, for future reference:

                      Invalid serverAddress: (HTTP code 500) server error - Get "https://reg.cloudron.example/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
                      

                      I tried manually changing the auth backend to "htpasswd" per documentation on the Docker site, and created a valid htpass file... but that doesn't change anything.

                      I have no idea why I cannot configure my cloudron instance to use a private registry.

                      I use Cloudron on a Dell 7040 I bought on eBay.

                      robiR 1 Reply Last reply
                      0
                      • jadudmJ jadudm

                        @jadudm

                        And, for myself, for future reference:

                        Invalid serverAddress: (HTTP code 500) server error - Get "https://reg.cloudron.example/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
                        

                        I tried manually changing the auth backend to "htpasswd" per documentation on the Docker site, and created a valid htpass file... but that doesn't change anything.

                        I have no idea why I cannot configure my cloudron instance to use a private registry.

                        robiR Offline
                        robiR Offline
                        robi
                        wrote on last edited by
                        #13

                        @jadudm The issue seems to be from your cloudron to the private registry.

                        So where is each? DNS? ping test? etc

                        Conscious tech

                        jadudmJ 1 Reply Last reply
                        0
                        • robiR robi

                          @jadudm The issue seems to be from your cloudron to the private registry.

                          So where is each? DNS? ping test? etc

                          jadudmJ Offline
                          jadudmJ Offline
                          jadudm
                          wrote on last edited by jadudm
                          #14

                          @robi Many thanks. I see where you're coming from. What confuses me... is that everything is on the same machine.

                          Which is why I'm (O_o), and think something really simple is going on.

                          1. I can hit the registry from the web (meaning port 443 gets through the firewall from the outside world), and authenticate as a Cloudron-based user.
                          2. I can hit the registry from the command line, and am authenticated as the same Cloudron user. This means port 5000 is getting through.
                          3. I cannot convince the Cloudron admin interface to let me authenticate as that user. The Cloudron admin interface (which is on the same machine as the registry) will not authenticate; it gets the 500 error.

                          I'm thoroughly confused, and am uncertain where to find the debug information that I need to solve this particular problem. Hence why I'm wondering if there's some kind of config that might have... "hung around" from previous attempts?

                          I use Cloudron on a Dell 7040 I bought on eBay.

                          jadudmJ 1 Reply Last reply
                          0
                          • jadudmJ jadudm

                            @robi Many thanks. I see where you're coming from. What confuses me... is that everything is on the same machine.

                            Which is why I'm (O_o), and think something really simple is going on.

                            1. I can hit the registry from the web (meaning port 443 gets through the firewall from the outside world), and authenticate as a Cloudron-based user.
                            2. I can hit the registry from the command line, and am authenticated as the same Cloudron user. This means port 5000 is getting through.
                            3. I cannot convince the Cloudron admin interface to let me authenticate as that user. The Cloudron admin interface (which is on the same machine as the registry) will not authenticate; it gets the 500 error.

                            I'm thoroughly confused, and am uncertain where to find the debug information that I need to solve this particular problem. Hence why I'm wondering if there's some kind of config that might have... "hung around" from previous attempts?

                            jadudmJ Offline
                            jadudmJ Offline
                            jadudm
                            wrote on last edited by
                            #15

                            A new not-what-I-was-looking-for-but-it-works-solution:

                            1. I'm mirroring the Dockerfile/project from my local Gitea to Gitlab.
                            2. I've set up Gitlab as my registry for the project.

                            (The project in this case is a Grocy package for Cloudron.)

                            This arguably gives me a private-enough repo for what I'm exploring, but I was hoping to be able to entirely leverage my Cloudron for this.

                            I'm just plain confused why, at some point in the past, my locally-hosted registry worked, and now, it does not. This is why I believe there is "crufty" config laying around, somewhere, and things therefore don't work.

                            (But... this makes almost no sense, because I have uninstalled and reinstalled the registry, so it is not clear how any state could persist.)

                            Anyway. This remains a mystery, and I would need some explicit guidance as to where I should dig around in the backend of Cloudron to figure out why things are not behaving. (It could also be that there's still some port that I don't have open, or... 🤷)

                            I use Cloudron on a Dell 7040 I bought on eBay.

                            1 Reply Last reply
                            0
                            • girishG girish has marked this topic as solved on
                            Reply
                            • Reply as topic
                            Log in to reply
                            • Oldest to Newest
                            • Newest to Oldest
                            • Most Votes


                              • Login

                              • Don't have an account? Register

                              • Login or register to search.
                              • First post
                                Last post
                              0
                              • Categories
                              • Recent
                              • Tags
                              • Popular
                              • Bookmarks
                              • Search