Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    SOLVED Cannot install apps from docker-registry because authentication fails

    Docker Registry
    docker registry authentication
    4
    9
    155
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      msbt App Dev last edited by

      Does anyone have the docker-registry working with authentication? I've tried and it works fine without auth (like my old setup with my custom registry solution), but as soon as I enable user management, the images can't be pushed to the target-cloudron.

      Recap of what I was doing:

      • installed docker and Cloudron cli on a new linux machine
      • installed the docker-registry app on Cloudron X (docker.example.com), added a user ("docker") on X and set its credentials in Cloudron Y settings
      • git cloned an app of mine on the linux machine
      • cloudron login (on Cloudron Y) and docker login docker.example.com
      • docker build -t docker.example.com/my-app .
      • docker push docker.example.com/my-app
      • cloudron install --image docker.example.com/my-app -l myapp

      Up until the last step everything works fine, but the containers can't get pushed/downloaded on Cloudron Y, this is what happens:

      CLI response:
      App installation error: Installation failed: Unable to pull image docker.example/my-app. Please check the network or if the image needs authentication. statusCode: 500

      App log:

      Feb 06 22:35:08 box:docker downloadImage docker.example.com/my-app
Feb 06 22:35:08 box:docker Downloading image docker.example.com/my-app. attempt: 1
Feb 06 22:35:08 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:15 box:docker Downloading image docker.example.com/my-app. attempt: 2
Feb 06 22:35:15 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:21 box:docker Downloading image docker.example.com/my-app. attempt: 3
Feb 06 22:35:21 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:26 box:docker Downloading image docker.example.com/my-app. attempt: 4
Feb 06 22:35:26 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:31 box:docker Downloading image docker.example.com/my-app. attempt: 5
Feb 06 22:35:31 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:37 box:docker Downloading image docker.example.com/my-app. attempt: 6
Feb 06 22:35:37 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:42 box:docker Downloading image docker.example.com/my-app. attempt: 7
Feb 06 22:35:42 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:47 box:docker Downloading image docker.example.com/my-app. attempt: 8
Feb 06 22:35:47 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:53 box:docker Downloading image docker.example.com/my-app. attempt: 9
Feb 06 22:35:53 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:58 box:docker Downloading image docker.example.com/my-app. attempt: 10
Feb 06 22:35:58 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:58 box:apptask myapp.cloudrony.com error installing app: BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500
Feb 06 22:35:58 box:apptask myapp.cloudrony.com updating app with values: {"installationState":"error","error":{"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500","reason":"Docker Error","taskId":"6145","installationState":"pending_install"}}
Feb 06 22:35:58 box:taskworker Task took 55.403 seconds
Feb 06 22:35:58 box:tasks setCompleted - 6145: {"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}
Feb 06 22:35:58 box:tasks 6145: {"percent":100,"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}

      Docker logs:

      Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977369199Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977424454Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:13 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:13.063771584Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277251730Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277311467Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573447894Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573505543Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857404427Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857459284Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156592002Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156667957Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455109662Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455163318Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:41 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:41.812898340Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.576830368Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748936700Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748989045Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041591964Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041658716Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335916491Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335978423Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616154482Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616210521Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"

      Does anyone have this working as a standalone registry (without gitlab)? No idea what to make of the docker logs with the character thing.

      Cheers, M

      BrutalBirdie girish 3 Replies Last reply Reply Quote 3
      • BrutalBirdie
        BrutalBirdie App Dev @msbt last edited by

        I got also issues with this.
        Some auth problem.

        Like my work? Consider donating a beer 🍻 Cheers!

        1 Reply Last reply Reply Quote 0
        • girish
          girish Staff @msbt last edited by

          @msbt Thanks for the details report. Indeed, I can confirm that Cloudron Y is unable to pull down a private iamge. I get the same error message.

          @BrutalBirdie Is your issue with pushing an image or when downloading the image ? (atleast from the messages on chat the error seemed something else)

          BrutalBirdie 1 Reply Last reply Reply Quote 0
          • girish
            girish Staff @msbt last edited by

            @msbt OK, so if do docker login the-x-registry and docker pull private-image it works. So, this looks like some Cloudron bug.

            1 Reply Last reply Reply Quote 0
            • girish
              girish Staff last edited by

              Looks to be something with the node module (dockerode) we use. It seems to work fine with docker.io private registry just not this custom registry app we have. Have to debug tomorrow.

              1 Reply Last reply Reply Quote 1
              • girish
                girish Staff last edited by girish

                @msbt Found the problem after much debugging. It seems there is some special code that changes the UA string depending on the client. So, the proxyAuth addon does not allow dockerode to authenticate properly.

                https://git.cloudron.io/cloudron/box/-/commit/1d0ad3cb47f85b05eabb31853c8c3a585d06c2e9 is the fix. It's really just changing docker-client to docker. If you apply the patch, restart the box code and also go to Docker registry app -> Location -> Save to regenerate the nginx config.

                M 1 Reply Last reply Reply Quote 4
                • M
                  msbt App Dev @girish last edited by

                  @girish nice, great find! I'll check it out tomorrow

                  1 Reply Last reply Reply Quote 0
                  • M
                    msbt App Dev last edited by

                    looking good @girish, just made those changes and did a cloudron install, worked like a charm!

                    1 Reply Last reply Reply Quote 0
                    • atrilahiji
                      atrilahiji App Dev last edited by atrilahiji

                      I'm having this exact same issue with a newly installed docker registry. I checked that particular box file but it has definitely made it into the current release. Is there anything I'm missing?

                      Nevermind... it was my bad.

                      Owner of Lahiji Apps Consulting
                      https://atrilahiji.dev/
                      https://lahijiapps.dev/

                      1 Reply Last reply Reply Quote 1
                      • First post
                        Last post