SOLVED Cannot install apps from docker-registry because authentication fails

msbt

Does anyone have the docker-registry working with authentication? I've tried and it works fine without auth (like my old setup with my custom registry solution), but as soon as I enable user management, the images can't be pushed to the target-cloudron.

Recap of what I was doing:

installed docker and Cloudron cli on a new linux machine
installed the docker-registry app on Cloudron X (docker.example.com), added a user ("docker") on X and set its credentials in Cloudron Y settings
git cloned an app of mine on the linux machine
cloudron login (on Cloudron Y) and docker login docker.example.com
docker build -t docker.example.com/my-app .
docker push docker.example.com/my-app
cloudron install --image docker.example.com/my-app -l myapp

Up until the last step everything works fine, but the containers can't get pushed/downloaded on Cloudron Y, this is what happens:

CLI response:
App installation error: Installation failed: Unable to pull image docker.example/my-app. Please check the network or if the image needs authentication. statusCode: 500

App log:

Feb 06 22:35:08 box:docker downloadImage docker.example.com/my-app
Feb 06 22:35:08 box:docker Downloading image docker.example.com/my-app. attempt: 1
Feb 06 22:35:08 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:15 box:docker Downloading image docker.example.com/my-app. attempt: 2
Feb 06 22:35:15 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:21 box:docker Downloading image docker.example.com/my-app. attempt: 3
Feb 06 22:35:21 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:26 box:docker Downloading image docker.example.com/my-app. attempt: 4
Feb 06 22:35:26 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:31 box:docker Downloading image docker.example.com/my-app. attempt: 5
Feb 06 22:35:31 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:37 box:docker Downloading image docker.example.com/my-app. attempt: 6
Feb 06 22:35:37 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:42 box:docker Downloading image docker.example.com/my-app. attempt: 7
Feb 06 22:35:42 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:47 box:docker Downloading image docker.example.com/my-app. attempt: 8
Feb 06 22:35:47 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:53 box:docker Downloading image docker.example.com/my-app. attempt: 9
Feb 06 22:35:53 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:58 box:docker Downloading image docker.example.com/my-app. attempt: 10
Feb 06 22:35:58 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:58 box:apptask myapp.cloudrony.com error installing app: BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500
Feb 06 22:35:58 box:apptask myapp.cloudrony.com updating app with values: {"installationState":"error","error":{"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500","reason":"Docker Error","taskId":"6145","installationState":"pending_install"}}
Feb 06 22:35:58 box:taskworker Task took 55.403 seconds
Feb 06 22:35:58 box:tasks setCompleted - 6145: {"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}
Feb 06 22:35:58 box:tasks 6145: {"percent":100,"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}

Docker logs:

Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977369199Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977424454Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:13 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:13.063771584Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277251730Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277311467Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573447894Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573505543Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857404427Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857459284Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156592002Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156667957Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455109662Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455163318Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:41 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:41.812898340Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.576830368Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748936700Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748989045Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041591964Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041658716Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335916491Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335978423Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616154482Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616210521Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"

Does anyone have this working as a standalone registry (without gitlab)? No idea what to make of the docker logs with the character thing.

Cheers, M

BrutalBirdie

I got also issues with this.
Some auth problem.

girish

@msbt Thanks for the details report. Indeed, I can confirm that Cloudron Y is unable to pull down a private iamge. I get the same error message.

@BrutalBirdie Is your issue with pushing an image or when downloading the image ? (atleast from the messages on chat the error seemed something else)

girish

@msbt OK, so if do docker login the-x-registry and docker pull private-image it works. So, this looks like some Cloudron bug.

girish

Looks to be something with the node module (dockerode) we use. It seems to work fine with docker.io private registry just not this custom registry app we have. Have to debug tomorrow.

girish

@msbt Found the problem after much debugging. It seems there is some special code that changes the UA string depending on the client. So, the proxyAuth addon does not allow dockerode to authenticate properly.

https://git.cloudron.io/cloudron/box/-/commit/1d0ad3cb47f85b05eabb31853c8c3a585d06c2e9 is the fix. It's really just changing docker-client to docker. If you apply the patch, restart the box code and also go to Docker registry app -> Location -> Save to regenerate the nginx config.

msbt

@girish nice, great find! I'll check it out tomorrow

msbt

looking good @girish, just made those changes and did a cloudron install, worked like a charm!

atridad

I'm having this exact same issue with a newly installed docker registry. I checked that particular box file but it has definitely made it into the current release. Is there anything I'm missing?

Nevermind... it was my bad.

doodlemania2

@atrilahiji What was your resolution? I'm getting it as well.
Registry is local to my cloudron I'm installing from.
Getting 500 from cloudron update and logs showing (on the app side) grabbing with auth failing. Some switch I need to throw?

jk

@girish Sorry for reopening this after quite a while.

If have the exact same problem when I am not using the docker client, but the RedHat-built containers projects (podman / skopeo / buildah / etc).

The solution is the exact same patch, but then with container instead of docker. I tried to create a pull request for it, but apparently I have no right to fork the box code and make one.

Either way, on my server, the code looks like this now:

in nginxcoonfig.ejs:

    location @proxy-auth-login {
        if ($http_user_agent ~* "docker") {
            return 401;
        }
        if ($http_user_agent ~* "container") {
            return 401;
        }
        return 302 /login?redirect=$request_uri;
    }

in proxyauth.js

function isBrowser(req) {
    const userAgent = req.get('user-agent');
    if (!userAgent) return false;

    // https://github.com/docker/engine/blob/master/dockerversion/useragent.go#L18
    return !userAgent.toLowerCase().includes('docker') && !userAgent.toLowerCase().includes('container');
}

Would it be possible to patch this too with the next box release?

girish

@jk thanks! I have applied the patch.

girish

@jk This is you right https://git.cloudron.io/admin/users/jacobkiers ? I have fixed up your permissions on gitlab.

jk

@girish Yes, that's me. Thanks!

jk

@girish Is there any update on when a new box will be released?

I've been waiting to upgrade because I don't want to lose these changes.

Sadly, that also means that apps are not automatically upgraded any more, which is somewhat annoying.

girish

@jk the next release 6.4 will contain the change. The patch is already in - https://git.cloudron.io/cloudron/box/-/commit/85e3e4b955 . We are still working on 6.4 features - https://forum.cloudron.io/topic/5319/what-s-coming-in-6-4 . You can track the progress there. No intermediate release between now and 6.4 is planned.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

SOLVED Cannot install apps from docker-registry because authentication fails