Cannot install apps from docker-registry because authentication fails

msbt

Does anyone have the docker-registry working with authentication? I've tried and it works fine without auth (like my old setup with my custom registry solution), but as soon as I enable user management, the images can't be pushed to the target-cloudron.

Recap of what I was doing:

installed docker and Cloudron cli on a new linux machine
installed the docker-registry app on Cloudron X (docker.example.com), added a user ("docker") on X and set its credentials in Cloudron Y settings
git cloned an app of mine on the linux machine
cloudron login (on Cloudron Y) and docker login docker.example.com
docker build -t docker.example.com/my-app .
docker push docker.example.com/my-app
cloudron install --image docker.example.com/my-app -l myapp

Up until the last step everything works fine, but the containers can't get pushed/downloaded on Cloudron Y, this is what happens:

CLI response:
App installation error: Installation failed: Unable to pull image docker.example/my-app. Please check the network or if the image needs authentication. statusCode: 500

App log:

Feb 06 22:35:08 box:docker downloadImage docker.example.com/my-app
Feb 06 22:35:08 box:docker Downloading image docker.example.com/my-app. attempt: 1
Feb 06 22:35:08 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:15 box:docker Downloading image docker.example.com/my-app. attempt: 2
Feb 06 22:35:15 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:21 box:docker Downloading image docker.example.com/my-app. attempt: 3
Feb 06 22:35:21 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:26 box:docker Downloading image docker.example.com/my-app. attempt: 4
Feb 06 22:35:26 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:31 box:docker Downloading image docker.example.com/my-app. attempt: 5
Feb 06 22:35:31 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:37 box:docker Downloading image docker.example.com/my-app. attempt: 6
Feb 06 22:35:37 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:42 box:docker Downloading image docker.example.com/my-app. attempt: 7
Feb 06 22:35:42 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:47 box:docker Downloading image docker.example.com/my-app. attempt: 8
Feb 06 22:35:47 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:53 box:docker Downloading image docker.example.com/my-app. attempt: 9
Feb 06 22:35:53 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:58 box:docker Downloading image docker.example.com/my-app. attempt: 10
Feb 06 22:35:58 box:docker pullImage: will pull docker.example.com/my-app. auth: yes
Feb 06 22:35:58 box:apptask myapp.cloudrony.com error installing app: BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500
Feb 06 22:35:58 box:apptask myapp.cloudrony.com updating app with values: {"installationState":"error","error":{"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500","reason":"Docker Error","taskId":"6145","installationState":"pending_install"}}
Feb 06 22:35:58 box:taskworker Task took 55.403 seconds
Feb 06 22:35:58 box:tasks setCompleted - 6145: {"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}
Feb 06 22:35:58 box:tasks 6145: {"percent":100,"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}

Docker logs:

Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977369199Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977424454Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:13 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:13.063771584Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277251730Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277311467Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573447894Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573505543Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857404427Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857459284Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156592002Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156667957Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455109662Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455163318Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:41 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:41.812898340Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.576830368Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748936700Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748989045Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041591964Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041658716Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335916491Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335978423Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616154482Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value"
Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616210521Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"

Does anyone have this working as a standalone registry (without gitlab)? No idea what to make of the docker logs with the character thing.

Cheers, M

BrutalBirdie

I got also issues with this.
Some auth problem.

girish

@msbt Thanks for the details report. Indeed, I can confirm that Cloudron Y is unable to pull down a private iamge. I get the same error message.

@BrutalBirdie Is your issue with pushing an image or when downloading the image ? (atleast from the messages on chat the error seemed something else)

girish

@msbt OK, so if do docker login the-x-registry and docker pull private-image it works. So, this looks like some Cloudron bug.

girish

Looks to be something with the node module (dockerode) we use. It seems to work fine with docker.io private registry just not this custom registry app we have. Have to debug tomorrow.

girish

@msbt Found the problem after much debugging. It seems there is some special code that changes the UA string depending on the client. So, the proxyAuth addon does not allow dockerode to authenticate properly.

https://git.cloudron.io/cloudron/box/-/commit/1d0ad3cb47f85b05eabb31853c8c3a585d06c2e9 is the fix. It's really just changing docker-client to docker. If you apply the patch, restart the box code and also go to Docker registry app -> Location -> Save to regenerate the nginx config.

msbt

@girish nice, great find! I'll check it out tomorrow

msbt

looking good @girish, just made those changes and did a cloudron install, worked like a charm!

atridad

I'm having this exact same issue with a newly installed docker registry. I checked that particular box file but it has definitely made it into the current release. Is there anything I'm missing?

Nevermind... it was my bad.

doodlemania2

@atrilahiji What was your resolution? I'm getting it as well.
Registry is local to my cloudron I'm installing from.
Getting 500 from cloudron update and logs showing (on the app side) grabbing with auth failing. Some switch I need to throw?

jk

@girish Sorry for reopening this after quite a while.

If have the exact same problem when I am not using the docker client, but the RedHat-built containers projects (podman / skopeo / buildah / etc).

The solution is the exact same patch, but then with container instead of docker. I tried to create a pull request for it, but apparently I have no right to fork the box code and make one.

Either way, on my server, the code looks like this now:

in nginxcoonfig.ejs:

    location @proxy-auth-login {
        if ($http_user_agent ~* "docker") {
            return 401;
        }
        if ($http_user_agent ~* "container") {
            return 401;
        }
        return 302 /login?redirect=$request_uri;
    }

in proxyauth.js

function isBrowser(req) {
    const userAgent = req.get('user-agent');
    if (!userAgent) return false;

    // https://github.com/docker/engine/blob/master/dockerversion/useragent.go#L18
    return !userAgent.toLowerCase().includes('docker') && !userAgent.toLowerCase().includes('container');
}

Would it be possible to patch this too with the next box release?

girish

@jk thanks! I have applied the patch.

girish

@jk This is you right https://git.cloudron.io/admin/users/jacobkiers ? I have fixed up your permissions on gitlab.

jk

@girish Yes, that's me. Thanks!

jk

@girish Is there any update on when a new box will be released?

I've been waiting to upgrade because I don't want to lose these changes.

Sadly, that also means that apps are not automatically upgraded any more, which is somewhat annoying.

girish

@jk the next release 6.4 will contain the change. The patch is already in - https://git.cloudron.io/cloudron/box/-/commit/85e3e4b955 . We are still working on 6.4 features - https://forum.cloudron.io/topic/5319/what-s-coming-in-6-4 . You can track the progress there. No intermediate release between now and 6.4 is planned.

ochoseis

Edit 2: I just rebooted and installed a new instance of the Docker Registry app, which seemed to pick up the fixes from the template at /home/yellowtent/box/src/nginxconfig.ejs (it had been a while since I last rebooted), but I don't see the fix on the existing installations' nginx configs under /etc/nginx/applications. Just wanted to see if that's working as intended? I.e. to catch a bugfix in the nginx config it seems like you need to install a new instance of an app after rebooting.

Edit: I just reread the thread above and it seems like the below issue should've been fixed in Cloudron v6.4/v7.0. I'm on v7.0.4 w/ Docker Registry 2.7.1 (Registry UI 2.1.0) and com.docker.registry@1.4.0. After logging into the box I can see the updates in /home/yellowtent/box/src/proxyauth.js but not in the apps' nginx configs under /etc/nginx/applications. Do I need to do anything to pick up the fixes for the nginx configs?

Original post:
I arrived at this thread because I was having trouble using podman to push and pull an images to the Docker Registry app with Cloudron user management turned on. podman shows the following error when I attempt to pull the image, and the < hinted at a redirect to an HTML page:

$ podman pull images.mycloudrondomain/debian

Error: initializing image from source docker://images.mycloudrondomain/debian:latest: invalid character '<' looking for beginning of value

I am able to push and pull images on a different instance of the app where I disabled auth.

From the nginx access logs on my server and the source, it looks like podman sets a user agent of containers/{version} (github.com/containers/image). Would it be possible to include containers in the user-agent checks above?

girish

@ochoseis thanks for debugging! If I understand you correctly, the nginx config does not have the block below for you ?

    location @proxy-auth-login {
        if ($http_user_agent ~* "docker") {
            return 401;
        }
        if ($http_user_agent ~* "container") {
            return 401;
        }
        return 302 /login?redirect=$request_uri;
    }

For this, go to Location section of the app and click save without making any changes. This re-generates the nginx config. In 7.1, we re-generate all the nginx configs, so this would get fixed if you update to 7.1 as well.

(This is a bug in Cloudron because we don't version these nginx configs. When we change the nginx template, currently, we have not mechanism to regenerate the nginx configs of apps. We have an issue to fix this though).

ochoseis

@girish Thanks -- you summed up the bug I noticed, and I'll be sure to try the Location-save workaround if I run into any nginx issues in the future.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Cannot install apps from docker-registry because authentication fails