Cannot install apps from docker-registry because authentication fails

[girish]

Looks to be something with the node module (dockerode) we use. It seems to work fine with docker.io private registry just not this custom registry app we have. Have to debug tomorrow.

[girish]

@msbt Found the problem after much debugging. It seems there is some special code that changes the UA string depending on the client. So, the proxyAuth addon does not allow dockerode to authenticate properly.

https://git.cloudron.io/cloudron/box/-/commit/1d0ad3cb47f85b05eabb31853c8c3a585d06c2e9 is the fix. It's really just changing docker-client to docker. If you apply the patch, restart the box code and also go to Docker registry app -> Location -> Save to regenerate the nginx config.

[msbt]

@girish nice, great find! I'll check it out tomorrow

[msbt]

looking good @girish, just made those changes and did a cloudron install, worked like a charm!

[A Former User]

I'm having this exact same issue with a newly installed docker registry. I checked that particular box file but it has definitely made it into the current release. Is there anything I'm missing?

Nevermind... it was my bad.

[doodlemania2]

@atrilahiji What was your resolution? I'm getting it as well.
Registry is local to my cloudron I'm installing from.
Getting 500 from cloudron update and logs showing (on the app side) grabbing with auth failing. Some switch I need to throw?

[jk]

@girish Sorry for reopening this after quite a while.

If have the exact same problem when I am not using the docker client, but the RedHat-built containers projects (podman / skopeo / buildah / etc).

The solution is the exact same patch, but then with container instead of docker. I tried to create a pull request for it, but apparently I have no right to fork the box code and make one.

Either way, on my server, the code looks like this now:

in nginxcoonfig.ejs:

    location @proxy-auth-login {
        if ($http_user_agent ~* "docker") {
            return 401;
        }
        if ($http_user_agent ~* "container") {
            return 401;
        }
        return 302 /login?redirect=$request_uri;
    }

in proxyauth.js

function isBrowser(req) {
    const userAgent = req.get('user-agent');
    if (!userAgent) return false;

    // https://github.com/docker/engine/blob/master/dockerversion/useragent.go#L18
    return !userAgent.toLowerCase().includes('docker') && !userAgent.toLowerCase().includes('container');
}

Would it be possible to patch this too with the next box release?

[girish]

@jk thanks! I have applied the patch.

[girish]

@jk This is you right https://git.cloudron.io/admin/users/jacobkiers ? I have fixed up your permissions on gitlab.

[jk]

@girish Yes, that's me. Thanks!

[jk]

@girish Is there any update on when a new box will be released?

I've been waiting to upgrade because I don't want to lose these changes.

Sadly, that also means that apps are not automatically upgraded any more, which is somewhat annoying.

[girish]

@jk the next release 6.4 will contain the change. The patch is already in - https://git.cloudron.io/cloudron/box/-/commit/85e3e4b955 . We are still working on 6.4 features - https://forum.cloudron.io/topic/5319/what-s-coming-in-6-4 . You can track the progress there. No intermediate release between now and 6.4 is planned.

[ochoseis]

Edit 2: I just rebooted and installed a new instance of the Docker Registry app, which seemed to pick up the fixes from the template at /home/yellowtent/box/src/nginxconfig.ejs (it had been a while since I last rebooted), but I don't see the fix on the existing installations' nginx configs under /etc/nginx/applications. Just wanted to see if that's working as intended? I.e. to catch a bugfix in the nginx config it seems like you need to install a new instance of an app after rebooting.

Edit: I just reread the thread above and it seems like the below issue should've been fixed in Cloudron v6.4/v7.0. I'm on v7.0.4 w/ Docker Registry 2.7.1 (Registry UI 2.1.0) and com.docker.registry@1.4.0. After logging into the box I can see the updates in /home/yellowtent/box/src/proxyauth.js but not in the apps' nginx configs under /etc/nginx/applications. Do I need to do anything to pick up the fixes for the nginx configs?

Original post:
I arrived at this thread because I was having trouble using podman to push and pull an images to the Docker Registry app with Cloudron user management turned on. podman shows the following error when I attempt to pull the image, and the < hinted at a redirect to an HTML page:

$ podman pull images.mycloudrondomain/debian

Error: initializing image from source docker://images.mycloudrondomain/debian:latest: invalid character '<' looking for beginning of value

I am able to push and pull images on a different instance of the app where I disabled auth.

From the nginx access logs on my server and the source, it looks like podman sets a user agent of containers/{version} (github.com/containers/image). Would it be possible to include containers in the user-agent checks above?

[girish]

@ochoseis thanks for debugging! If I understand you correctly, the nginx config does not have the block below for you ?

    location @proxy-auth-login {
        if ($http_user_agent ~* "docker") {
            return 401;
        }
        if ($http_user_agent ~* "container") {
            return 401;
        }
        return 302 /login?redirect=$request_uri;
    }

For this, go to Location section of the app and click save without making any changes. This re-generates the nginx config. In 7.1, we re-generate all the nginx configs, so this would get fixed if you update to 7.1 as well.

(This is a bug in Cloudron because we don't version these nginx configs. When we change the nginx template, currently, we have not mechanism to regenerate the nginx configs of apps. We have an issue to fix this though).

[ochoseis]

@girish Thanks -- you summed up the bug I noticed, and I'll be sure to try the Location-save workaround if I run into any nginx issues in the future.

[nottheend]

Few years after the original bug got fixed, I am encountering the same error for my selfhosted private registry:

Docker Error - Unable to pull image mydomain.com/myusername/myapp:1.0.0 Please check the network or if the image needs authentication. statusCode 500

In the logs are 10 not successful pull attempts visible.

The Docker registry is hosted on the same Cloudron server, it is Docker Registry App from official App Store.

In the settings if my Cloudron instance, I configured a Private Docker Registry with the same Credentials I used to successfully push the Docker build from local. I changed from Docker to Other just to be sure, with the same credentials.
After building the build, I can successfullypush it with the very same credentials to the private Docker Hub.

Is there any way how to come closer to the probable authentication error?

[nottheend]

@nottheend have created a new topic since it is most likely not this bug:
https://forum.cloudron.io/topic/12216/docker-error-unable-to-pull-image-on-same-instance

[jk]

Hi, I have encountered this bug once more, with the following user agents:

• Podman: libpod/<version>
• Skopeo: skopeo/<version>

@girish Would it be possible to add those as well? That would be much appreciated.

In src/proxyAuth.js

// someday this can be more sophisticated and check for a real browser
function isBrowser(req) {
    const userAgent = req.get('user-agent');
    if (!userAgent) return false;

    // https://github.com/docker/engine/blob/master/dockerversion/useragent.go#L18
    return !userAgent.toLowerCase().includes('docker') && !userAgent.toLowerCase().includes('container') && !userAgent.toLowerCase().includes('libpod') && !userAgent.toLowerCase().includes('skopeo');
}

In src/nginxconfig.ejs

    location @proxy-auth-login {
        if ($http_user_agent ~* "docker") {
            return 401;
        }
        if ($http_user_agent ~* "container") {
            return 401;
        }
        if ($http_user_agent ~* "libpod") {
            return 401;
        }
        if ($http_user_agent ~* "skopeo") {
            return 401;
        }

        return 302 /login?redirect=$request_uri;
    }

[girish]

@jk fixed in https://git.cloudron.io/platform/box/-/commit/2b30f5591cddabe6b2a0db1fc23bf151c86274b7 . thanks!

[jk]

Thanks a lot!