Docker registry

[girish]

So strange, I am getting a "invalid checksum digest format" whenever I push now to this registry. Has anyone seen such an error before?

The push refers to repository [xxx.xxx.xxx/cloudron/base]
fcdfeda3e242: Layer already exists 
0ea3bde29271: Layer already exists 
d75ccb14b8b6: Layer already exists 
74b4389a43ab: Layer already exists 
5f38ae1e1a63: Layer already exists 
3479c151673d: Layer already exists 
7a307b866f25: Layer already exists 
ce3a66c20e17: Layer already exists 
7197b970ebb9: Layer already exists 
16542a8fc3be: Layer already exists 
6597da2e2e52: Layer already exists 
977183d4e999: Layer already exists 
c8be1b8f4d60: Layer already exists 
invalid checksum digest format

[mario]

@girish local filesystem?

[girish]

@mario Yes, with the local storage. I wonder if it's something to do with the proxy auth. I am trying it without auth now.

edit: indeed, something to do with the proxy auth. It works fine without proxy auth. Debugging.

[mehdi]

@girish Are you on 6.1 ? Maybe your 2FA implementation broke something with the basic auth ?

[girish]

@mehdi yeah, i had that in mind and tried with 6.0 as well. fails the same. I am pretty sure this worked when I tested it back then, so I must have broke something !

[mehdi]

@girish You can try with an app-password, or try another Basic Auth ProxyAuth app, like Transmission (with an android app or a browser extension)

[girish]

What I am seeing is that docker doesn't send any authorization header at all. The issue is very similar to https://stackoverflow.com/questions/55516317/docker-login-not-passing-basic-authentication-headers-to-nginx . I can curl just fine.

[girish]

It seems that v2 registry auth does not use the basic bearer based authentication at all. https://docs.docker.com/registry/recipes/nginx/ is possibly obsolete, but I am trying to setup a registry from scratch now to double check.

[mario]

@girish it definitely can, that's how GitLab etc integration works.

[mehdi]

@girish Their doc indeed appears to be outdated. Different pages seem to indicate different things ...

[girish]

@mario thanks! i needed such a confident statement to help me keep looking further

I managed to get it to work. The issue is that proxyAuth on an auth fail redirects to the login page. But the docker registry wants it to return a 401 with a www-authenticate header. The header also causes issues with browsers since it starts popping up the login dialog.

In essence, even though the basic auth works, proxyAuth is not compatible. I thought about adding an flag to the manifest to have a different behavior but then again I don't like the current approach where we just install this registry and land on an empty page (any page even some static html with instructions would be better).

I ended up packaging it together the docker registry UI and a small LDAP server (from https://git.cloudron.io/cloudron/cloudron-serve). I haven't pushed the changes since they are not working entirely. But it's what I am working on in parallel with getting 6.1 out.

[fbartels]

@girish said in Docker registry:

I ended up packaging it together the docker registry UI and a small LDAP server

That sounds intriguing. What role does the ldap server serve? Just for auth against the registry ui?

[mehdi]

@girish I think the best approach would be to do a bit of user-agent parsing magic... Yeah, it would be quite specific for this use-case, but

[girish]

@fbartels said in Docker registry:

That sounds intriguing. What role does the ldap server serve? Just for auth against the registry ui?

Yes, pretty much. It's just a proxy that redirects to login page and auths against LDAP. The code itself is very small, just ~100 lines or so.

[girish]

@mehdi Right, I considered UA string hack but I think dropping users in a blank page is a bit rough. So, my first step was to do the UA testing with nginx in the app itself. But, that brought the dreaded browser auth modal dialog which I really dislike. It's the main reason I ended up making proxyAuth in the first place So.. I ended up making a node server.

[mehdi]

@girish No, I mean, after testing you could keep the proxyAuth, but do a test on the proxyAuth that could show the page for browsers, and send the expected 401 for docker client. Then we could have the best of both worlds : integration with platform LDAP, a simple registry UI, and working CLI.

[girish]

@mehdi Ah, understood you better now. I am actually ok to add this hack in proxy auth code. We will still need some nginx/apache in the app code though to serve the registry UI (which is just static html).

Suddenly, I am tempted to abandon my node server because I am struggling to make this proxy middleware work. It seems to have some bug with PATCH requests which docker registry uses.

[girish]

I have published this app as unstable now. It also has an integrated UI. I have only very mildly tested it, so do not use it in production. I have created an app category for this, please report any issues there.

[LoudLemur]

@robi Thanks,

Quay is a Free alternative to DockerHub. Hopefully, Cloudron makes good use of it... ?

[robi]

@LoudLemur You can already use it by pointing your Cloudron to it.