Docker registry

robi

@mario how's it going?

mario

@robi I have managed to integrate the registry with GitLab.

@girish where are we at with making proper MR for AuthProxy + making SSO optional? Then I can document GitLab integration, you can write some tests and off we go!

girish

@mario Fantastic news. So, all we need is docs to make it work with GitLab registry or does it need any packages changes to gitlab app or docker registry app ?

mario

@girish doesn't seem like it'll need package changes, documentation will be enough.

mario

I take that back, I did add some package changes. Had no time to test, but things seem to be working ok from the initial glimpse at it:

https://git.cloudron.io/cloudron/docker-registry-app/-/merge_requests/1

Please test and report back @girish and others

robi

Might be useful to add auto deletion of old images:
https://github.com/jeffstephens/retention-manager

mario

@robi GitLab does that for me Maybe a separate app?

girish

@mario Just looking into this now.

Wondering, what is the best way forward. The app has no UI, but can have a login screen (via proxyAuth). So, when they login, they see a blank screen. Not ideal. Does it make sense to bundle any of the docker uis like https://github.com/Joxit/docker-registry-ui/ ? Seems quite easy to do. I can look into it.

robi

@mario that's great, but the standalone private registry app that's coming may need it and as @girish pointed out a simple UI.

Also not everyone is interested in the ruby laden GitLab and all it's complexity.

mario

@girish depends on what the community needs. I'm more than happy to have a separate registry + other things as separate apps for those who need it.

If I needed to pick the best registry solution with UI and everything else that's well maintained and suitable for Cloudron, I'd probably look at Quay which supports LDAP auth.

https://github.com/quay/quay

mehdi

I am 100% in favor of bundling a simple UI together with the registry. Even if one does not need it and wants to use the gitlab UI, there's basically nothing to lose besides a few kB of storage ^^

jimcavoli

Yeah, Quay and Harbor are definitely the big players in this space. Very similar products - harbor is CNCF graduated and Quay is upstream for the corresponding Red Hat product. Either (or both) would be good UI adds.

girish

Last I checked harbor was impractical to package (as in way too much effort, it's really geared for the k8s crowd). Quay is a good option, but let me get this basic docker registry out first, I am almost there.

mario

@jimcavoli Quay afaik implements the protocol as well, so no need for registry separately.

girish

So strange, I am getting a "invalid checksum digest format" whenever I push now to this registry. Has anyone seen such an error before?

The push refers to repository [xxx.xxx.xxx/cloudron/base]
fcdfeda3e242: Layer already exists 
0ea3bde29271: Layer already exists 
d75ccb14b8b6: Layer already exists 
74b4389a43ab: Layer already exists 
5f38ae1e1a63: Layer already exists 
3479c151673d: Layer already exists 
7a307b866f25: Layer already exists 
ce3a66c20e17: Layer already exists 
7197b970ebb9: Layer already exists 
16542a8fc3be: Layer already exists 
6597da2e2e52: Layer already exists 
977183d4e999: Layer already exists 
c8be1b8f4d60: Layer already exists 
invalid checksum digest format

mario

@girish local filesystem?

girish

@mario Yes, with the local storage. I wonder if it's something to do with the proxy auth. I am trying it without auth now.

edit: indeed, something to do with the proxy auth. It works fine without proxy auth. Debugging.

mehdi

@girish Are you on 6.1 ? Maybe your 2FA implementation broke something with the basic auth ?

girish

@mehdi yeah, i had that in mind and tried with 6.0 as well. fails the same. I am pretty sure this worked when I tested it back then, so I must have broke something !

mehdi

@girish You can try with an app-password, or try another Basic Auth ProxyAuth app, like Transmission (with an android app or a browser extension)

girish

What I am seeing is that docker doesn't send any authorization header at all. The issue is very similar to https://stackoverflow.com/questions/55516317/docker-login-not-passing-basic-authentication-headers-to-nginx . I can curl just fine.

girish

It seems that v2 registry auth does not use the basic bearer based authentication at all. https://docs.docker.com/registry/recipes/nginx/ is possibly obsolete, but I am trying to setup a registry from scratch now to double check.

mario

@girish it definitely can, that's how GitLab etc integration works.

mehdi

@girish Their doc indeed appears to be outdated. Different pages seem to indicate different things ...

girish

@mario thanks! i needed such a confident statement to help me keep looking further

I managed to get it to work. The issue is that proxyAuth on an auth fail redirects to the login page. But the docker registry wants it to return a 401 with a www-authenticate header. The header also causes issues with browsers since it starts popping up the login dialog.

In essence, even though the basic auth works, proxyAuth is not compatible. I thought about adding an flag to the manifest to have a different behavior but then again I don't like the current approach where we just install this registry and land on an empty page (any page even some static html with instructions would be better).

I ended up packaging it together the docker registry UI and a small LDAP server (from https://git.cloudron.io/cloudron/cloudron-serve). I haven't pushed the changes since they are not working entirely. But it's what I am working on in parallel with getting 6.1 out.

fbartels

@girish said in Docker registry:

I ended up packaging it together the docker registry UI and a small LDAP server

That sounds intriguing. What role does the ldap server serve? Just for auth against the registry ui?

mehdi

@girish I think the best approach would be to do a bit of user-agent parsing magic... Yeah, it would be quite specific for this use-case, but

girish

@fbartels said in Docker registry:

That sounds intriguing. What role does the ldap server serve? Just for auth against the registry ui?

Yes, pretty much. It's just a proxy that redirects to login page and auths against LDAP. The code itself is very small, just ~100 lines or so.

girish

@mehdi Right, I considered UA string hack but I think dropping users in a blank page is a bit rough. So, my first step was to do the UA testing with nginx in the app itself. But, that brought the dreaded browser auth modal dialog which I really dislike. It's the main reason I ended up making proxyAuth in the first place So.. I ended up making a node server.

mehdi

@girish No, I mean, after testing you could keep the proxyAuth, but do a test on the proxyAuth that could show the page for browsers, and send the expected 401 for docker client. Then we could have the best of both worlds : integration with platform LDAP, a simple registry UI, and working CLI.

girish

@mehdi Ah, understood you better now. I am actually ok to add this hack in proxy auth code. We will still need some nginx/apache in the app code though to serve the registry UI (which is just static html).

Suddenly, I am tempted to abandon my node server because I am struggling to make this proxy middleware work. It seems to have some bug with PATCH requests which docker registry uses.

girish

I have published this app as unstable now. It also has an integrated UI. I have only very mildly tested it, so do not use it in production. I have created an app category for this, please report any issues there.

LoudLemur

@robi Thanks,

Quay is a Free alternative to DockerHub. Hopefully, Cloudron makes good use of it... ?

robi

@LoudLemur You can already use it by pointing your Cloudron to it.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Docker registry