Docker registry

mario

https://github.com/docker/distribution-library-image

This would enable us to tie it in with Gitlab.

Lonkle

@mario To keep and maintain our own private Docker hubs?

robi

Adding Quay
https://github.com/quay/quay
Project Quay builds, stores, and distributes your container images.

High-level features include:

Docker Registry Protocol v2
Docker Manifest Schema v2.1, v2.2
AppC Image Discovery via on-demand transcoding
Image Squashing via on-demand transcoding
Authentication provided by LDAP, Keystone, OIDC, Google, and GitHub
ACLs, team management, and auditability logs
Geo-replicated storage provided by local filesystems, S3, GCS, Swift, and Ceph
Continuous Integration integrated with GitHub, Bitbucket, GitLab, and git
Security Vulnerability Analysis via Clair
Swagger-compliant HTTP API

mario

I got the registry working, need to get it integrated with GitLab now.

girish

@mario Did you get the registry working as a Cloudron package or outside Cloudron?

Lonkle

@mario What he said ^^^

robi

anyone check his gitlab repo?

Lonkle

@robi I don't know it.

mario

@girish as a Cloudron package. I'll push it to gitlab once I clean it up, but it's useless to me without the GitLab integration and that part seems tricky.

mario

@girish
https://git.cloudron.io/mario/docker-registry

mario

default user:

admin/admin

mehdi

@mario This looks good ! Combined with the authProxy of Cloudron6, we could disable the htpasswd auth of the app and since it uses basic auth it should work with the LDAP users !

We would just have to slap a basic web interface on that and it would be perfect

mario

@mehdi I'm updating it further now with redis support etc.

mario

Updated and working.

robi

Amazing progress, thank you @mario

Now to figure out how to install this and inch another step in packaging a custom app..

fbartels

@mario would you mind me making an attempt of adding https://github.com/Joxit/docker-registry-ui/ to your app? Or would you rather do it yourself, or choose a different ui?

Another solution could be the reg cli utility. A simple docker binary that can also expose a ui.

@robi after you installed to Cloudron cli (should be on a Linux machine with docker installed) you just need to run cloudron build && cloudron install and then follow the prompts.

mario

@fbartels honestly, I'd prefer UI as a separate app and would attempt Portus - possibly together with you. What are your thoughts?

An alternative would indeed be, if people prefer, to have Registry + UI together -> but in any case I'd strongly prefer Portus to anything else.

girish

@mario Oh wow, this is awesome. I had no idea one could run a registry this way. I thought one has to make some use of the docker addon! This way is so much simpler and nicer.

I forked the code to https://git.cloudron.io/cloudron/docker-registry-app/ and gave you permissions. It just worked (tm). Do you think you can put in a LICENSE file and keep developing there? It's a holiday for thanksgiving here, but I will look into this soonish.

fbartels

@mario TIL portus does not implement it's own registry, but simply uses the official one.

Yes, having them separate can have it's benefits. Would need to refresh my knowledge in regards to portus first before I know if I could be of much help.

mario

@girish happy Thanksgiving!

Thanks for the fork, will see what I can do in the coming days

girish

@mehdi said in Docker registry:

Combined with the authProxy of Cloudron6, we could disable the htpasswd auth of the app and since it uses basic auth it should work with the LDAP users !

I gave this a try and this worked great! https://git.cloudron.io/cloudron/docker-registry-app/-/commit/547e3b30b0d9038d9fe73416a7df7b3d32f265ec

mario

@fbartels said in Docker registry:

@mario TIL portus does not implement it's own registry, but simply uses the official one.

Yes, having them separate can have it's benefits. Would need to refresh my knowledge in regards to portus first before I know if I could be of much help.

Indeed It basically takes advantage of the official registry support for token-auth, giving you a nice UI, permissions, etc.

When you get a moment to check it out, let's talk!

mario

@girish said in Docker registry:

@mehdi said in Docker registry:

Combined with the authProxy of Cloudron6, we could disable the htpasswd auth of the app and since it uses basic auth it should work with the LDAP users !

I gave this a try and this worked great! https://git.cloudron.io/cloudron/docker-registry-app/-/commit/547e3b30b0d9038d9fe73416a7df7b3d32f265ec

The only problem here is that this would not work for me - we basically only have admins on Cloudron itself, and this would limit Registry access to them alone.

Edit: this is because we do auth via Azure AD/SAML for pretty much everything in the company.

girish

@mario said in Docker registry:

The only problem here is that this would not work for me

Good point. I forgot to add the optionalSso flag to manifest. With that flag, you can install the app without Cloudron Directory integration (like you do with other apps) and then we can have the default admin/admin setup that you have when LDAP is disabled. Would that work?

mario

@girish yes, though there's a bug in your commit So let's make sure we fix that too.

girish

@mario Ha ha, possibly. I only hacked it up quickly and checked if proxyAuth code in 6.0 will work before I make the release.

mario

@girish enjoy your holiday and let me know when you're back around next week so we can take this further

mehdi

@girish great ! I was 90% sure it would work, I'm glad I got it right

About interfaces, Portus indeed looks really great. However, I really don't see how it would work as a separate app. I really think it makes sense to bundle them together.

girish

Is Portus still developed? It seems it has seen no commits since Mar 25 2020?

I am both excited about this and confused about where it is at. My endgame is using GitLab to manage containers, but I need to point it at a registry. Would this ultimately work? And is there a way to have auth go through gitlab for this?

mario

@atrilahiji it's currently working as a stand-alone registry via basic auth powered by htpasswd file. It'll also support Cloudron SSO shortly, after that I'll work on making it work with GitLab.

robi

@mario how's it going?

mario

@robi I have managed to integrate the registry with GitLab.

@girish where are we at with making proper MR for AuthProxy + making SSO optional? Then I can document GitLab integration, you can write some tests and off we go!

girish

@mario Fantastic news. So, all we need is docs to make it work with GitLab registry or does it need any packages changes to gitlab app or docker registry app ?

mario

@girish doesn't seem like it'll need package changes, documentation will be enough.

mario

I take that back, I did add some package changes. Had no time to test, but things seem to be working ok from the initial glimpse at it:

https://git.cloudron.io/cloudron/docker-registry-app/-/merge_requests/1

Please test and report back @girish and others

robi

Might be useful to add auto deletion of old images:
https://github.com/jeffstephens/retention-manager

mario

@robi GitLab does that for me Maybe a separate app?

girish

@mario Just looking into this now.

Wondering, what is the best way forward. The app has no UI, but can have a login screen (via proxyAuth). So, when they login, they see a blank screen. Not ideal. Does it make sense to bundle any of the docker uis like https://github.com/Joxit/docker-registry-ui/ ? Seems quite easy to do. I can look into it.

robi

@mario that's great, but the standalone private registry app that's coming may need it and as @girish pointed out a simple UI.

Also not everyone is interested in the ruby laden GitLab and all it's complexity.

mario

@girish depends on what the community needs. I'm more than happy to have a separate registry + other things as separate apps for those who need it.

If I needed to pick the best registry solution with UI and everything else that's well maintained and suitable for Cloudron, I'd probably look at Quay which supports LDAP auth.

https://github.com/quay/quay

mehdi

I am 100% in favor of bundling a simple UI together with the registry. Even if one does not need it and wants to use the gitlab UI, there's basically nothing to lose besides a few kB of storage ^^

jimcavoli

Yeah, Quay and Harbor are definitely the big players in this space. Very similar products - harbor is CNCF graduated and Quay is upstream for the corresponding Red Hat product. Either (or both) would be good UI adds.

girish

Last I checked harbor was impractical to package (as in way too much effort, it's really geared for the k8s crowd). Quay is a good option, but let me get this basic docker registry out first, I am almost there.

mario

@jimcavoli Quay afaik implements the protocol as well, so no need for registry separately.

girish

So strange, I am getting a "invalid checksum digest format" whenever I push now to this registry. Has anyone seen such an error before?

The push refers to repository [xxx.xxx.xxx/cloudron/base]
fcdfeda3e242: Layer already exists 
0ea3bde29271: Layer already exists 
d75ccb14b8b6: Layer already exists 
74b4389a43ab: Layer already exists 
5f38ae1e1a63: Layer already exists 
3479c151673d: Layer already exists 
7a307b866f25: Layer already exists 
ce3a66c20e17: Layer already exists 
7197b970ebb9: Layer already exists 
16542a8fc3be: Layer already exists 
6597da2e2e52: Layer already exists 
977183d4e999: Layer already exists 
c8be1b8f4d60: Layer already exists 
invalid checksum digest format

mario

@girish local filesystem?

girish

@mario Yes, with the local storage. I wonder if it's something to do with the proxy auth. I am trying it without auth now.

edit: indeed, something to do with the proxy auth. It works fine without proxy auth. Debugging.

mehdi

@girish Are you on 6.1 ? Maybe your 2FA implementation broke something with the basic auth ?

girish

@mehdi yeah, i had that in mind and tried with 6.0 as well. fails the same. I am pretty sure this worked when I tested it back then, so I must have broke something !

mehdi

@girish You can try with an app-password, or try another Basic Auth ProxyAuth app, like Transmission (with an android app or a browser extension)

girish

What I am seeing is that docker doesn't send any authorization header at all. The issue is very similar to https://stackoverflow.com/questions/55516317/docker-login-not-passing-basic-authentication-headers-to-nginx . I can curl just fine.

girish

It seems that v2 registry auth does not use the basic bearer based authentication at all. https://docs.docker.com/registry/recipes/nginx/ is possibly obsolete, but I am trying to setup a registry from scratch now to double check.

mario

@girish it definitely can, that's how GitLab etc integration works.

mehdi

@girish Their doc indeed appears to be outdated. Different pages seem to indicate different things ...

girish

@mario thanks! i needed such a confident statement to help me keep looking further

I managed to get it to work. The issue is that proxyAuth on an auth fail redirects to the login page. But the docker registry wants it to return a 401 with a www-authenticate header. The header also causes issues with browsers since it starts popping up the login dialog.

In essence, even though the basic auth works, proxyAuth is not compatible. I thought about adding an flag to the manifest to have a different behavior but then again I don't like the current approach where we just install this registry and land on an empty page (any page even some static html with instructions would be better).

I ended up packaging it together the docker registry UI and a small LDAP server (from https://git.cloudron.io/cloudron/cloudron-serve). I haven't pushed the changes since they are not working entirely. But it's what I am working on in parallel with getting 6.1 out.

fbartels

@girish said in Docker registry:

I ended up packaging it together the docker registry UI and a small LDAP server

That sounds intriguing. What role does the ldap server serve? Just for auth against the registry ui?

mehdi

@girish I think the best approach would be to do a bit of user-agent parsing magic... Yeah, it would be quite specific for this use-case, but

girish

@fbartels said in Docker registry:

That sounds intriguing. What role does the ldap server serve? Just for auth against the registry ui?

Yes, pretty much. It's just a proxy that redirects to login page and auths against LDAP. The code itself is very small, just ~100 lines or so.

girish

@mehdi Right, I considered UA string hack but I think dropping users in a blank page is a bit rough. So, my first step was to do the UA testing with nginx in the app itself. But, that brought the dreaded browser auth modal dialog which I really dislike. It's the main reason I ended up making proxyAuth in the first place So.. I ended up making a node server.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Docker registry