I looked through the logs again today and found something conspicuous:

Apr 23 10:57:32 from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:178:in `new_message_ids' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:163:in `new_messages' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:135:in `process_mailbox' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:45:in `wait' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/mailbox_watcher.rb:37:in `block in run' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client/connection_mixin.rb:71:in `call_pipelined': WRONGPASS invalid username-password pair or user is disabled. (redis://redis-19727d9f-cec9-4b73-8b77-3cf41a03008c:6379) (RedisClient::AuthenticationError) <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client.rb:771:in `block in connect' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client/middlewares.rb:16:in `call' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client.rb:770:in `connect' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client.rb:732:in `raw_connection' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client.rb:697:in `ensure_connected' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client.rb:292:in `call_v' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-5.0.8/lib/redis/client.rb:90:in `call_v' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-5.0.8/lib/redis.rb:152:in `block in send_command' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-5.0.8/lib/redis.rb:151:in `synchronize' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-5.0.8/lib/redis.rb:151:in `send_command' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-5.0.8/lib/redis/commands/strings.rb:95:in `set' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-namespace-1.10.0/lib/redis/namespace.rb:558:in `wrapped_send' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-namespace-1.10.0/lib/redis/namespace.rb:515:in `call_with_namespace' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-namespace-1.10.0/lib/redis/namespace.rb:389:in `block (2 levels) in <class:Namespace>' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/arbitration/redis.rb:41:in `deliver?' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/mailbox.rb:108:in `deliver?' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:178:in `block in new_message_ids' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:178:in `select' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:178:in `new_message_ids' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:163:in `new_messages' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:135:in `process_mailbox' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/imap/connection.rb:45:in `wait' <30>1 2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - from /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/gitlab-mail_room-0.0.24/lib/mail_room/mailbox_watcher.rb:37:in `block in run' Apr 23 10:57:32 2024-04-23 08:57:32,492 INFO exited: mail_room (exit status 1; not expected) Apr 23 10:57:33 2024-04-23 08:57:33,494 INFO spawned: 'mail_room' with pid 60817

I find that in particular a bit strange:

2024-04-23T08:57:32Z cbe-svpv-http01 19727d9f-cec9-4b73-8b77-3cf41a03008c 1126 19727d9f-cec9-4b73-8b77-3cf41a03008c - /home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/redis-client-0.21.1/lib/redis_client/connection_mixin.rb:71:in `call_pipelined': WRONGPASS invalid username-password pair or user is disabled. (redis://redis-19727d9f-cec9-4b73-8b77-3cf41a03008c:6379) (RedisClient::AuthenticationError) <30>1

FYI the changed username fixed it, everything is running fine now. 👍

@Zoldir Looks like you are using http auth, is that correct? Any reason to not use SSH for git push/pull?

In any case, the recent update of GitLab moved from LDAP to OIDC. Once you login to gitlab , visit https://git.domain.com/-/user_settings/personal_access_tokens. You can use that personal access token for HTTP auth ("You can also use personal access tokens to authenticate against Git over HTTP.")

Damned, it was too obvious to be seen.

It works, thanks 😄

I did read through those pages, and that is in fact how I arrived at the queue state representation that I posted above. Unfortunately clearing the queue did not help for long: The tasks kept reappearing.

I have, however -through many trials and tribulations- possibly managed to resolve this problem over the weekend: Logging into the GitLab postgres DB, I found a table called container_repositories which contained references to the repos that were used to test the registry feature. After truncating that table, clearing the queued jobs and restarting GitLab (as well as the entire instance for good measure), it seems to have slowly reverted back to normal over the past 24 hours, at least for the time being. I'm gonna keep an eye on it some more, but it's looking like that might've been it.

It's a bit weird that GitLab would keep spinning tasks endlessly like this, but then again it's a massive piece of software, so perhaps this is simply a strange bug in GitLab itself. In any case, the container registry instructions for GitLab integration did not seem to work that well for us as they produce this behavior whenever I try them out (I tried it twice to make sure), so maybe another look at those could be warranted.

@timconsidine since the difference is mostly the postgresql fixes, it might be worth updating already in your case.

Yes after upgrading it solved

This should be fixed from 16.7 onwards I believe.

@djxx I wouldn't say running a CI runner a "little thing" 🙂 But anyway, it's up to you if you feel comfortable and are aware of the repercussions of installing things give access to all your containers.

I have no idea about sysbox, never tried it.

It's better now, using 3250MB of RAM.

@girish I'll test it and let you know, but if multiDomain doesn't cut it i'm pretty sure httpsPorts would, at least for the basic setups.

@opensourced that warning comes from bundler and it can be ignored (it's not an error).

Are you able to see the issue on git.cloudron.io ?

@girish I actually managed to get this running. The lower end of the section in the docs did the trick.

This usually happens when docker registry is down (or is undergoing some maintanence).

@andreas
That front end code and the code server ware out of sync.
So any file change on the front end was not updated un till a commit was done.

For example if you have a Vue app, vite will not update your dev build untill you didn't commit

Problem with your hoster or DNS maybe? Logs are as they should be. Do you have to error messsage from your failed git push?

@allanbowe Please try with the latest package. For good measure, I added the client_max_body_size directive to the top level.

@tobiasReichel Is it an option to use the Jumpcloud LDAP integration using https://docs.cloudron.io/user-management/#external-directory ? This way you don't need to configure GitLab specifically to use Jumpcloud.