Hey @djxx I made this topic into a normal topic for now since this seems more like a discussion than an issue that can be marked as solved. I hope you find this agreeable and if not, feel free to comment and mark this topic as a question again.

Thank you, Nebulon, for your reply. The runners are on a different machine. I don't think that the machine with the runners is running out of space because it has a very large amount available on all partitions (the root has 300GB available). I think the error is originating from the cloudron machine (although I may be wrong). In order to try to confirm the location of the error, I will setup another gitlab runner client on a different machine, and see if that has the same problem. Hopefully that will isolate the error either to the Cloudron server, or else the gitlab runner machines.

@arnaudguy yes

Thanks for the answers—so to be clear it’s not possible to reduce the features as their docs suggest? (Through some rb config) To be honest the gitea instance was hacked, so that was not ideal, and I switched from gogs to gitea awhile ago and cannot exactly recall why.

@lukaszj could it be same as https://gitlab.com/gitlab-org/gitlab/-/issues/348565 ? From a comment there, One "workarround" that seems to work is to trigger the HouseKeeping on the general settings . The issue is still not fixed but maybe you can look for log string "Internal API error" ? There are some detailed logs in /run/gitlab/log/ (only available via Web Terminal)

@AriusII No plans for next 3 months atleast (but that's as far as our planning goes). But given EE is selfhosted, it's not outside our scope.

So despite having 11Gb free, the restore fails due to insufficient disk space? I am not sure how we can help here though. Can you install a fresh gitlab instance fine?

@girish Works like a charm. Many thanks for the fix!

FYI the changed username fixed it, everything is running fine now.

@Zoldir Looks like you are using http auth, is that correct? Any reason to not use SSH for git push/pull? In any case, the recent update of GitLab moved from LDAP to OIDC. Once you login to gitlab , visit https://git.domain.com/-/user_settings/personal_access_tokens. You can use that personal access token for HTTP auth ("You can also use personal access tokens to authenticate against Git over HTTP.")

Damned, it was too obvious to be seen. It works, thanks

I did read through those pages, and that is in fact how I arrived at the queue state representation that I posted above. Unfortunately clearing the queue did not help for long: The tasks kept reappearing. I have, however -through many trials and tribulations- possibly managed to resolve this problem over the weekend: Logging into the GitLab postgres DB, I found a table called container_repositories which contained references to the repos that were used to test the registry feature. After truncating that table, clearing the queued jobs and restarting GitLab (as well as the entire instance for good measure), it seems to have slowly reverted back to normal over the past 24 hours, at least for the time being. I'm gonna keep an eye on it some more, but it's looking like that might've been it. It's a bit weird that GitLab would keep spinning tasks endlessly like this, but then again it's a massive piece of software, so perhaps this is simply a strange bug in GitLab itself. In any case, the container registry instructions for GitLab integration did not seem to work that well for us as they produce this behavior whenever I try them out (I tried it twice to make sure), so maybe another look at those could be warranted.

@timconsidine since the difference is mostly the postgresql fixes, it might be worth updating already in your case.

Yes after upgrading it solved

This should be fixed from 16.7 onwards I believe.

@djxx I wouldn't say running a CI runner a "little thing" But anyway, it's up to you if you feel comfortable and are aware of the repercussions of installing things give access to all your containers. I have no idea about sysbox, never tried it.

It's better now, using 3250MB of RAM.

@girish I'll test it and let you know, but if multiDomain doesn't cut it i'm pretty sure httpsPorts would, at least for the basic setups.