@nebulon said in Gigantic blunder caused by a lack of warnings: Curiously nothing about that possible side-effect is mentioned in the gitlab docs either https://docs.gitlab.com/administration/incoming_email/ or at least I didn't see it. It could be clearer but it does say „We recommend using an email address that receives only messages that are intended for the GitLab instance“ and it goes on about „dedicated“ addresses….

@timconsidine Yes, this is what I run today and it has worked really, really well ... integrates nicely with Gitlab. But the deprecation the Gitlab folks have in mind for 3rd party registries is a worry (https://docs.gitlab.com/administration/packages/container_registry/#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). There's talk about them doing a hard deprecation which I'm taking to mean would stop it working completely rather than just being unsupported as today

Hello @allanbowe Not by default from Cloudron. (Maybe in the future) I would advise to temporary edit the GitLab NGINX file to only allow certain IP addresses. This manual change will get reset with every Cloudron / Server / App restart. So it is really temporary. Example for APP ID 682ca768-93e5-4bcb-a760-677daa9a8e3b Go into the application NGINX config folder: cd /home/yellowtent/platformdata/nginx/applications/682ca768-93e5-4bcb-a760-677daa9a8e3b Edit the sub.domain.tld.conf file, in this case dokuwiki.cloudron.dev: nano dokuwiki.cloudron.dev.conf Inside this section, add: # https server server { [...] # allow localhost allow 127.0.0.1; # allow cloudron proxy allow 172.18.0.1; # allow this servers public ipv4 allow REDACTED-IPV4; # allow this servers public ipv6 allow REDACTED-IPV6; # Allow some other specific IPv4 e.g VPN allow VPN-IP; # deny all other deny all; [...] } Reload the NGINX service: systemctl reload nginx.service This will result for other IPs that are not explicitly allowed to return a 403 Forbidden: [image: 1753639364452-52b04087-811c-4d9f-8a90-2b20cb5de9f4-image.png] Keep in mind, every Cloudron / Server / App Restart will reset this change!

Thank you, Nebulon, for your reply. The runners are on a different machine. I don't think that the machine with the runners is running out of space because it has a very large amount available on all partitions (the root has 300GB available). I think the error is originating from the cloudron machine (although I may be wrong). In order to try to confirm the location of the error, I will setup another gitlab runner client on a different machine, and see if that has the same problem. Hopefully that will isolate the error either to the Cloudron server, or else the gitlab runner machines.

@arnaudguy yes

Thanks for the answers—so to be clear it’s not possible to reduce the features as their docs suggest? (Through some rb config) To be honest the gitea instance was hacked, so that was not ideal, and I switched from gogs to gitea awhile ago and cannot exactly recall why.

@lukaszj could it be same as https://gitlab.com/gitlab-org/gitlab/-/issues/348565 ? From a comment there, One "workarround" that seems to work is to trigger the HouseKeeping on the general settings . The issue is still not fixed but maybe you can look for log string "Internal API error" ? There are some detailed logs in /run/gitlab/log/ (only available via Web Terminal)

@AriusII No plans for next 3 months atleast (but that's as far as our planning goes). But given EE is selfhosted, it's not outside our scope.

So despite having 11Gb free, the restore fails due to insufficient disk space? I am not sure how we can help here though. Can you install a fresh gitlab instance fine?

@girish Works like a charm. Many thanks for the fix!

FYI the changed username fixed it, everything is running fine now.

@Zoldir Looks like you are using http auth, is that correct? Any reason to not use SSH for git push/pull? In any case, the recent update of GitLab moved from LDAP to OIDC. Once you login to gitlab , visit https://git.domain.com/-/user_settings/personal_access_tokens. You can use that personal access token for HTTP auth ("You can also use personal access tokens to authenticate against Git over HTTP.")

Damned, it was too obvious to be seen. It works, thanks

I did read through those pages, and that is in fact how I arrived at the queue state representation that I posted above. Unfortunately clearing the queue did not help for long: The tasks kept reappearing. I have, however -through many trials and tribulations- possibly managed to resolve this problem over the weekend: Logging into the GitLab postgres DB, I found a table called container_repositories which contained references to the repos that were used to test the registry feature. After truncating that table, clearing the queued jobs and restarting GitLab (as well as the entire instance for good measure), it seems to have slowly reverted back to normal over the past 24 hours, at least for the time being. I'm gonna keep an eye on it some more, but it's looking like that might've been it. It's a bit weird that GitLab would keep spinning tasks endlessly like this, but then again it's a massive piece of software, so perhaps this is simply a strange bug in GitLab itself. In any case, the container registry instructions for GitLab integration did not seem to work that well for us as they produce this behavior whenever I try them out (I tried it twice to make sure), so maybe another look at those could be warranted.

@timconsidine since the difference is mostly the postgresql fixes, it might be worth updating already in your case.

Yes after upgrading it solved

This should be fixed from 16.7 onwards I believe.

@djxx I wouldn't say running a CI runner a "little thing" But anyway, it's up to you if you feel comfortable and are aware of the repercussions of installing things give access to all your containers. I have no idea about sysbox, never tried it.