I would advise to temporary edit the GitLab NGINX file to only allow certain IP addresses.
This manual change will get reset with every Cloudron / Server / App restart.
So it is really temporary.

Example for APP ID 682ca768-93e5-4bcb-a760-677daa9a8e3b

Go into the application NGINX config folder:

cd /home/yellowtent/platformdata/nginx/applications/682ca768-93e5-4bcb-a760-677daa9a8e3b

Edit the sub.domain.tld.conf file, in this case dokuwiki.cloudron.dev:

nano dokuwiki.cloudron.dev.conf

Inside this section, add:

# https server
server {
    
    # allow localhost
    allow 127.0.0.1;
    # allow cloudron proxy
    allow 172.18.0.1;
    # allow this servers public ipv4
    allow REDACTED-IPV4;
    # allow this servers public ipv6
    allow REDACTED-IPV6;
    # Allow some other specific IPv4 e.g VPN
    allow VPN-IP;
    # deny all other
    deny all;
    [...]
}

Reload the NGINX service:

systemctl reload nginx.service

This will result for other IPs that are not explicitly allowed to return a 403 Forbidden:

Keep in mind, every Cloudron / Server / App Restart will reset this change!

I did not realise this was a feature - it would be amazing, very (very) useful indeed

So our new approach is to delete the old accounts and create new ones.

Any suggestions?

Section: Restricted visibility levels
Setting: Public - If selected, only administrators are able to create public groups, projects, and snippets. Also, profiles are only visible to authenticated users.

After checking this, and testing with CURL, the /api/v4/users/XXX endpoints now consistently return a 404 whether authenticated or not!!

I suspect this is the fix, but will wait and see if there are any more "Unlock Instructions" emails tonight / tomorrow.

Weirdly, after checking this checkbox and hitting save, it gets unchecked immediately after - but refreshing the page shows that it was indeed checked.

Another side note - we saw in our email logs that we were getting a large number of requests from a subdomain of https://academyforinternetresearch.org/

So it seems that this could be an issue on their radar.

Suggestions (to the packaging team) for improvement:

A hardened Gitlab configuration "out of the box" in cloudron
Updates to the documentation (eg that the logs location is under /home/git/gitlab/log). Maybe even putting that location in the file explorer, to make log capture / analysis easier.
Options within the cloudron platform itself to more aggressively reject IP addresses. It was noted that some attacker IPs were re-used after some time.

Agreed.

I'd add: copy the salient bits of above post (or at least link to it) into the docs too.

Furthermore, the gitlab team do not plan to fix the issue:

• https://gitlab.com/gitlab-org/gitlab/-/issues/16179
• https://gitlab.com/gitlab-org/gitlab/-/issues/336601

To mitigate the risk from such attacks in the future we took the following measures:

Actions taken on the server:

• Installed Fail2ban

Actions taken on the platform (cloudron):

• Removed several platform apps that were not being used
• Restricted visibility of (and access to) the gitlab instance to just those who need it
• Removed several users

Actions taken on the gitlab instance (cloudron container):

• Enabled 2FA (we had it natively on cloudron / OIDC but not for guest users)
• Removed Oauth apps that are no longer used
• Enabled "Deactivate dormant users after a period of inactivity" (90 days)
• Enabled "Enable unauthenticated API request rate limit" (1 per second)
• Enabled "Enable unauthenticated web request rate limit" (1 per second)
• Enabled "Enable authenticated API request rate limit" (2 per second)
• Enabled "Enable authenticated API request rate limit" (2 per second)
• Deleted 3 inactive runners and removed one active but no longer needed runner
• Removed dormant users

Suggestions (to the packaging team) for improvement:

• A hardened Gitlab configuration "out of the box" in cloudron
• Updates to the documentation (eg that the logs location is under /home/git/gitlab/log). Maybe even putting that location in the file explorer, to make log capture / analysis easier.
• Options within the cloudron platform itself to more aggressively reject IP addresses. It was noted that some attacker IPs were re-used after some time.

Running grep -i "failed" revealed that the attack started in the early morning of 20th June. Somehow the list of usernames was known (probably relates to the issue in the link in my previous post) and signin requests are being made from random ip addresses.

First 5 entries shown below (this pattern has continued since):

./application_json.log:{"severity":"INFO","time":"2025-07-20T03:17:13.349Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"156.146.59.50","meta.client_id":"ip/156.146.59.50","message":"Failed Login: username=xxx1 ip=156.146.59.50"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:18:20.163Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"193.176.84.35","meta.client_id":"ip/193.176.84.35","message":"Failed Login: username=xxx2 ip=193.176.84.35"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:18:39.636Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"20.205.138.223","meta.client_id":"ip/20.205.138.223","message":"Failed Login: username=xxxx3 ip=20.205.138.223"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:19:04.255Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"98.152.200.61","meta.client_id":"ip/98.152.200.61","message":"Failed Login: username=xxx4 ip=98.152.200.61"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:21:03.314Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"200.34.32.138","meta.client_id":"ip/200.34.32.138","message":"Failed Login: username=xxx5 ip=200.34.32.138"}

Neither gave any result.

I wasn't able to find the gitlab internal logs..

Audit events of failed logins are currently recorded only on GitLab Starter and visible 
under GitLab Premium (via Admin Area > Audit Log)🤕. Having those events surfaced 
under Authentication log would means either one of these two things:

1. Move failed login audit events completely to GitLab Core
2. Add an extra EE version of Authentication log for licensed customer (i.e. for GitLab Premium)

Here is what I do see in that location:

So for some reason the failed events are not shown.

What is also interesting. When the event happened, the affected user(s) DID have a value in the "Locked account email verification code last sent at:" field under "/admin/users/XXXX". But after restarting the box, that entry is empty again. Not sure if that happened automatically after 10 minutes though.

The admin UI doesn't appear to have a "System Logs" option

The user accounts affected are indeed locked