--troubleshoot gave everything as "ok"
however:
root@my ~ # cloudron-support --check-db-migration
/usr/bin/cloudron-support: unrecognized option '--check-db-migration'
Running the start.sh fixed the backups page issue, and now the updates appear to be running (and the error is gone)
thankyou!
Our cloudron updated to v9 automatically, and ever since we've been getting "Table 'box.backupSites' doesn't exist" in our logs
The /backup-sites endpoint just spins (previous backup config not displayed) and when I try to add a new backup location we again get the error "Table 'box.backupSites' doesn't exist"
How to fix?
Is there a way to restrict access to a cloudron app to users on the cloudron VPN?
I did not realise this was a feature - it would be amazing, very (very) useful indeed
One thought is that now the usernames are "known", the attacker can continue the login attempts (even though they are futile).
So our new approach is to delete the old accounts and create new ones.
So it turns out this does NOT stop the "Unlock Instructions" email being sent. They even continue after forcing 2FA for all users.
What is more, we even get the emails for internal staff, who don't even have a password - because they authenticate using OIDC in cloudron.
Any suggestions?
Just discovered a setting at the following path: /admin/application_settings/general#js-visibility-settings
Section: Restricted visibility levels
Setting: Public - If selected, only administrators are able to create public groups, projects, and snippets. Also, profiles are only visible to authenticated users.
After checking this, and testing with CURL, the /api/v4/users/XXX endpoints now consistently return a 404 whether authenticated or not!!
I suspect this is the fix, but will wait and see if there are any more "Unlock Instructions" emails tonight / tomorrow.
Weirdly, after checking this checkbox and hitting save, it gets unchecked immediately after - but refreshing the page shows that it was indeed checked.
Another side note - we saw in our email logs that we were getting a large number of requests from a subdomain of https://academyforinternetresearch.org/
So it seems that this could be an issue on their radar.
So it appears that unauthenticated users (or attackers) are able to brute force usernames due to the fact that the corresponding API endpoints are not authenticated: https://gitlab.com/gitlab-org/gitlab/-/issues/297473
Furthermore, the gitlab team do not plan to fix the issue:
To mitigate the risk from such attacks in the future we took the following measures:
Actions taken on the server:
Actions taken on the platform (cloudron):
Actions taken on the gitlab instance (cloudron container):
Suggestions (to the packaging team) for improvement:
/home/git/gitlab/log). Maybe even putting that location in the file explorer, to make log capture / analysis easier.I found the logs - they were inside the container at /home/git/gitlab/log
Running grep -i "failed" revealed that the attack started in the early morning of 20th June. Somehow the list of usernames was known (probably relates to the issue in the link in my previous post) and signin requests are being made from random ip addresses.
First 5 entries shown below (this pattern has continued since):
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:17:13.349Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"156.146.59.50","meta.client_id":"ip/156.146.59.50","message":"Failed Login: username=xxx1 ip=156.146.59.50"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:18:20.163Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"193.176.84.35","meta.client_id":"ip/193.176.84.35","message":"Failed Login: username=xxx2 ip=193.176.84.35"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:18:39.636Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"20.205.138.223","meta.client_id":"ip/20.205.138.223","message":"Failed Login: username=xxxx3 ip=20.205.138.223"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:19:04.255Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"98.152.200.61","meta.client_id":"ip/98.152.200.61","message":"Failed Login: username=xxx4 ip=98.152.200.61"}
./application_json.log:{"severity":"INFO","time":"2025-07-20T03:21:03.314Z","correlation_id":"xxx","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"meta.remote_ip":"200.34.32.138","meta.client_id":"ip/200.34.32.138","message":"Failed Login: username=xxx5 ip=200.34.32.138"}
I found this thread which implies that it is a known issue in gitlab: https://gitlab.com/gitlab-org/gitlab/-/issues/297473
It was from my user but I also did this with one of the locked users and it was the same result (only shows successful logins). This was after the restart.
I found the cloudron app logs of the gitlab app under /home/yellowtent/platformdata/logs/$APPID and tried the following:
Neither gave any result.
I wasn't able to find the gitlab internal logs..
Gotcha. According to the link you provided:
Audit events of failed logins are currently recorded only on GitLab Starter and visible
under GitLab Premium (via Admin Area > Audit Log)🤕. Having those events surfaced
under Authentication log would means either one of these two things:
1. Move failed login audit events completely to GitLab Core
2. Add an extra EE version of Authentication log for licensed customer (i.e. for GitLab Premium)
Here is what I do see in that location:
So for some reason the failed events are not shown.
What is also interesting. When the event happened, the affected user(s) DID have a value in the "Locked account email verification code last sent at:" field under "/admin/users/XXXX". But after restarting the box, that entry is empty again. Not sure if that happened automatically after 10 minutes though.
I was thinking about cloudron, yes
The admin UI doesn't appear to have a "System Logs" option
The user accounts affected are indeed locked
Not on the day that it happened, happy to dm a (sanitised) copy
Running v1.104.4 since 5 days ago. Suddenly a large number of ourGitlab users have received the message below (both cloudron and external login accounts). There are no failed signin attempts in the log, that I can see. Did anyone else have this issue?
From: GitLab git.app@xxxx
Sent: xxxx
To: xxxxx
Subject: Unlock instructions
GitLab
Hello, xxxx!
Your GitLab account has been locked due to an excessive number of unsuccessful sign in attempts. You can wait for your account to automatically unlock in 10 minutes or you can click the link below to unlock now.
Unlock account
If you did not initiate these sign-in attempts, please reach out to your administrator or enable two-factor authentication (2FA) on your account.
Thankyou! This fixed it up.
We have started getting random repositories / users appear in our gitea instance, eg "AccidentInjuryLawyers". Before that, we had a sofa company. It looks like spam, I have to keep deleting them. How to prevent such signups?
Finally got things working again after taking the following steps:
Asking Hetzner to replace the machine (not drives) - although this didn't fix the issue, and may have been unnecessary
Opening the rescue system and running fsck -fy /dev/md2 per this guide: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/filesystem-check
This seemed to fix a load of disk issues and now everything is running fine.
Update - found the following entries in /mnt/var/logs/kern.log:
Mar 18 04:21:45 Ubuntu-2204-jammy-amd64-base kernel: [318852.281592] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=171.66.3.88 DST=135.181.208.188 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=22532 DF PROTO=TCP SPT=35116 DPT=8448 WINDOW=64240 RES=0x00 SYN URGP=0
Mar 18 04:21:52 Ubuntu-2204-jammy-amd64-base kernel: [318859.452276] traps: peertube[16048] trap int3 ip:1e7c0a2 sp:7ffd2cbb3540 error:0 in node[400000+4d82000]
Mar 18 04:22:17 Ubuntu-2204-jammy-amd64-base kernel: [318884.925726] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=117.241.169.216 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=19666 DF PROTO=TCP SPT=61376 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.590347] traps: node[6431] trap invalid opcode ip:98580d sp:7f99437fc788 error:0 in node[400000+3fa2000]
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752053] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752196] veth3fcec37: renamed from eth0
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.932254] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934204] device vethf657129 left promiscuous mode
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934209] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068033] br-529ee8c76856: port 6(veth35c0129) entered blocking state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068039] br-529ee8c76856: port 6(veth35c0129) entered disabled state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068499] device veth35c0129 entered promiscuous mode
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068733] br-529ee8c76856: port 6(veth35c0129) entered blocking state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068736] br-529ee8c76856: port 6(veth35c0129) entered forwarding state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.364420] eth0: renamed from veth7f4a61b
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.477049] IPv6: ADDRCONF(NETDEV_CHANGE): veth35c0129: link becomes ready
Mar 18 04:22:45 Ubuntu-2204-jammy-amd64-base kernel: [318912.782286] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=43.157.198.99 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=48 ID=24399 DF PROTO=TCP SPT=25423 DPT=8322 WINDOW=61690 RES=0x00 SYN URGP=0
Mar 18 04:23:18 Ubuntu-2204-jammy-amd64-base kernel: [318945.090800] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.150.24 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=52716 DPT=50994 WINDOW=65535 RES=0x00 SYN URGP=0
...
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.367304] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=131.196.169.70 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=116 ID=13790 DF PROTO=TCP SPT=51615 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.571834] traps: python3[12999] general protection fault ip:55f2c75e9533 sp:7fffba101780 error:0 in python3.10[55f2c7535000+2b1000]
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738177] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738513] veth6e69b9e: renamed from eth0
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.857409] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858868] device veth5f04ad6 left promiscuous mode
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858874] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976133] br-529ee8c76856: port 51(vethed47cd1) entered blocking state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976141] br-529ee8c76856: port 51(vethed47cd1) entered disabled state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976249] device vethed47cd1 entered promiscuous mode
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976548] br-529ee8c76856: port 51(vethed47cd1) entered blocking state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976553] br-529ee8c76856: port 51(vethed47cd1) entered forwarding state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.302125] eth0: renamed from veth0d5c1c0
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.385025] IPv6: ADDRCONF(NETDEV_CHANGE): vethed47cd1: link becomes ready
Mar 18 05:05:43 Ubuntu-2204-jammy-amd64-base kernel: [321490.900463] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=167.94.145.29 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=39 ID=34509 PROTO=TCP SPT=47410 DPT=5671 WINDOW=1024 RES=0x00 SYN URGP=0
Mar 18 05:06:16 Ubuntu-2204-jammy-amd64-base kernel: [321523.323443] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=9352 PROTO=TCP SPT=45117 DPT=24659 WINDOW=1024 RES=0x00 SYN URGP=0
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582718] http-server[716977]: segfault at 8 ip 0000000000f885df sp 00007ffc03770a68 error 4 in node[400000+4d82000]
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582732] Code: 89 fe 4c 89 e7 e8 91 c4 fe ff eb 92 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 c8 48 25 00 00 fc ff 80 7f 2a 00 75 1e <4c> 8b 40 08 41 f7 c0 00 00 10 00 0f 85 a0 00 00 00 41 f7 c0 00 00
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772510] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772603] veth3266337: renamed from eth0
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.896691] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898120] device veth84f6609 left promiscuous mode
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898123] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029787] br-529ee8c76856: port 26(vethe94cd48) entered blocking state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029792] br-529ee8c76856: port 26(vethe94cd48) entered disabled state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029922] device vethe94cd48 entered promiscuous mode
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030052] br-529ee8c76856: port 26(vethe94cd48) entered blocking state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030054] br-529ee8c76856: port 26(vethe94cd48) entered forwarding state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.228955] eth0: renamed from veth11cd8bb
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.294679] IPv6: ADDRCONF(NETDEV_CHANGE): vethe94cd48: link becomes ready
Mar 18 05:06:51 Ubuntu-2204-jammy-amd64-base kernel: [321558.189801] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.142 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=5666 PROTO=TCP SPT=58280 DPT=20000 WINDOW=1025 RES=0x00 SYN URGP=0
...
Mar 18 05:10:44 Ubuntu-2204-jammy-amd64-base kernel: [321791.258846] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=72.85.159.195 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=118 ID=24002 DF PROTO=TCP SPT=58004 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 18 05:10:46 Ubuntu-2204-jammy-amd64-base kernel: [321793.388813] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:10:49 Ubuntu-2204-jammy-amd64-base kernel: [321796.580665] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:11:20 Ubuntu-2204-jammy-amd64-base kernel: [321827.871185] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480
Mar 18 05:11:43 Ubuntu-2204-jammy-amd64-base kernel: [321850.887860] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=8 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480
Mar 18 05:12:21 Ubuntu-2204-jammy-amd64-base kernel: [321888.001543] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=121.137.215.153 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=47 ID=27760 PROTO=TCP SPT=48678 DPT=5555 WINDOW=15506 RES=0x00 SYN URGP=0
Mar 18 05:12:36 Ubuntu-2204-jammy-amd64-base kernel: [321903.649875] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:38 Ubuntu-2204-jammy-amd64-base kernel: [321905.462336] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.446469] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.765031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:44 Ubuntu-2204-jammy-amd64-base kernel: [321910.991925] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:47 Ubuntu-2204-jammy-amd64-base kernel: [321914.779540] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:54 Ubuntu-2204-jammy-amd64-base kernel: [321921.137411] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.246 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=38214 PROTO=TCP SPT=58717 DPT=19340 WINDOW=1025 RES=0x00 SYN URGP=0
Mar 18 05:13:06 Ubuntu-2204-jammy-amd64-base kernel: [321933.932419] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:86:dd SRC=2a06:4880:8000:0000:0000:0000:0000:009d DST=2a01:04f9:003a:26a7:0000:0000:0000:0002 LEN=64 TC=0 HOPLIMIT=245 FLOWLBL=120218 PROTO=TCP SPT=43770 DPT=17280 WINDOW=14600 RES=0x00 SYN URGP=0
Mar 18 05:13:16 Ubuntu-2204-jammy-amd64-base kernel: [321943.103123] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=35.203.210.132 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=56954 DPT=9536 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 05:13:20 Ubuntu-2204-jammy-amd64-base kernel: [321947.831031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:13:47 Ubuntu-2204-jammy-amd64-base kernel: [321974.682282] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.146 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=25084 PROTO=TCP SPT=58717 DPT=34227 WINDOW=1025 RES=0x00 SYN URGP=0
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181116] sshd[3585541]: segfault at 8 ip 00007febf3852001 sp 00007fff2ec0c9b8 error 6 in libwrap.so.0.7.6[7febf384f000+5000]
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181129] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.605248] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.194 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=47146 PROTO=TCP SPT=58702 DPT=6227 WINDOW=1025 RES=0x00 SYN URGP=0
Mar 18 05:14:46 Ubuntu-2204-jammy-amd64-base kernel: [322033.390701] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=41109 PROTO=TCP SPT=45117 DPT=60640 WINDOW=1024 RES=0x00 SYN URGP=0
Mar 18 05:15:15 Ubuntu-2204-jammy-amd64-base kernel: [322062.318473] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=205.210.31.235 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=49467 DPT=8088 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577857] sshd[3586207]: segfault at 8 ip 00007fdbf075f001 sp 00007fff0156c978 error 6 in libwrap.so.0.7.6[7fdbf075c000+5000]
Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577884] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956091] sshd[3586428]: segfault at 8 ip 00007f348f597001 sp 00007fff7318dac8 error 6 in libwrap.so.0.7.6[7f348f594000+5000]
Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956103] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426077] sshd[3586463]: segfault at 8 ip 00007fc9da4f5001 sp 00007ffe697f7678 error 6 in libwrap.so.0.7.6[7fc9da4f2000+5000]
Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426090] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:47 Ubuntu-2204-jammy-amd64-base kernel: [322094.050525] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=63495 PROTO=TCP SPT=45117 DPT=18564 WINDOW=1024 RES=0x00 SYN URGP=0
...
Mar 18 05:18:46 Ubuntu-2204-jammy-amd64-base kernel: [322273.803053] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=220.135.239.243 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=53 ID=25239 DF PROTO=TCP SPT=63737 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566504] sshd[3588460]: segfault at 8 ip 00007efe8ed10001 sp 00007ffca1d4c3e8 error 6 in libwrap.so.0.7.6[7efe8ed0d000+5000]
Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566516] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:19:32 Ubuntu-2204-jammy-amd64-base kernel: [322319.942150] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.149.104 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=51737 DPT=46123 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 05:19:49 Ubuntu-2204-jammy-amd64-base kernel: [322336.877850] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.40.70 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=12083 PROTO=TCP SPT=59137 DPT=36507 WINDOW=1025 RES=0x00 SYN URGP=0
Mar 18 05:20:13 Ubuntu-2204-jammy-amd64-base kernel: [322360.607262] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=80.75.212.75 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=54321 PROTO=TCP SPT=53436 DPT=11955 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 05:20:44 Ubuntu-2204-jammy-amd64-base kernel: [322391.418386] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.130 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=1343 PROTO=TCP SPT=59154 DPT=43259 WINDOW=1025 RES=0x00 SYN URGP=0
Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521570] sshd[3589634]: segfault at 8 ip 00007f8716350001 sp 00007ffe66709038 error 6 in libwrap.so.0.7.6[7f871634d000+5000]
Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521582] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:21:29 Ubuntu-2204-jammy-amd64-base kernel: [322436.294493] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=68.183.193.242 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=54321 PROTO=TCP SPT=52091 DPT=10443 WINDOW=65535 RES=0x00 SYN URGP=0
...
Mar 18 05:32:17 Ubuntu-2204-jammy-amd64-base kernel: [323084.689916] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=193.163.125.66 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=248 ID=3486 PROTO=TCP SPT=35193 DPT=50501 WINDOW=14600 RES=0x00 SYN URGP=0
Today our cloudron instance (on a hetzner dedicated machine) became unresponsive, both to web and ssh requests
Power cycling did not help. Booting into recovery mode, the last lines of the box.log are as follows:
2024-03-18T05:29:50.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:00.005Z box:disks checkDiskSpace: checking disk space
2024-03-18T05:30:00.042Z box:janitor Cleaning up expired tokens
2024-03-18T05:30:00.044Z box:eventlog cleanup: pruning events. creationTime: Tue Dec 19 2023 05:30:00 GMT+0000 (Coordinated Universal Time)
2024-03-18T05:30:00.046Z box:tasks startTask - starting task 6699 with options {}. logs at /home/yellowtent/platformdata/logs/tasks/6699.log
2024-03-18T05:30:00.046Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6699 /home/yellowtent/platformdata/logs/tasks/6699.log 0 400
2024-03-18T05:30:00.054Z box:janitor Cleaned up 0 expired tokens
2024-03-18T05:30:00.126Z box:shell startTask (stderr): Running as unit: box-task-6699.service
2024-03-18T05:30:00.400Z box:disks checkDiskSpace: disk space checked. out of space: no
2024-03-18T05:30:00.431Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:10.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:20.210Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:30.215Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:40.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:50.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:00.211Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:10.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:20.227Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:25.823Z box:shell startTask (stderr): Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 1min 25.708s
CPU time consumed: 6.591s
2024-03-18T05:31:25.829Z box:shell startTask (stdout): Service box-task-6699 finished with exit code 0
2024-03-18T05:31:25.831Z box:tasks startTask: 6699 completed with code 0
2024-03-18T05:31:25.833Z box:tasks startTask: 6699 done. error: null
2024-03-18T05:31:30.205Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:40.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:50.204Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:00.209Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:10.220Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:20.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:30.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
df output:
Any idea how we can further troubleshoot?