Cloudron Forum

Thankyou! This fixed it up.

We have started getting random repositories / users appear in our gitea instance, eg "AccidentInjuryLawyers". Before that, we had a sofa company. It looks like spam, I have to keep deleting them. How to prevent such signups?

https://git.datacontroller.io/explore/repos

Finally got things working again after taking the following steps:

1. Asking Hetzner to replace the machine (not drives) - although this didn't fix the issue, and may have been unnecessary

2. Opening the rescue system and running fsck -fy /dev/md2 per this guide: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/filesystem-check

This seemed to fix a load of disk issues and now everything is running fine.

Update - found the following entries in /mnt/var/logs/kern.log:

Mar 18 04:21:45 Ubuntu-2204-jammy-amd64-base kernel: [318852.281592] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=171.66.3.88 DST=135.181.208.188 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=22532 DF PROTO=TCP SPT=35116 DPT=8448 WINDOW=64240 RES=0x00 SYN URGP=0 
Mar 18 04:21:52 Ubuntu-2204-jammy-amd64-base kernel: [318859.452276] traps: peertube[16048] trap int3 ip:1e7c0a2 sp:7ffd2cbb3540 error:0 in node[400000+4d82000]
Mar 18 04:22:17 Ubuntu-2204-jammy-amd64-base kernel: [318884.925726] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=117.241.169.216 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=19666 DF PROTO=TCP SPT=61376 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.590347] traps: node[6431] trap invalid opcode ip:98580d sp:7f99437fc788 error:0 in node[400000+3fa2000]
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752053] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752196] veth3fcec37: renamed from eth0
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.932254] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934204] device vethf657129 left promiscuous mode
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934209] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068033] br-529ee8c76856: port 6(veth35c0129) entered blocking state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068039] br-529ee8c76856: port 6(veth35c0129) entered disabled state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068499] device veth35c0129 entered promiscuous mode
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068733] br-529ee8c76856: port 6(veth35c0129) entered blocking state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068736] br-529ee8c76856: port 6(veth35c0129) entered forwarding state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.364420] eth0: renamed from veth7f4a61b
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.477049] IPv6: ADDRCONF(NETDEV_CHANGE): veth35c0129: link becomes ready
Mar 18 04:22:45 Ubuntu-2204-jammy-amd64-base kernel: [318912.782286] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=43.157.198.99 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=48 ID=24399 DF PROTO=TCP SPT=25423 DPT=8322 WINDOW=61690 RES=0x00 SYN URGP=0 
Mar 18 04:23:18 Ubuntu-2204-jammy-amd64-base kernel: [318945.090800] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.150.24 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=52716 DPT=50994 WINDOW=65535 RES=0x00 SYN URGP=0 
...
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.367304] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=131.196.169.70 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=116 ID=13790 DF PROTO=TCP SPT=51615 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.571834] traps: python3[12999] general protection fault ip:55f2c75e9533 sp:7fffba101780 error:0 in python3.10[55f2c7535000+2b1000]
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738177] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738513] veth6e69b9e: renamed from eth0
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.857409] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858868] device veth5f04ad6 left promiscuous mode
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858874] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976133] br-529ee8c76856: port 51(vethed47cd1) entered blocking state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976141] br-529ee8c76856: port 51(vethed47cd1) entered disabled state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976249] device vethed47cd1 entered promiscuous mode
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976548] br-529ee8c76856: port 51(vethed47cd1) entered blocking state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976553] br-529ee8c76856: port 51(vethed47cd1) entered forwarding state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.302125] eth0: renamed from veth0d5c1c0
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.385025] IPv6: ADDRCONF(NETDEV_CHANGE): vethed47cd1: link becomes ready
Mar 18 05:05:43 Ubuntu-2204-jammy-amd64-base kernel: [321490.900463] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=167.94.145.29 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=39 ID=34509 PROTO=TCP SPT=47410 DPT=5671 WINDOW=1024 RES=0x00 SYN URGP=0 
Mar 18 05:06:16 Ubuntu-2204-jammy-amd64-base kernel: [321523.323443] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=9352 PROTO=TCP SPT=45117 DPT=24659 WINDOW=1024 RES=0x00 SYN URGP=0 
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582718] http-server[716977]: segfault at 8 ip 0000000000f885df sp 00007ffc03770a68 error 4 in node[400000+4d82000]
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582732] Code: 89 fe 4c 89 e7 e8 91 c4 fe ff eb 92 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 c8 48 25 00 00 fc ff 80 7f 2a 00 75 1e <4c> 8b 40 08 41 f7 c0 00 00 10 00 0f 85 a0 00 00 00 41 f7 c0 00 00
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772510] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772603] veth3266337: renamed from eth0
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.896691] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898120] device veth84f6609 left promiscuous mode
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898123] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029787] br-529ee8c76856: port 26(vethe94cd48) entered blocking state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029792] br-529ee8c76856: port 26(vethe94cd48) entered disabled state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029922] device vethe94cd48 entered promiscuous mode
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030052] br-529ee8c76856: port 26(vethe94cd48) entered blocking state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030054] br-529ee8c76856: port 26(vethe94cd48) entered forwarding state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.228955] eth0: renamed from veth11cd8bb
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.294679] IPv6: ADDRCONF(NETDEV_CHANGE): vethe94cd48: link becomes ready
Mar 18 05:06:51 Ubuntu-2204-jammy-amd64-base kernel: [321558.189801] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.142 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=5666 PROTO=TCP SPT=58280 DPT=20000 WINDOW=1025 RES=0x00 SYN URGP=0 
...
Mar 18 05:10:44 Ubuntu-2204-jammy-amd64-base kernel: [321791.258846] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=72.85.159.195 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=118 ID=24002 DF PROTO=TCP SPT=58004 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 05:10:46 Ubuntu-2204-jammy-amd64-base kernel: [321793.388813] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:10:49 Ubuntu-2204-jammy-amd64-base kernel: [321796.580665] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:11:20 Ubuntu-2204-jammy-amd64-base kernel: [321827.871185] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480 
Mar 18 05:11:43 Ubuntu-2204-jammy-amd64-base kernel: [321850.887860] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=8 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480 
Mar 18 05:12:21 Ubuntu-2204-jammy-amd64-base kernel: [321888.001543] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=121.137.215.153 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=47 ID=27760 PROTO=TCP SPT=48678 DPT=5555 WINDOW=15506 RES=0x00 SYN URGP=0 
Mar 18 05:12:36 Ubuntu-2204-jammy-amd64-base kernel: [321903.649875] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:38 Ubuntu-2204-jammy-amd64-base kernel: [321905.462336] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.446469] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.765031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:44 Ubuntu-2204-jammy-amd64-base kernel: [321910.991925] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:47 Ubuntu-2204-jammy-amd64-base kernel: [321914.779540] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:54 Ubuntu-2204-jammy-amd64-base kernel: [321921.137411] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.246 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=38214 PROTO=TCP SPT=58717 DPT=19340 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:13:06 Ubuntu-2204-jammy-amd64-base kernel: [321933.932419] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:86:dd SRC=2a06:4880:8000:0000:0000:0000:0000:009d DST=2a01:04f9:003a:26a7:0000:0000:0000:0002 LEN=64 TC=0 HOPLIMIT=245 FLOWLBL=120218 PROTO=TCP SPT=43770 DPT=17280 WINDOW=14600 RES=0x00 SYN URGP=0 
Mar 18 05:13:16 Ubuntu-2204-jammy-amd64-base kernel: [321943.103123] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=35.203.210.132 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=56954 DPT=9536 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:13:20 Ubuntu-2204-jammy-amd64-base kernel: [321947.831031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:13:47 Ubuntu-2204-jammy-amd64-base kernel: [321974.682282] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.146 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=25084 PROTO=TCP SPT=58717 DPT=34227 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181116] sshd[3585541]: segfault at 8 ip 00007febf3852001 sp 00007fff2ec0c9b8 error 6 in libwrap.so.0.7.6[7febf384f000+5000]
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181129] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.605248] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.194 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=47146 PROTO=TCP SPT=58702 DPT=6227 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:14:46 Ubuntu-2204-jammy-amd64-base kernel: [322033.390701] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=41109 PROTO=TCP SPT=45117 DPT=60640 WINDOW=1024 RES=0x00 SYN URGP=0 
Mar 18 05:15:15 Ubuntu-2204-jammy-amd64-base kernel: [322062.318473] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=205.210.31.235 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=49467 DPT=8088 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577857] sshd[3586207]: segfault at 8 ip 00007fdbf075f001 sp 00007fff0156c978 error 6 in libwrap.so.0.7.6[7fdbf075c000+5000]
Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577884] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956091] sshd[3586428]: segfault at 8 ip 00007f348f597001 sp 00007fff7318dac8 error 6 in libwrap.so.0.7.6[7f348f594000+5000]
Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956103] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426077] sshd[3586463]: segfault at 8 ip 00007fc9da4f5001 sp 00007ffe697f7678 error 6 in libwrap.so.0.7.6[7fc9da4f2000+5000]
Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426090] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:47 Ubuntu-2204-jammy-amd64-base kernel: [322094.050525] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=63495 PROTO=TCP SPT=45117 DPT=18564 WINDOW=1024 RES=0x00 SYN URGP=0 
...
Mar 18 05:18:46 Ubuntu-2204-jammy-amd64-base kernel: [322273.803053] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=220.135.239.243 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=53 ID=25239 DF PROTO=TCP SPT=63737 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566504] sshd[3588460]: segfault at 8 ip 00007efe8ed10001 sp 00007ffca1d4c3e8 error 6 in libwrap.so.0.7.6[7efe8ed0d000+5000]
Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566516] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:19:32 Ubuntu-2204-jammy-amd64-base kernel: [322319.942150] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.149.104 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=51737 DPT=46123 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:19:49 Ubuntu-2204-jammy-amd64-base kernel: [322336.877850] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.40.70 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=12083 PROTO=TCP SPT=59137 DPT=36507 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:20:13 Ubuntu-2204-jammy-amd64-base kernel: [322360.607262] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=80.75.212.75 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=54321 PROTO=TCP SPT=53436 DPT=11955 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:20:44 Ubuntu-2204-jammy-amd64-base kernel: [322391.418386] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.130 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=1343 PROTO=TCP SPT=59154 DPT=43259 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521570] sshd[3589634]: segfault at 8 ip 00007f8716350001 sp 00007ffe66709038 error 6 in libwrap.so.0.7.6[7f871634d000+5000]
Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521582] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:21:29 Ubuntu-2204-jammy-amd64-base kernel: [322436.294493] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=68.183.193.242 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=54321 PROTO=TCP SPT=52091 DPT=10443 WINDOW=65535 RES=0x00 SYN URGP=0 
...
Mar 18 05:32:17 Ubuntu-2204-jammy-amd64-base kernel: [323084.689916] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=193.163.125.66 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=248 ID=3486 PROTO=TCP SPT=35193 DPT=50501 WINDOW=14600 RES=0x00 SYN URGP=0 

Today our cloudron instance (on a hetzner dedicated machine) became unresponsive, both to web and ssh requests

Power cycling did not help. Booting into recovery mode, the last lines of the box.log are as follows:

2024-03-18T05:29:50.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:00.005Z box:disks checkDiskSpace: checking disk space
2024-03-18T05:30:00.042Z box:janitor Cleaning up expired tokens
2024-03-18T05:30:00.044Z box:eventlog cleanup: pruning events. creationTime: Tue Dec 19 2023 05:30:00 GMT+0000 (Coordinated Universal Time)
2024-03-18T05:30:00.046Z box:tasks startTask - starting task 6699 with options {}. logs at /home/yellowtent/platformdata/logs/tasks/6699.log
2024-03-18T05:30:00.046Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6699 /home/yellowtent/platformdata/logs/tasks/6699.log 0 400
2024-03-18T05:30:00.054Z box:janitor Cleaned up 0 expired tokens
2024-03-18T05:30:00.126Z box:shell startTask (stderr): Running as unit: box-task-6699.service

2024-03-18T05:30:00.400Z box:disks checkDiskSpace: disk space checked. out of space: no
2024-03-18T05:30:00.431Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:10.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:20.210Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:30.215Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:40.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:50.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:00.211Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:10.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:20.227Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:25.823Z box:shell startTask (stderr): Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 1min 25.708s
CPU time consumed: 6.591s

2024-03-18T05:31:25.829Z box:shell startTask (stdout): Service box-task-6699 finished with exit code 0

2024-03-18T05:31:25.831Z box:tasks startTask: 6699 completed with code 0
2024-03-18T05:31:25.833Z box:tasks startTask: 6699 done. error: null
2024-03-18T05:31:30.205Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:40.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:50.204Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:00.209Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:10.220Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:20.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:30.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive

df output:

Any idea how we can further troubleshoot?

This is a great proposal - how do we vote for it?

@oj the problem with the cloudron jitsi install is the lack of a TURN server, which means it is not possible to have video calls with customers on enterprise platforms (that require traffic over 443)

So, any major customer basically.

We would love to see the following:

• TURN server support for jitsi (to enable enterprise calls over 443)
• Well Known support - to enable arbitrary well-known configs (not just a predefined list)

@nebulon possibly..

search also doesn't work in Brave, but does in Firefox.

Actually - I was able to fix this by taking two actions:

1. Increase memory to 640mb
2. Use Firefox instead of Brave

API keys now generate successfully.

If I go to settings / api / Create New Token I am presented with an empty dialog box.

The network request returns a 302 as follows:

The console contains the following error:

vendor.js?id=26004b76c717902bb63e:2 TypeError: Cannot read properties of undefined (reading 'id')
    at app.js?id=a95fae2c1c73027afa79:2:236145
    at a.zt [as _l] (vendor.js?id=26004b76c717902bb63e:2:762963)
    at a.<anonymous> (app.js?id=a95fae2c1c73027afa79:2:236098)
    at e._render (vendor.js?id=26004b76c717902bb63e:2:782289)
    at a.r (vendor.js?id=26004b76c717902bb63e:2:815411)
    at yn.get (vendor.js?id=26004b76c717902bb63e:2:773680)
    at yn.run (vendor.js?id=26004b76c717902bb63e:2:774564)
    at vn (vendor.js?id=26004b76c717902bb63e:2:772672)
    at Array.<anonymous> (vendor.js?id=26004b76c717902bb63e:2:759271)
    at et (vendor.js?id=26004b76c717902bb63e:2:758670)

How to obtain an API key?

Edit: it's definitely jitsi in Matrix, this is what we were presented with for our 3 way (successful) video call, with screen sharing:

Was just stock cloudron matrix, using the stock cloudron element web client on the customer side of the call.

We could not do a successful 3-way video call with this same customer, on the same browser, using the stock standalone cloudron jitsi instance.

We're on cloudron 7.2.5

We've had several instances now where clients (usually from government or large financial firms) are unable to use our cloudron Jitsi instance. They can connect but muted and no video.

We did an experiment today and found that the embedded Matrix instance of jitsi DOES work ok. So it's definitely not a browser issue on their side.

Rather, I suspect it is this specific configuration issue https://stackoverflow.com/questions/65916064/jitsi-for-corporate-usage

Could we request for the default jitsi to be set up to keep all traffic over 443? This should result in a far more reliable system for business web conferencing.

More info:

• https://jitsi.github.io/handbook/docs/devops-guide/turn/#use-turn-server-on-port-443
• https://github.com/jitsi/jitsi-meet/issues/6807
• https://github.com/jitsi/jitsi-meet/issues/929

EDIT - I see now this is an open issue, slated for a future release: https://git.cloudron.io/cloudron/box/-/issues/764

@robi yes - it appears that way - I tried to modify the nginx.conf but it's read only from the cloudron shell

When pushing artefacts in a gitlab runner, we get the following message:

in text form:

[2:26:53 PM] [semantic-release] [@semantic-release/gitlab] › ✖  An error occurred while uploading /builds/clients/dwp_frs/artefacts/viya.sas to the GitLab project uploads API:
HTTPError: Response code 413 (Request Entity Too Large)
    at Request.<anonymous> (/builds/clients/dwp_frs/node_modules/got/dist/source/as-promise/index.js:118:42)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'ERR_NON_2XX_3XX_RESPONSE',
  timings: {
    start: 1662647207472,
    socket: 1662647207474,
    lookup: 1662647207481,
    connect: 1662647207520,
    secureConnect: 1662647207559,
    upload: 1662647213057,
    response: 1662647213651,
    end: 1662647213651,
    error: undefined,
    abort: undefined,
    phases: {
      wait: 2,
      dns: 7,
      tcp: 39,
      tls: 39,
      request: 5498,
      firstByte: 594,
      download: 0,
      total: 6179
    }
  }
}

We tried increasing the max artifacts size as described here, but the error persists, even after a restart:

Is there something we are missing?