Cloudron unresponsive, cannot ssh
-
Today our cloudron instance (on a hetzner dedicated machine) became unresponsive, both to web and ssh requests
Power cycling did not help. Booting into recovery mode, the last lines of the box.log are as follows:
2024-03-18T05:29:50.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:30:00.005Z box:disks checkDiskSpace: checking disk space 2024-03-18T05:30:00.042Z box:janitor Cleaning up expired tokens 2024-03-18T05:30:00.044Z box:eventlog cleanup: pruning events. creationTime: Tue Dec 19 2023 05:30:00 GMT+0000 (Coordinated Universal Time) 2024-03-18T05:30:00.046Z box:tasks startTask - starting task 6699 with options {}. logs at /home/yellowtent/platformdata/logs/tasks/6699.log 2024-03-18T05:30:00.046Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6699 /home/yellowtent/platformdata/logs/tasks/6699.log 0 400 2024-03-18T05:30:00.054Z box:janitor Cleaned up 0 expired tokens 2024-03-18T05:30:00.126Z box:shell startTask (stderr): Running as unit: box-task-6699.service 2024-03-18T05:30:00.400Z box:disks checkDiskSpace: disk space checked. out of space: no 2024-03-18T05:30:00.431Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:30:10.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:30:20.210Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:30:30.215Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:30:40.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:30:50.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:31:00.211Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:31:10.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:31:20.227Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:31:25.823Z box:shell startTask (stderr): Finished with result: success Main processes terminated with: code=exited/status=0 Service runtime: 1min 25.708s CPU time consumed: 6.591s 2024-03-18T05:31:25.829Z box:shell startTask (stdout): Service box-task-6699 finished with exit code 0 2024-03-18T05:31:25.831Z box:tasks startTask: 6699 completed with code 0 2024-03-18T05:31:25.833Z box:tasks startTask: 6699 done. error: null 2024-03-18T05:31:30.205Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:31:40.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:31:50.204Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:32:00.209Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:32:10.220Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:32:20.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive 2024-03-18T05:32:30.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsivedf output:
Any idea how we can further troubleshoot?
-
Update - found the following entries in
/mnt/var/logs/kern.log:Mar 18 04:21:45 Ubuntu-2204-jammy-amd64-base kernel: [318852.281592] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=171.66.3.88 DST=135.181.208.188 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=22532 DF PROTO=TCP SPT=35116 DPT=8448 WINDOW=64240 RES=0x00 SYN URGP=0 Mar 18 04:21:52 Ubuntu-2204-jammy-amd64-base kernel: [318859.452276] traps: peertube[16048] trap int3 ip:1e7c0a2 sp:7ffd2cbb3540 error:0 in node[400000+4d82000] Mar 18 04:22:17 Ubuntu-2204-jammy-amd64-base kernel: [318884.925726] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=117.241.169.216 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=19666 DF PROTO=TCP SPT=61376 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.590347] traps: node[6431] trap invalid opcode ip:98580d sp:7f99437fc788 error:0 in node[400000+3fa2000] Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752053] br-529ee8c76856: port 6(vethf657129) entered disabled state Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752196] veth3fcec37: renamed from eth0 Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.932254] br-529ee8c76856: port 6(vethf657129) entered disabled state Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934204] device vethf657129 left promiscuous mode Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934209] br-529ee8c76856: port 6(vethf657129) entered disabled state Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068033] br-529ee8c76856: port 6(veth35c0129) entered blocking state Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068039] br-529ee8c76856: port 6(veth35c0129) entered disabled state Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068499] device veth35c0129 entered promiscuous mode Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068733] br-529ee8c76856: port 6(veth35c0129) entered blocking state Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068736] br-529ee8c76856: port 6(veth35c0129) entered forwarding state Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.364420] eth0: renamed from veth7f4a61b Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.477049] IPv6: ADDRCONF(NETDEV_CHANGE): veth35c0129: link becomes ready Mar 18 04:22:45 Ubuntu-2204-jammy-amd64-base kernel: [318912.782286] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=43.157.198.99 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=48 ID=24399 DF PROTO=TCP SPT=25423 DPT=8322 WINDOW=61690 RES=0x00 SYN URGP=0 Mar 18 04:23:18 Ubuntu-2204-jammy-amd64-base kernel: [318945.090800] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.150.24 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=52716 DPT=50994 WINDOW=65535 RES=0x00 SYN URGP=0 ... Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.367304] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=131.196.169.70 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=116 ID=13790 DF PROTO=TCP SPT=51615 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.571834] traps: python3[12999] general protection fault ip:55f2c75e9533 sp:7fffba101780 error:0 in python3.10[55f2c7535000+2b1000] Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738177] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738513] veth6e69b9e: renamed from eth0 Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.857409] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858868] device veth5f04ad6 left promiscuous mode Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858874] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976133] br-529ee8c76856: port 51(vethed47cd1) entered blocking state Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976141] br-529ee8c76856: port 51(vethed47cd1) entered disabled state Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976249] device vethed47cd1 entered promiscuous mode Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976548] br-529ee8c76856: port 51(vethed47cd1) entered blocking state Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976553] br-529ee8c76856: port 51(vethed47cd1) entered forwarding state Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.302125] eth0: renamed from veth0d5c1c0 Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.385025] IPv6: ADDRCONF(NETDEV_CHANGE): vethed47cd1: link becomes ready Mar 18 05:05:43 Ubuntu-2204-jammy-amd64-base kernel: [321490.900463] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=167.94.145.29 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=39 ID=34509 PROTO=TCP SPT=47410 DPT=5671 WINDOW=1024 RES=0x00 SYN URGP=0 Mar 18 05:06:16 Ubuntu-2204-jammy-amd64-base kernel: [321523.323443] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=9352 PROTO=TCP SPT=45117 DPT=24659 WINDOW=1024 RES=0x00 SYN URGP=0 Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582718] http-server[716977]: segfault at 8 ip 0000000000f885df sp 00007ffc03770a68 error 4 in node[400000+4d82000] Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582732] Code: 89 fe 4c 89 e7 e8 91 c4 fe ff eb 92 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 c8 48 25 00 00 fc ff 80 7f 2a 00 75 1e <4c> 8b 40 08 41 f7 c0 00 00 10 00 0f 85 a0 00 00 00 41 f7 c0 00 00 Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772510] br-529ee8c76856: port 26(veth84f6609) entered disabled state Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772603] veth3266337: renamed from eth0 Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.896691] br-529ee8c76856: port 26(veth84f6609) entered disabled state Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898120] device veth84f6609 left promiscuous mode Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898123] br-529ee8c76856: port 26(veth84f6609) entered disabled state Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029787] br-529ee8c76856: port 26(vethe94cd48) entered blocking state Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029792] br-529ee8c76856: port 26(vethe94cd48) entered disabled state Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029922] device vethe94cd48 entered promiscuous mode Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030052] br-529ee8c76856: port 26(vethe94cd48) entered blocking state Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030054] br-529ee8c76856: port 26(vethe94cd48) entered forwarding state Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.228955] eth0: renamed from veth11cd8bb Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.294679] IPv6: ADDRCONF(NETDEV_CHANGE): vethe94cd48: link becomes ready Mar 18 05:06:51 Ubuntu-2204-jammy-amd64-base kernel: [321558.189801] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.142 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=5666 PROTO=TCP SPT=58280 DPT=20000 WINDOW=1025 RES=0x00 SYN URGP=0 ... Mar 18 05:10:44 Ubuntu-2204-jammy-amd64-base kernel: [321791.258846] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=72.85.159.195 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=118 ID=24002 DF PROTO=TCP SPT=58004 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 Mar 18 05:10:46 Ubuntu-2204-jammy-amd64-base kernel: [321793.388813] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:10:49 Ubuntu-2204-jammy-amd64-base kernel: [321796.580665] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:11:20 Ubuntu-2204-jammy-amd64-base kernel: [321827.871185] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480 Mar 18 05:11:43 Ubuntu-2204-jammy-amd64-base kernel: [321850.887860] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=8 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480 Mar 18 05:12:21 Ubuntu-2204-jammy-amd64-base kernel: [321888.001543] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=121.137.215.153 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=47 ID=27760 PROTO=TCP SPT=48678 DPT=5555 WINDOW=15506 RES=0x00 SYN URGP=0 Mar 18 05:12:36 Ubuntu-2204-jammy-amd64-base kernel: [321903.649875] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:12:38 Ubuntu-2204-jammy-amd64-base kernel: [321905.462336] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.446469] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.765031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:12:44 Ubuntu-2204-jammy-amd64-base kernel: [321910.991925] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:12:47 Ubuntu-2204-jammy-amd64-base kernel: [321914.779540] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:12:54 Ubuntu-2204-jammy-amd64-base kernel: [321921.137411] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.246 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=38214 PROTO=TCP SPT=58717 DPT=19340 WINDOW=1025 RES=0x00 SYN URGP=0 Mar 18 05:13:06 Ubuntu-2204-jammy-amd64-base kernel: [321933.932419] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:86:dd SRC=2a06:4880:8000:0000:0000:0000:0000:009d DST=2a01:04f9:003a:26a7:0000:0000:0000:0002 LEN=64 TC=0 HOPLIMIT=245 FLOWLBL=120218 PROTO=TCP SPT=43770 DPT=17280 WINDOW=14600 RES=0x00 SYN URGP=0 Mar 18 05:13:16 Ubuntu-2204-jammy-amd64-base kernel: [321943.103123] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=35.203.210.132 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=56954 DPT=9536 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 18 05:13:20 Ubuntu-2204-jammy-amd64-base kernel: [321947.831031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0 Mar 18 05:13:47 Ubuntu-2204-jammy-amd64-base kernel: [321974.682282] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.146 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=25084 PROTO=TCP SPT=58717 DPT=34227 WINDOW=1025 RES=0x00 SYN URGP=0 Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181116] sshd[3585541]: segfault at 8 ip 00007febf3852001 sp 00007fff2ec0c9b8 error 6 in libwrap.so.0.7.6[7febf384f000+5000] Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181129] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.605248] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.194 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=47146 PROTO=TCP SPT=58702 DPT=6227 WINDOW=1025 RES=0x00 SYN URGP=0 Mar 18 05:14:46 Ubuntu-2204-jammy-amd64-base kernel: [322033.390701] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=41109 PROTO=TCP SPT=45117 DPT=60640 WINDOW=1024 RES=0x00 SYN URGP=0 Mar 18 05:15:15 Ubuntu-2204-jammy-amd64-base kernel: [322062.318473] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=205.210.31.235 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=49467 DPT=8088 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577857] sshd[3586207]: segfault at 8 ip 00007fdbf075f001 sp 00007fff0156c978 error 6 in libwrap.so.0.7.6[7fdbf075c000+5000] Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577884] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956091] sshd[3586428]: segfault at 8 ip 00007f348f597001 sp 00007fff7318dac8 error 6 in libwrap.so.0.7.6[7f348f594000+5000] Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956103] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426077] sshd[3586463]: segfault at 8 ip 00007fc9da4f5001 sp 00007ffe697f7678 error 6 in libwrap.so.0.7.6[7fc9da4f2000+5000] Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426090] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c Mar 18 05:15:47 Ubuntu-2204-jammy-amd64-base kernel: [322094.050525] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=63495 PROTO=TCP SPT=45117 DPT=18564 WINDOW=1024 RES=0x00 SYN URGP=0 ... Mar 18 05:18:46 Ubuntu-2204-jammy-amd64-base kernel: [322273.803053] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=220.135.239.243 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=53 ID=25239 DF PROTO=TCP SPT=63737 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566504] sshd[3588460]: segfault at 8 ip 00007efe8ed10001 sp 00007ffca1d4c3e8 error 6 in libwrap.so.0.7.6[7efe8ed0d000+5000] Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566516] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c Mar 18 05:19:32 Ubuntu-2204-jammy-amd64-base kernel: [322319.942150] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.149.104 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=51737 DPT=46123 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 18 05:19:49 Ubuntu-2204-jammy-amd64-base kernel: [322336.877850] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.40.70 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=12083 PROTO=TCP SPT=59137 DPT=36507 WINDOW=1025 RES=0x00 SYN URGP=0 Mar 18 05:20:13 Ubuntu-2204-jammy-amd64-base kernel: [322360.607262] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=80.75.212.75 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=54321 PROTO=TCP SPT=53436 DPT=11955 WINDOW=65535 RES=0x00 SYN URGP=0 Mar 18 05:20:44 Ubuntu-2204-jammy-amd64-base kernel: [322391.418386] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.130 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=1343 PROTO=TCP SPT=59154 DPT=43259 WINDOW=1025 RES=0x00 SYN URGP=0 Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521570] sshd[3589634]: segfault at 8 ip 00007f8716350001 sp 00007ffe66709038 error 6 in libwrap.so.0.7.6[7f871634d000+5000] Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521582] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c Mar 18 05:21:29 Ubuntu-2204-jammy-amd64-base kernel: [322436.294493] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=68.183.193.242 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=54321 PROTO=TCP SPT=52091 DPT=10443 WINDOW=65535 RES=0x00 SYN URGP=0 ... Mar 18 05:32:17 Ubuntu-2204-jammy-amd64-base kernel: [323084.689916] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=193.163.125.66 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=248 ID=3486 PROTO=TCP SPT=35193 DPT=50501 WINDOW=14600 RES=0x00 SYN URGP=0 -
Finally got things working again after taking the following steps:
-
Asking Hetzner to replace the machine (not drives) - although this didn't fix the issue, and may have been unnecessary
-
Opening the rescue system and running
fsck -fy /dev/md2per this guide: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/filesystem-check
This seemed to fix a load of disk issues and now everything is running fine.
-
-
N nebulon marked this topic as a question on
-
N nebulon has marked this topic as solved on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login