Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor
  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum
Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Cloudron unresponsive, cannot ssh

Cloudron unresponsive, cannot ssh

Scheduled Pinned Locked Moved Solved Support
sshhetznerfsck
4 Posts 2 Posters 189 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • allanboweA Offline
    allanboweA Offline
    allanbowe
    wrote on last edited by girish
    #1

    Today our cloudron instance (on a hetzner dedicated machine) became unresponsive, both to web and ssh requests

    Power cycling did not help. Booting into recovery mode, the last lines of the box.log are as follows:

    2024-03-18T05:29:50.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:00.005Z box:disks checkDiskSpace: checking disk space
2024-03-18T05:30:00.042Z box:janitor Cleaning up expired tokens
2024-03-18T05:30:00.044Z box:eventlog cleanup: pruning events. creationTime: Tue Dec 19 2023 05:30:00 GMT+0000 (Coordinated Universal Time)
2024-03-18T05:30:00.046Z box:tasks startTask - starting task 6699 with options {}. logs at /home/yellowtent/platformdata/logs/tasks/6699.log
2024-03-18T05:30:00.046Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6699 /home/yellowtent/platformdata/logs/tasks/6699.log 0 400
2024-03-18T05:30:00.054Z box:janitor Cleaned up 0 expired tokens
2024-03-18T05:30:00.126Z box:shell startTask (stderr): Running as unit: box-task-6699.service

2024-03-18T05:30:00.400Z box:disks checkDiskSpace: disk space checked. out of space: no
2024-03-18T05:30:00.431Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:10.217Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:20.210Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:30.215Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:40.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:30:50.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:00.211Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:10.202Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:20.227Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:25.823Z box:shell startTask (stderr): Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 1min 25.708s
CPU time consumed: 6.591s

2024-03-18T05:31:25.829Z box:shell startTask (stdout): Service box-task-6699 finished with exit code 0

2024-03-18T05:31:25.831Z box:tasks startTask: 6699 completed with code 0
2024-03-18T05:31:25.833Z box:tasks startTask: 6699 done. error: null
2024-03-18T05:31:30.205Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:40.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:31:50.204Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:00.209Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:10.220Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:20.208Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive
2024-03-18T05:32:30.216Z box:apphealthmonitor app health: 43 running / 2 stopped / 0 unresponsive

    df output:

    d287ed89-4790-4aca-aea0-25dd88f16d9a-image.png

    Any idea how we can further troubleshoot?

    1 Reply Last reply
    2
    • allanboweA Offline
      allanboweA Offline
      allanbowe
      wrote on last edited by
      #2

      Update - found the following entries in /mnt/var/logs/kern.log:

      Mar 18 04:21:45 Ubuntu-2204-jammy-amd64-base kernel: [318852.281592] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=171.66.3.88 DST=135.181.208.188 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=22532 DF PROTO=TCP SPT=35116 DPT=8448 WINDOW=64240 RES=0x00 SYN URGP=0 
Mar 18 04:21:52 Ubuntu-2204-jammy-amd64-base kernel: [318859.452276] traps: peertube[16048] trap int3 ip:1e7c0a2 sp:7ffd2cbb3540 error:0 in node[400000+4d82000]
Mar 18 04:22:17 Ubuntu-2204-jammy-amd64-base kernel: [318884.925726] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=117.241.169.216 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=19666 DF PROTO=TCP SPT=61376 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.590347] traps: node[6431] trap invalid opcode ip:98580d sp:7f99437fc788 error:0 in node[400000+3fa2000]
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752053] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.752196] veth3fcec37: renamed from eth0
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.932254] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934204] device vethf657129 left promiscuous mode
Mar 18 04:22:26 Ubuntu-2204-jammy-amd64-base kernel: [318893.934209] br-529ee8c76856: port 6(vethf657129) entered disabled state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068033] br-529ee8c76856: port 6(veth35c0129) entered blocking state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068039] br-529ee8c76856: port 6(veth35c0129) entered disabled state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068499] device veth35c0129 entered promiscuous mode
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068733] br-529ee8c76856: port 6(veth35c0129) entered blocking state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.068736] br-529ee8c76856: port 6(veth35c0129) entered forwarding state
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.364420] eth0: renamed from veth7f4a61b
Mar 18 04:22:27 Ubuntu-2204-jammy-amd64-base kernel: [318894.477049] IPv6: ADDRCONF(NETDEV_CHANGE): veth35c0129: link becomes ready
Mar 18 04:22:45 Ubuntu-2204-jammy-amd64-base kernel: [318912.782286] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=43.157.198.99 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=48 ID=24399 DF PROTO=TCP SPT=25423 DPT=8322 WINDOW=61690 RES=0x00 SYN URGP=0 
Mar 18 04:23:18 Ubuntu-2204-jammy-amd64-base kernel: [318945.090800] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.150.24 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=52716 DPT=50994 WINDOW=65535 RES=0x00 SYN URGP=0 
...
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.367304] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=131.196.169.70 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=116 ID=13790 DF PROTO=TCP SPT=51615 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.571834] traps: python3[12999] general protection fault ip:55f2c75e9533 sp:7fffba101780 error:0 in python3.10[55f2c7535000+2b1000]
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738177] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.738513] veth6e69b9e: renamed from eth0
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.857409] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858868] device veth5f04ad6 left promiscuous mode
Mar 18 05:05:19 Ubuntu-2204-jammy-amd64-base kernel: [321466.858874] br-529ee8c76856: port 51(veth5f04ad6) entered disabled state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976133] br-529ee8c76856: port 51(vethed47cd1) entered blocking state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976141] br-529ee8c76856: port 51(vethed47cd1) entered disabled state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976249] device vethed47cd1 entered promiscuous mode
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976548] br-529ee8c76856: port 51(vethed47cd1) entered blocking state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321466.976553] br-529ee8c76856: port 51(vethed47cd1) entered forwarding state
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.302125] eth0: renamed from veth0d5c1c0
Mar 18 05:05:20 Ubuntu-2204-jammy-amd64-base kernel: [321467.385025] IPv6: ADDRCONF(NETDEV_CHANGE): vethed47cd1: link becomes ready
Mar 18 05:05:43 Ubuntu-2204-jammy-amd64-base kernel: [321490.900463] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=167.94.145.29 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=39 ID=34509 PROTO=TCP SPT=47410 DPT=5671 WINDOW=1024 RES=0x00 SYN URGP=0 
Mar 18 05:06:16 Ubuntu-2204-jammy-amd64-base kernel: [321523.323443] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=9352 PROTO=TCP SPT=45117 DPT=24659 WINDOW=1024 RES=0x00 SYN URGP=0 
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582718] http-server[716977]: segfault at 8 ip 0000000000f885df sp 00007ffc03770a68 error 4 in node[400000+4d82000]
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.582732] Code: 89 fe 4c 89 e7 e8 91 c4 fe ff eb 92 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 c8 48 25 00 00 fc ff 80 7f 2a 00 75 1e <4c> 8b 40 08 41 f7 c0 00 00 10 00 0f 85 a0 00 00 00 41 f7 c0 00 00
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772510] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.772603] veth3266337: renamed from eth0
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.896691] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898120] device veth84f6609 left promiscuous mode
Mar 18 05:06:48 Ubuntu-2204-jammy-amd64-base kernel: [321555.898123] br-529ee8c76856: port 26(veth84f6609) entered disabled state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029787] br-529ee8c76856: port 26(vethe94cd48) entered blocking state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029792] br-529ee8c76856: port 26(vethe94cd48) entered disabled state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.029922] device vethe94cd48 entered promiscuous mode
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030052] br-529ee8c76856: port 26(vethe94cd48) entered blocking state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.030054] br-529ee8c76856: port 26(vethe94cd48) entered forwarding state
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.228955] eth0: renamed from veth11cd8bb
Mar 18 05:06:49 Ubuntu-2204-jammy-amd64-base kernel: [321556.294679] IPv6: ADDRCONF(NETDEV_CHANGE): vethe94cd48: link becomes ready
Mar 18 05:06:51 Ubuntu-2204-jammy-amd64-base kernel: [321558.189801] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.142 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=5666 PROTO=TCP SPT=58280 DPT=20000 WINDOW=1025 RES=0x00 SYN URGP=0 
...
Mar 18 05:10:44 Ubuntu-2204-jammy-amd64-base kernel: [321791.258846] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=72.85.159.195 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=118 ID=24002 DF PROTO=TCP SPT=58004 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 05:10:46 Ubuntu-2204-jammy-amd64-base kernel: [321793.388813] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:10:49 Ubuntu-2204-jammy-amd64-base kernel: [321796.580665] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:11:20 Ubuntu-2204-jammy-amd64-base kernel: [321827.871185] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480 
Mar 18 05:11:43 Ubuntu-2204-jammy-amd64-base kernel: [321850.887860] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=139.45.200.129 DST=135.181.208.188 LEN=1500 TOS=0x00 PREC=0x00 TTL=8 ID=0 DF PROTO=UDP SPT=49670 DPT=42436 LEN=1480 
Mar 18 05:12:21 Ubuntu-2204-jammy-amd64-base kernel: [321888.001543] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=121.137.215.153 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=47 ID=27760 PROTO=TCP SPT=48678 DPT=5555 WINDOW=15506 RES=0x00 SYN URGP=0 
Mar 18 05:12:36 Ubuntu-2204-jammy-amd64-base kernel: [321903.649875] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:38 Ubuntu-2204-jammy-amd64-base kernel: [321905.462336] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.446469] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:41 Ubuntu-2204-jammy-amd64-base kernel: [321908.765031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:44 Ubuntu-2204-jammy-amd64-base kernel: [321910.991925] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:47 Ubuntu-2204-jammy-amd64-base kernel: [321914.779540] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:12:54 Ubuntu-2204-jammy-amd64-base kernel: [321921.137411] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.246 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=38214 PROTO=TCP SPT=58717 DPT=19340 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:13:06 Ubuntu-2204-jammy-amd64-base kernel: [321933.932419] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:86:dd SRC=2a06:4880:8000:0000:0000:0000:0000:009d DST=2a01:04f9:003a:26a7:0000:0000:0000:0002 LEN=64 TC=0 HOPLIMIT=245 FLOWLBL=120218 PROTO=TCP SPT=43770 DPT=17280 WINDOW=14600 RES=0x00 SYN URGP=0 
Mar 18 05:13:16 Ubuntu-2204-jammy-amd64-base kernel: [321943.103123] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=35.203.210.132 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=56954 DPT=9536 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:13:20 Ubuntu-2204-jammy-amd64-base kernel: [321947.831031] EXT4-fs error (device md2): htree_dirblock_to_tree:1108: inode #527919: block 85950503: comm imap: bad entry in directory: directory entry overrun - offset=152644, inode=1294872888, rec_len=12852, size=4096 fake=0
Mar 18 05:13:47 Ubuntu-2204-jammy-amd64-base kernel: [321974.682282] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.146 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=25084 PROTO=TCP SPT=58717 DPT=34227 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181116] sshd[3585541]: segfault at 8 ip 00007febf3852001 sp 00007fff2ec0c9b8 error 6 in libwrap.so.0.7.6[7febf384f000+5000]
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.181129] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:14:26 Ubuntu-2204-jammy-amd64-base kernel: [322013.605248] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.60.194 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=47146 PROTO=TCP SPT=58702 DPT=6227 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:14:46 Ubuntu-2204-jammy-amd64-base kernel: [322033.390701] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=41109 PROTO=TCP SPT=45117 DPT=60640 WINDOW=1024 RES=0x00 SYN URGP=0 
Mar 18 05:15:15 Ubuntu-2204-jammy-amd64-base kernel: [322062.318473] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=205.210.31.235 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=49467 DPT=8088 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577857] sshd[3586207]: segfault at 8 ip 00007fdbf075f001 sp 00007fff0156c978 error 6 in libwrap.so.0.7.6[7fdbf075c000+5000]
Mar 18 05:15:27 Ubuntu-2204-jammy-amd64-base kernel: [322074.577884] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956091] sshd[3586428]: segfault at 8 ip 00007f348f597001 sp 00007fff7318dac8 error 6 in libwrap.so.0.7.6[7f348f594000+5000]
Mar 18 05:15:38 Ubuntu-2204-jammy-amd64-base kernel: [322085.956103] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426077] sshd[3586463]: segfault at 8 ip 00007fc9da4f5001 sp 00007ffe697f7678 error 6 in libwrap.so.0.7.6[7fc9da4f2000+5000]
Mar 18 05:15:46 Ubuntu-2204-jammy-amd64-base kernel: [322093.426090] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:15:47 Ubuntu-2204-jammy-amd64-base kernel: [322094.050525] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=45.137.201.9 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=63495 PROTO=TCP SPT=45117 DPT=18564 WINDOW=1024 RES=0x00 SYN URGP=0 
...
Mar 18 05:18:46 Ubuntu-2204-jammy-amd64-base kernel: [322273.803053] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=220.135.239.243 DST=135.181.208.188 LEN=52 TOS=0x00 PREC=0x00 TTL=53 ID=25239 DF PROTO=TCP SPT=63737 DPT=445 WINDOW=8192 RES=0x00 SYN URGP=0 
Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566504] sshd[3588460]: segfault at 8 ip 00007efe8ed10001 sp 00007ffca1d4c3e8 error 6 in libwrap.so.0.7.6[7efe8ed0d000+5000]
Mar 18 05:18:56 Ubuntu-2204-jammy-amd64-base kernel: [322283.566516] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:19:32 Ubuntu-2204-jammy-amd64-base kernel: [322319.942150] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=162.216.149.104 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=51737 DPT=46123 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:19:49 Ubuntu-2204-jammy-amd64-base kernel: [322336.877850] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=79.124.40.70 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=12083 PROTO=TCP SPT=59137 DPT=36507 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:20:13 Ubuntu-2204-jammy-amd64-base kernel: [322360.607262] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=80.75.212.75 DST=135.181.208.188 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=54321 PROTO=TCP SPT=53436 DPT=11955 WINDOW=65535 RES=0x00 SYN URGP=0 
Mar 18 05:20:44 Ubuntu-2204-jammy-amd64-base kernel: [322391.418386] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=91.148.190.130 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=249 ID=1343 PROTO=TCP SPT=59154 DPT=43259 WINDOW=1025 RES=0x00 SYN URGP=0 
Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521570] sshd[3589634]: segfault at 8 ip 00007f8716350001 sp 00007ffe66709038 error 6 in libwrap.so.0.7.6[7f871634d000+5000]
Mar 18 05:20:54 Ubuntu-2204-jammy-amd64-base kernel: [322401.521582] Code: 89 ee 48 8d 3d a4 22 00 00 31 c0 e8 69 e8 ff ff eb b6 e8 22 d7 ff ff 66 90 f3 0f 1e fa 41 55 41 54 55 53 48 81 ec 00 10 00 00 <4f> 83 0c 24 00 48 81 ec 0d 10 00 00 48 83 0c 24 02 48 83 ec 28 4c
Mar 18 05:21:29 Ubuntu-2204-jammy-amd64-base kernel: [322436.294493] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=68.183.193.242 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=54321 PROTO=TCP SPT=52091 DPT=10443 WINDOW=65535 RES=0x00 SYN URGP=0 
...
Mar 18 05:32:17 Ubuntu-2204-jammy-amd64-base kernel: [323084.689916] Packet dropped: IN=enp35s0 OUT= MAC=a8:a1:59:48:f5:e6:4c:6d:58:45:29:f9:08:00 SRC=193.163.125.66 DST=135.181.208.188 LEN=44 TOS=0x00 PREC=0x00 TTL=248 ID=3486 PROTO=TCP SPT=35193 DPT=50501 WINDOW=14600 RES=0x00 SYN URGP=0
      1 Reply Last reply
      0
      • allanboweA Offline
        allanboweA Offline
        allanbowe
        wrote on last edited by
        #3

        Finally got things working again after taking the following steps:

        1. Asking Hetzner to replace the machine (not drives) - although this didn't fix the issue, and may have been unnecessary

        2. Opening the rescue system and running fsck -fy /dev/md2 per this guide: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/filesystem-check

        This seemed to fix a load of disk issues and now everything is running fine.

        1 Reply Last reply
        2
        • nebulonN Away
          nebulonN Away
          nebulon Staff
          wrote on last edited by
          #4

          Glad it worked out in the end, and thanks for sharing the insights here!

          1 Reply Last reply
          2
          • nebulonN nebulon marked this topic as a question on
          • nebulonN nebulon has marked this topic as solved on

          • Login
          • Don't have an account? Register
          • Login or register to search.
          • First post
            Last post
          0
          • Categories
          • Recent
          • Tags
          • Popular
          • Bookmarks