Solved Issue with Cloudron Unresponsive

[applecrusher]

We are having problems with the cloudron portal and related applications being unresponsive. When looking further into this, we discovered an error with the mysql service not running. We have backups, any help would be appreciated to figure out what went wrong.

Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2022-08-05 02:55:19 UTC; 76ms ago
    Process: 791878 ExecStart=/home/yellowtent/box/box.js (code=exited, status=1/FAILURE)
   Main PID: 791878 (code=exited, status=1/FAILURE)

After further looking into the issue, it appears mysql may be corrupted.

Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-08-05 02:47:15 UTC; 10min ago
    Process: 791643 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Process: 791651 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
   Main PID: 791651 (code=exited, status=1/FAILURE)
     Status: "Server shutdown complete"

Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Aug 05 02:47:15 thepeoplesclub systemd[1]: Stopped MySQL Community Server.
Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Start request repeated too quickly.
Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 05 02:47:15 thepeoplesclub systemd[1]: Failed to start MySQL Community Server.

Trying to connect with the force recovery (innodb_force_recovery 1 to 6) doesn't seem to restart the mysql server.

[murgero]

@applecrusher is the hard drive full?

[applecrusher]

@murgero

I don't think so. Here is the output of my df -h command:

Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  3.5M  6.3G   1% /run
/dev/md0        916G  161G  709G  19% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/nvme1n1p1  511M  5.3M  506M   2% /boot/efi
/dev/loop0       56M   56M     0 100% /snap/core18/2409
/dev/loop1       56M   56M     0 100% /snap/core18/2538
/dev/loop3       62M   62M     0 100% /snap/core20/1587
/dev/loop2       62M   62M     0 100% /snap/core20/1581
/dev/loop4      103M  103M     0 100% /snap/lxd/23270
/dev/loop5       47M   47M     0 100% /snap/snapd/16292
/dev/loop7      103M  103M     0 100% /snap/lxd/23339
/dev/loop6       47M   47M     0 100% /snap/snapd/16010
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/19335875c15977113cc5981692090aa71e3d5482b7537515f94fe79528de55e0/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/edf1255cdb3c082d46c0c5bbf928eb123cd4f9326ccbc4d085998bcc99501128/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/efa85b7a7525ef51677c0c6dba17169539d850b9318cdc38f19e36b165ad33e4/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/b36eb471c7a80ff7cb6e0f276b1da33d4984a9c537f3d5d9715e94c761711690/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/e437a2fba5c29e570696aa4056a27555022dda55a810e6864e5a8d8e211f1c0e/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/eca78709876f666d43aad8761c8e790c22ce6706d8395ed9c1ff936eca455b38/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/51759da04bf9b4915239e22080ff816c68c29b2f53d548c8dfa741a6fbefdc93/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/e727d447511b7f3add76801ac60c3602c5905204861dc6626090272cb7fb5e4d/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/874ef66c04a2bd23e9e947679ce12d04e8cc1068317a34946dd72c22f4ffbfb6/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/fbc897e82ddb4d5c8c08318cb3180ee146c6339c7ac6aa64303783ba0389450d/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/670c1ba8aba17ce920c4256c9e76551772525d927f4c845209c4604284a5fcc6/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/2507f7c381878af91e03fa2be13916cb07b6e772dfb5e5be6b585c6a9a991bea/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/31157f1b51b9fc08373818ee6e8ce8b38e7c40b1e676246631020eb08320041a/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/d890469008aeb2a6ffefb147a290b647cda03228e78b9b8cdae362c9ee6a724c/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/a7b92c3581456ff3e7d04802c57256096e66eee0a8dd0b85ab14f369ebb0331c/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/f9632cd45ff55b8d111d29f962ce5304be6ef40e5aec57662078cf2a896934a1/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/a0915918491079eff673567866563275216fc3226af270c9f33441fc123f9c0b/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/d6497febd82b403f9a539d1b65fbc9c33b9585b4ad9f0768443c990039daf39a/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/b98abd517e49a91ecab399a61bb83332a92ee3b8891fc362d5bb6233e50b2789/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/6c01c2472e56ed203409480f9c57ec2fb386ed8123a40ab4c52c32268dda72b6/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/0b74d337a6116ba70f31c3d486aef9334f70e5c191f5aaf33a76e0248797e34b/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/997a4fd2da791d021cb225919638fec8b6ee8970d35ac587000d5b9c54c5b881/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/41b02e2dfa663e3263f40405b78d8c97b6ff6bdbf4e288d20da187f0f55cf8e7/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/cbfa6348d38bdadf263590bcfec1843b54e5f6beb38f788eee11879e26ee4310/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/5073c771f885f6697c3db88af4f52c356bccd30e92103c6d9f9ac45506101fed/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/b2118a9e4212fa2ef88b2c0d8fcf6039894aa367ff4eadcf8d845c95a46ceb5e/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/7b0f620146620c58705692172a4a464ab98c89a6c7a9074c27f0c7be2c0837a8/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/b65e7575da9a23629988a55e82e1909791042b222e0b5ccc9d1327f619e34f00/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/336c2a420261d59619370794f2bfbbe1e176fd4d091b688ce47344114245a402/merged
overlay         916G  161G  709G  19% /var/lib/docker/overlay2/3d86ae2e8ecb26ce964dac5598be2750d30f5c64b38d43abed800b3b81570a36/merged
tmpfs           6.3G     0  6.3G   0% /run/user/0

[applecrusher]

To Follow Up with the resolution of this issue, we were told by our server provider that one of our hard drives was corrupted, which is likely the reason for this issue overall.

[girish]

@applecrusher Oh I hope you had backups to recover.

[applecrusher]

@girish We did. We also had a RAID1 configuration which is why it probably why the server didn't go down all together. I was just surprised that the NVMe memory got corrupted within 6 months of having this server which is why I didn't suspect it until they ran the bench test.