Issue with Cloudron Unresponsive
-
We are having problems with the cloudron portal and related applications being unresponsive. When looking further into this, we discovered an error with the mysql service not running. We have backups, any help would be appreciated to figure out what went wrong.
Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Fri 2022-08-05 02:55:19 UTC; 76ms ago Process: 791878 ExecStart=/home/yellowtent/box/box.js (code=exited, status=1/FAILURE) Main PID: 791878 (code=exited, status=1/FAILURE)
After further looking into the issue, it appears mysql may be corrupted.
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2022-08-05 02:47:15 UTC; 10min ago Process: 791643 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Process: 791651 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE) Main PID: 791651 (code=exited, status=1/FAILURE) Status: "Server shutdown complete" Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5. Aug 05 02:47:15 thepeoplesclub systemd[1]: Stopped MySQL Community Server. Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Start request repeated too quickly. Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Failed with result 'exit-code'. Aug 05 02:47:15 thepeoplesclub systemd[1]: Failed to start MySQL Community Server.
Trying to connect with the force recovery (innodb_force_recovery 1 to 6) doesn't seem to restart the mysql server.
-
@applecrusher is the hard drive full?
-
I don't think so. Here is the output of my
df -h
command:Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 3.5M 6.3G 1% /run /dev/md0 916G 161G 709G 19% / tmpfs 32G 0 32G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/nvme1n1p1 511M 5.3M 506M 2% /boot/efi /dev/loop0 56M 56M 0 100% /snap/core18/2409 /dev/loop1 56M 56M 0 100% /snap/core18/2538 /dev/loop3 62M 62M 0 100% /snap/core20/1587 /dev/loop2 62M 62M 0 100% /snap/core20/1581 /dev/loop4 103M 103M 0 100% /snap/lxd/23270 /dev/loop5 47M 47M 0 100% /snap/snapd/16292 /dev/loop7 103M 103M 0 100% /snap/lxd/23339 /dev/loop6 47M 47M 0 100% /snap/snapd/16010 overlay 916G 161G 709G 19% /var/lib/docker/overlay2/19335875c15977113cc5981692090aa71e3d5482b7537515f94fe79528de55e0/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/edf1255cdb3c082d46c0c5bbf928eb123cd4f9326ccbc4d085998bcc99501128/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/efa85b7a7525ef51677c0c6dba17169539d850b9318cdc38f19e36b165ad33e4/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b36eb471c7a80ff7cb6e0f276b1da33d4984a9c537f3d5d9715e94c761711690/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/e437a2fba5c29e570696aa4056a27555022dda55a810e6864e5a8d8e211f1c0e/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/eca78709876f666d43aad8761c8e790c22ce6706d8395ed9c1ff936eca455b38/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/51759da04bf9b4915239e22080ff816c68c29b2f53d548c8dfa741a6fbefdc93/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/e727d447511b7f3add76801ac60c3602c5905204861dc6626090272cb7fb5e4d/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/874ef66c04a2bd23e9e947679ce12d04e8cc1068317a34946dd72c22f4ffbfb6/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/fbc897e82ddb4d5c8c08318cb3180ee146c6339c7ac6aa64303783ba0389450d/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/670c1ba8aba17ce920c4256c9e76551772525d927f4c845209c4604284a5fcc6/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/2507f7c381878af91e03fa2be13916cb07b6e772dfb5e5be6b585c6a9a991bea/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/31157f1b51b9fc08373818ee6e8ce8b38e7c40b1e676246631020eb08320041a/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/d890469008aeb2a6ffefb147a290b647cda03228e78b9b8cdae362c9ee6a724c/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/a7b92c3581456ff3e7d04802c57256096e66eee0a8dd0b85ab14f369ebb0331c/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/f9632cd45ff55b8d111d29f962ce5304be6ef40e5aec57662078cf2a896934a1/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/a0915918491079eff673567866563275216fc3226af270c9f33441fc123f9c0b/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/d6497febd82b403f9a539d1b65fbc9c33b9585b4ad9f0768443c990039daf39a/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b98abd517e49a91ecab399a61bb83332a92ee3b8891fc362d5bb6233e50b2789/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/6c01c2472e56ed203409480f9c57ec2fb386ed8123a40ab4c52c32268dda72b6/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/0b74d337a6116ba70f31c3d486aef9334f70e5c191f5aaf33a76e0248797e34b/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/997a4fd2da791d021cb225919638fec8b6ee8970d35ac587000d5b9c54c5b881/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/41b02e2dfa663e3263f40405b78d8c97b6ff6bdbf4e288d20da187f0f55cf8e7/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/cbfa6348d38bdadf263590bcfec1843b54e5f6beb38f788eee11879e26ee4310/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/5073c771f885f6697c3db88af4f52c356bccd30e92103c6d9f9ac45506101fed/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b2118a9e4212fa2ef88b2c0d8fcf6039894aa367ff4eadcf8d845c95a46ceb5e/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/7b0f620146620c58705692172a4a464ab98c89a6c7a9074c27f0c7be2c0837a8/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b65e7575da9a23629988a55e82e1909791042b222e0b5ccc9d1327f619e34f00/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/336c2a420261d59619370794f2bfbbe1e176fd4d091b688ce47344114245a402/merged overlay 916G 161G 709G 19% /var/lib/docker/overlay2/3d86ae2e8ecb26ce964dac5598be2750d30f5c64b38d43abed800b3b81570a36/merged tmpfs 6.3G 0 6.3G 0% /run/user/0
-
To Follow Up with the resolution of this issue, we were told by our server provider that one of our hard drives was corrupted, which is likely the reason for this issue overall.
-
@applecrusher Oh I hope you had backups to recover.
-
-
-
@girish We did. We also had a RAID1 configuration which is why it probably why the server didn't go down all together. I was just surprised that the NVMe memory got corrupted within 6 months of having this server which is why I didn't suspect it until they ran the bench test.