@girish We did. We also had a RAID1 configuration which is why it probably why the server didn't go down all together. I was just surprised that the NVMe memory got corrupted within 6 months of having this server which is why I didn't suspect it until they ran the bench test.
To Follow Up with the resolution of this issue, we were told by our server provider that one of our hard drives was corrupted, which is likely the reason for this issue overall.
I don't think so. Here is the output of my df -h command:
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 3.5M 6.3G 1% /run
/dev/md0 916G 161G 709G 19% /
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/nvme1n1p1 511M 5.3M 506M 2% /boot/efi
/dev/loop0 56M 56M 0 100% /snap/core18/2409
/dev/loop1 56M 56M 0 100% /snap/core18/2538
/dev/loop3 62M 62M 0 100% /snap/core20/1587
/dev/loop2 62M 62M 0 100% /snap/core20/1581
/dev/loop4 103M 103M 0 100% /snap/lxd/23270
/dev/loop5 47M 47M 0 100% /snap/snapd/16292
/dev/loop7 103M 103M 0 100% /snap/lxd/23339
/dev/loop6 47M 47M 0 100% /snap/snapd/16010
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/19335875c15977113cc5981692090aa71e3d5482b7537515f94fe79528de55e0/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/edf1255cdb3c082d46c0c5bbf928eb123cd4f9326ccbc4d085998bcc99501128/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/efa85b7a7525ef51677c0c6dba17169539d850b9318cdc38f19e36b165ad33e4/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b36eb471c7a80ff7cb6e0f276b1da33d4984a9c537f3d5d9715e94c761711690/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/e437a2fba5c29e570696aa4056a27555022dda55a810e6864e5a8d8e211f1c0e/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/eca78709876f666d43aad8761c8e790c22ce6706d8395ed9c1ff936eca455b38/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/51759da04bf9b4915239e22080ff816c68c29b2f53d548c8dfa741a6fbefdc93/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/e727d447511b7f3add76801ac60c3602c5905204861dc6626090272cb7fb5e4d/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/874ef66c04a2bd23e9e947679ce12d04e8cc1068317a34946dd72c22f4ffbfb6/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/fbc897e82ddb4d5c8c08318cb3180ee146c6339c7ac6aa64303783ba0389450d/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/670c1ba8aba17ce920c4256c9e76551772525d927f4c845209c4604284a5fcc6/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/2507f7c381878af91e03fa2be13916cb07b6e772dfb5e5be6b585c6a9a991bea/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/31157f1b51b9fc08373818ee6e8ce8b38e7c40b1e676246631020eb08320041a/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/d890469008aeb2a6ffefb147a290b647cda03228e78b9b8cdae362c9ee6a724c/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/a7b92c3581456ff3e7d04802c57256096e66eee0a8dd0b85ab14f369ebb0331c/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/f9632cd45ff55b8d111d29f962ce5304be6ef40e5aec57662078cf2a896934a1/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/a0915918491079eff673567866563275216fc3226af270c9f33441fc123f9c0b/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/d6497febd82b403f9a539d1b65fbc9c33b9585b4ad9f0768443c990039daf39a/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b98abd517e49a91ecab399a61bb83332a92ee3b8891fc362d5bb6233e50b2789/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/6c01c2472e56ed203409480f9c57ec2fb386ed8123a40ab4c52c32268dda72b6/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/0b74d337a6116ba70f31c3d486aef9334f70e5c191f5aaf33a76e0248797e34b/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/997a4fd2da791d021cb225919638fec8b6ee8970d35ac587000d5b9c54c5b881/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/41b02e2dfa663e3263f40405b78d8c97b6ff6bdbf4e288d20da187f0f55cf8e7/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/cbfa6348d38bdadf263590bcfec1843b54e5f6beb38f788eee11879e26ee4310/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/5073c771f885f6697c3db88af4f52c356bccd30e92103c6d9f9ac45506101fed/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b2118a9e4212fa2ef88b2c0d8fcf6039894aa367ff4eadcf8d845c95a46ceb5e/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/7b0f620146620c58705692172a4a464ab98c89a6c7a9074c27f0c7be2c0837a8/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/b65e7575da9a23629988a55e82e1909791042b222e0b5ccc9d1327f619e34f00/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/336c2a420261d59619370794f2bfbbe1e176fd4d091b688ce47344114245a402/merged
overlay 916G 161G 709G 19% /var/lib/docker/overlay2/3d86ae2e8ecb26ce964dac5598be2750d30f5c64b38d43abed800b3b81570a36/merged
tmpfs 6.3G 0 6.3G 0% /run/user/0
We are having problems with the cloudron portal and related applications being unresponsive. When looking further into this, we discovered an error with the mysql service not running. We have backups, any help would be appreciated to figure out what went wrong.
Loaded: loaded (/etc/systemd/system/box.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2022-08-05 02:55:19 UTC; 76ms ago
Process: 791878 ExecStart=/home/yellowtent/box/box.js (code=exited, status=1/FAILURE)
Main PID: 791878 (code=exited, status=1/FAILURE)
After further looking into the issue, it appears mysql may be corrupted.
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2022-08-05 02:47:15 UTC; 10min ago
Process: 791643 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 791651 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 791651 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"
Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Aug 05 02:47:15 thepeoplesclub systemd[1]: Stopped MySQL Community Server.
Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Start request repeated too quickly.
Aug 05 02:47:15 thepeoplesclub systemd[1]: mysql.service: Failed with result 'exit-code'.
Aug 05 02:47:15 thepeoplesclub systemd[1]: Failed to start MySQL Community Server.
Trying to connect with the force recovery (innodb_force_recovery 1 to 6) doesn't seem to restart the mysql server.
My goal is to SSH directly into my app so that I could use that as a tunnel into the MYSQL database instance (Wordpress). The browser ssh terminal is not sufficient and does not allow for this. I tried running the following command to connect:
ssh karl@wordpress.myappdomain.com with the same password I use for the cloudron portal, but I get access denied.
I also tried adding an app password in the App Passwords section in the profile. But that seems to be for SFTP. What are the steps to remote access a cloudron app with SSH? I could not find much on this topic and feel I am missing something obvious. Thank you.
I am working to help a 503(c)(3) but they only have a debit card, not a credit card. They have a PayPal they can use. It would be really great if we can pay through an invoice on PayPal. Please contact me back via this email. Thanks.
@girish Sorry for the late and no reply. It was a new server with nothing on it, so it was just best for me to spend a couple of minutes and just reinstall everything. That worked. I know it doesn't help solve the problem, but that worked for me.
These are the logs until it gets to the loop:
Jan 17 12:58:15 Change Stream is available for your installation, give admin permissions to your database user to use this improved version.
Jan 17 12:58:20 box:tasks update 3422: {"percent":50,"message":"Stopping app services"}
Jan 17 12:58:20 box:tasks update 3422: {"percent":80,"message":"Removing collectd profile"}
Jan 17 12:58:20 box:shell removeCollectdProfile spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurecollectd.sh remove 10d59f6a-ad56-4f7a-917f-ece63fc6ac18
Jan 17 12:58:20 box:shell removeCollectdProfile (stdout): Restarting collectd
Jan 17 12:58:20 box:shell removeCollectdProfile (stdout): Removing collectd stats of 10d59f6a-ad56-4f7a-917f-ece63fc6ac18
Jan 17 12:58:20 box:tasks update 3422: {"percent":100,"message":"Done"}
Jan 17 12:58:20 box:taskworker Task took 9.975 seconds
Jan 17 12:58:20 box:tasks setCompleted - 3422: {"result":null,"error":null}
Jan 17 12:58:20 box:tasks update 3422: {"percent":100,"result":null,"error":null}
Jan 17 12:58:27 box:settings initCache: pre-load settings
Jan 17 12:58:27 box:taskworker Starting task 3423. Logs are at /home/yellowtent/platformdata/logs/10d59f6a-ad56-4f7a-917f-ece63fc6ac18/apptask.log
Jan 17 12:58:27 box:apptask run: startTask installationState: pending_start runState: running
Jan 17 12:58:27 box:tasks update 3423: {"percent":10,"message":"Starting app services"}
Jan 17 12:58:27 box:tasks update 3423: {"percent":35,"message":"Starting container"}
Jan 17 12:58:28 => Creating runtime directories
Jan 17 12:58:28 => Setting up LDAP integration
Jan 17 12:58:28 box:tasks update 3423: {"percent":60,"message":"Adding collectd profile"}
Jan 17 12:58:28 box:shell addCollectdProfile spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurecollectd.sh add 10d59f6a-ad56-4f7a-917f-ece63fc6ac18
Jan 17 12:58:28 box:shell addCollectdProfile (stdout): Restarting collectd
Jan 17 12:58:28 box:tasks update 3423: {"percent":80,"message":"Configuring reverse proxy"}
Jan 17 12:58:28 box:reverseproxy ensureCertificate: test2.nlavideo.com certificate already exists at /home/yellowtent/platformdata/nginx/cert/_.nlavideo.com.key
Jan 17 12:58:28 box:reverseproxy expiryDate: /home/yellowtent/platformdata/nginx/cert/_.nlavideo.com.cert notAfter=Feb 21 23:13:22 2022 GMT daysLeft=35.21867809027778
Jan 17 12:58:28 => Existing installation detected
Jan 17 12:58:28 box:reverseproxy providerMatchesSync: /home/yellowtent/platformdata/nginx/cert/_.nlavideo.com.cert subject=CN = *.nlavideo.com domain=*.nlavideo.com issuer=C = US, O = Let's Encrypt, CN = R3 wildcard=true/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Jan 17 12:58:28 MongoDB shell version v4.4.11
Jan 17 12:58:28 box:reverseproxy writeAppNginxConfig: writing config for "test2.nlavideo.com" to /home/yellowtent/platformdata/nginx/applications/10d59f6a-ad56-4f7a-917f-ece63fc6ac18.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"test2.nlavideo.com","hasIPv6":true,"ip":"172.18.19.66","port":3000,"endpoint":"app","certFilePath":"/home/yellowtent/platformdata/nginx/cert/_.nlavideo.com.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/_.nlavideo.com.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"id":"10d59f6a-ad56-4f7a-917f-ece63fc6ac18","location":"/"},"ocsp":true}
Jan 17 12:58:28 box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Jan 17 12:58:28 box:tasks update 3423: {"percent":100,"message":"Done"}
Jan 17 12:58:28 connecting to: mongodb://mongodb:27017/269e52ac0084de44?compressors=disabled&gssapiServiceName=mongodb
Jan 17 12:58:28 box:taskworker Task took 1.358 seconds
Jan 17 12:58:28 box:tasks setCompleted - 3423: {"result":null,"error":null}
Jan 17 12:58:28 box:tasks update 3423: {"percent":100,"result":null,"error":null}
Jan 17 12:58:28 Implicit session: session { "id" : UUID("6e558527-7ca5-49e1-9463-3da63fa2fbb4") }
Jan 17 12:58:28 MongoDB server version: 4.2.12
Jan 17 12:58:28 WARNING: shell and server versions do not match
Jan 17 12:58:28 WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })
Jan 17 12:58:28 MongoDB shell version v4.4.11
Jan 17 12:58:28 connecting to: mongodb://mongodb:27017/269e52ac0084de44?compressors=disabled&gssapiServiceName=mongodb
Jan 17 12:58:28 Implicit session: session { "id" : UUID("11750970-8bb8-4321-b73e-3b9133a30d3f") }
Jan 17 12:58:28 MongoDB server version: 4.2.12
Jan 17 12:58:28 WARNING: shell and server versions do not match
Jan 17 12:58:28 WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })
Jan 17 12:58:28 MongoDB shell version v4.4.11
I get the following errors now that I found the logs messages that just keep looping. I guess there is a mongodb vs mongo server mismatch? How would I fix that?
Jan 17 12:55:53 MongoDB shell version v4.4.11
Jan 17 12:55:53 connecting to: mongodb://mongodb:27017/269e52ac0084de44?compressors=disabled&gssapiServiceName=mongodb
Jan 17 12:55:53 Implicit session: session { "id" : UUID("b12deb71-1de3-4375-9330-41dfddfc9382") }
Jan 17 12:55:53 MongoDB server version: 4.2.12
Jan 17 12:55:53 WARNING: shell and server versions do not match
Jan 17 12:55:53 WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })
I have been installing various applications on the server and I have been mostly happy with cloudron. However, the one thing that I need is a chat, which is what I like about Rocket.Chat. It either is stuck in this long state of "Starting" or the app eventually shows "app not responsive". How could I debug the issue? Can I remove old docker instances of things on cloudron manually through the server?