Solved MySQL Crash After Server Suspension (Or Bare Metal Restore)

[cleveradmin]

I've run into an interesting issue here that I'm wondering if there may be value in troubleshooting. We have a Cloudron server that we used for Baserow. It's not a critical device, but I suspect it might have a few databases on it that I'd like to get access to. We did not have Cloudron setup to backup, but we did have a full image snapshot image that ran nightly on it (3rd party). Cloudron is running on a KVM VPS and this morning it was suspended due to non-payment (100% my error, forgot to update them with my new credit card). NBD, paid it and the server came back online. However, Cloudron was not starting properly and it appears to be MySQL related. So I set out to the MySQL recovery, but even with innodb_force_recovery = 4 it will not start (Performance schema tables initialization failed).

So I decided to restore to a local VM so I could at least get into Cloudron and see if the databases actually have data I care about (it's Baserow and we moved away to Nocodb, so I suspect it doesn't). The backup is from about 4 hours before the server was suspended, but I'm having the same issue. Now, this could make sense because it is a snapshot of a running system, I'm guessing the "damage" would be similar. Anyone interested in working through this? We can do a lot of messing around on the restored VM and then attempt a final run on the production. Let me know, thanks!

[girish]

@cleveradmin there are two MySQL's on Cloudron - the one used by the platform and the one by apps as an addon - https://docs.cloudron.io/troubleshooting/#mysql-host . I guess you mean the host mysql (if you didn't touch any docker).

Can you see if you can create a dump of existing stuff, nuke mysql and restore it back (the commands are in the doc link above)?

Alternately, when you say, you have no backups, I guess you mean the backups were not external? In that case, backups are probably in /var/backups . If there is a snapshot subdirectory there, you will see a box_xx.tar.gz file inside it. Extract that file and you will see a mysqldump file that you can use to restore MySQL.

[cleveradmin]

@girish Yes, I'm referring to the host MySQL. And I already followed the steps in that document, but even at innodb_force_recovery = 4, it will not start. Regarding backups, the last backup in /var/backups/snapshot is from last November.

And just to reiterate what I said in my post, this is purely educational. I've verified that the Baserow application's data has been copied elsewhere. So if it might be worth it to the community on a whole to dig into this, I'm happy to. But if you deem it a waste of time/resources, don't sweat it. Thanks.

[girish]

@cleveradmin OK, I reread your original post and understood now. You are offering your setup as a way to understand how to recover MySQL which is in some indeterminate state. In general, we don't attempt to recover databases unless it's super crucial. It's almost always just easy to recover from a backup and we also don't support setups with no backup.

(IMO, MySQL recovery is very complicated and kind of arcane. Fixing one issue leads to another and it can take days really. So, not worth the effort.)