Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. MySQL Crash After Server Suspension (Or Bare Metal Restore)

MySQL Crash After Server Suspension (Or Bare Metal Restore)

Scheduled Pinned Locked Moved Solved Support
mysqltroubleshooting
4 Posts 2 Posters 697 Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      C Offline
      cleveradmin
      wrote on last edited by girish
      #1

      I've run into an interesting issue here that I'm wondering if there may be value in troubleshooting. We have a Cloudron server that we used for Baserow. It's not a critical device, but I suspect it might have a few databases on it that I'd like to get access to. We did not have Cloudron setup to backup, but we did have a full image snapshot image that ran nightly on it (3rd party). Cloudron is running on a KVM VPS and this morning it was suspended due to non-payment (100% my error, forgot to update them with my new credit card). NBD, paid it and the server came back online. However, Cloudron was not starting properly and it appears to be MySQL related. So I set out to the MySQL recovery, but even with innodb_force_recovery = 4 it will not start (Performance schema tables initialization failed).

      So I decided to restore to a local VM so I could at least get into Cloudron and see if the databases actually have data I care about (it's Baserow and we moved away to Nocodb, so I suspect it doesn't). The backup is from about 4 hours before the server was suspended, but I'm having the same issue. Now, this could make sense because it is a snapshot of a running system, I'm guessing the "damage" would be similar. Anyone interested in working through this? We can do a lot of messing around on the restored VM and then attempt a final run on the production. Let me know, thanks!

      girishG 1 Reply Last reply
      1
      • C cleveradmin

        I've run into an interesting issue here that I'm wondering if there may be value in troubleshooting. We have a Cloudron server that we used for Baserow. It's not a critical device, but I suspect it might have a few databases on it that I'd like to get access to. We did not have Cloudron setup to backup, but we did have a full image snapshot image that ran nightly on it (3rd party). Cloudron is running on a KVM VPS and this morning it was suspended due to non-payment (100% my error, forgot to update them with my new credit card). NBD, paid it and the server came back online. However, Cloudron was not starting properly and it appears to be MySQL related. So I set out to the MySQL recovery, but even with innodb_force_recovery = 4 it will not start (Performance schema tables initialization failed).

        So I decided to restore to a local VM so I could at least get into Cloudron and see if the databases actually have data I care about (it's Baserow and we moved away to Nocodb, so I suspect it doesn't). The backup is from about 4 hours before the server was suspended, but I'm having the same issue. Now, this could make sense because it is a snapshot of a running system, I'm guessing the "damage" would be similar. Anyone interested in working through this? We can do a lot of messing around on the restored VM and then attempt a final run on the production. Let me know, thanks!

        girishG Offline
        girishG Offline
        girish
        Staff
        wrote on last edited by
        #2

        @cleveradmin there are two MySQL's on Cloudron - the one used by the platform and the one by apps as an addon - https://docs.cloudron.io/troubleshooting/#mysql-host . I guess you mean the host mysql (if you didn't touch any docker).

        Can you see if you can create a dump of existing stuff, nuke mysql and restore it back (the commands are in the doc link above)?

        Alternately, when you say, you have no backups, I guess you mean the backups were not external? In that case, backups are probably in /var/backups . If there is a snapshot subdirectory there, you will see a box_xx.tar.gz file inside it. Extract that file and you will see a mysqldump file that you can use to restore MySQL.

        C 1 Reply Last reply
        0
        • girishG girish marked this topic as a question on
        • girishG girish

          @cleveradmin there are two MySQL's on Cloudron - the one used by the platform and the one by apps as an addon - https://docs.cloudron.io/troubleshooting/#mysql-host . I guess you mean the host mysql (if you didn't touch any docker).

          Can you see if you can create a dump of existing stuff, nuke mysql and restore it back (the commands are in the doc link above)?

          Alternately, when you say, you have no backups, I guess you mean the backups were not external? In that case, backups are probably in /var/backups . If there is a snapshot subdirectory there, you will see a box_xx.tar.gz file inside it. Extract that file and you will see a mysqldump file that you can use to restore MySQL.

          C Offline
          C Offline
          cleveradmin
          wrote on last edited by
          #3

          @girish Yes, I'm referring to the host MySQL. And I already followed the steps in that document, but even at innodb_force_recovery = 4, it will not start. Regarding backups, the last backup in /var/backups/snapshot is from last November.

          And just to reiterate what I said in my post, this is purely educational. I've verified that the Baserow application's data has been copied elsewhere. So if it might be worth it to the community on a whole to dig into this, I'm happy to. But if you deem it a waste of time/resources, don't sweat it. Thanks.

          girishG 1 Reply Last reply
          0
          • C cleveradmin

            @girish Yes, I'm referring to the host MySQL. And I already followed the steps in that document, but even at innodb_force_recovery = 4, it will not start. Regarding backups, the last backup in /var/backups/snapshot is from last November.

            And just to reiterate what I said in my post, this is purely educational. I've verified that the Baserow application's data has been copied elsewhere. So if it might be worth it to the community on a whole to dig into this, I'm happy to. But if you deem it a waste of time/resources, don't sweat it. Thanks.

            girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #4

            @cleveradmin OK, I reread your original post and understood now. You are offering your setup as a way to understand how to recover MySQL which is in some indeterminate state. In general, we don't attempt to recover databases unless it's super crucial. It's almost always just easy to recover from a backup and we also don't support setups with no backup.

            (IMO, MySQL recovery is very complicated and kind of arcane. Fixing one issue leads to another and it can take days really. So, not worth the effort.)

            1 Reply Last reply
            0
            • girishG girish has marked this topic as solved on
            Reply
            • Reply as topic
            Log in to reply
            • Oldest to Newest
            • Newest to Oldest
            • Most Votes


              • Login

              • Don't have an account? Register

              • Login or register to search.
              • First post
                Last post
              0
              • Categories
              • Recent
              • Tags
              • Popular
              • Bookmarks
              • Search