mysql addon fails to start

[dfldadm]

Hi,

I'm running a cloudron 6.2.4 in which the disk went full (too many images of different versions in docker).

Now the mysql addon (not the ubuntu mysql) doesn't come up again:

2021-03-23T11:43:34.632721Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.23-0ubuntu0.20.04.1) starting as process 11
2021-03-23T11:43:34.646837Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-03-23T11:43:34.952073Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-03-23T11:43:34.959894Z 1 [ERROR] [MY-011092] [Server] Upgrading the data dictionary from dictionary version '0' is not supported.
2021-03-23T11:43:34.960213Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2021-03-23T11:43:34.960380Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-03-23T11:43:35.479431Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.23-0ubuntu0.20.04.1)  (Ubuntu).

Some people report this may happen if mysql is upgraded skipping versions but I cannot know whether this is the case/how cloudron handles this.

Is there a way to recover from this? I cannot use my apps for this reason, not even stopping/starting or recovery is possible.

Update: This is amazing. I encounter the exact same error on another instance in another hosting environment.

Trying to fix this desperately I tried cloudron/mysql 2.3.2, 3.0.0, 3.0.3 and 3.0.6 (3.0.4 seems to be default) but nothing helped.

[girish]

@dfldadm Can you send us a mail to support@cloudron.io instead? It's hard to diagnose the problem from just the logs above.

Also, it doesn't come up after freeing up some space is it? If you free up some space, all you have to do is systemctl restart box and it will re-initialize the addon as needed.

[girish]

The root cause was that both servers ran out of disk space. So, the export of db from old version failed and thus couldn't import to new version.

[dfldadm]

For the record, since the communication on this went private.

The issue was solved by manual intervention.
I have two takeaway messages:

1

If mysql doesn't come back up, it is possible to
remove /home/yellowtent/platformdata/mysql (this is where the mysql addon stores it's data) and restart the box code using systemctl restart box.
(Do this from outside the mysql container) This is apparently non-destructive.

This fixed the first machine which hung up as it ran out of disk space (and this didn't change as space was freed up again)

2

• Do not upgrade box or any app while there's not plenty (5 Gig perhaps) of free space. It seems as if cloudron apps can get stuck in a bad state when space runs out while doing system things [DB migration, backups, updates] (I hope this is correct)
• Monitor space closely, it's cheaper to buy more than to end up fixing a stuck setup...

[nebulon]

@dfldadm action 1) is only non-destricutive, if you can restore the apps afterwards! Anyways always ensure to have backups. We have had encountered various situations where a server runs out of disk space and to free up space, people deleted on-disk backups. So at least download those to a safe place beforehand.

[girish]

Generally, I am wary of putting this information for general use (and even in docs) because many things can go wrong . So, please, if you care about the data, don't hesitate to contact us on support@ . We help regardless of the subscription plan.