Cloudron is offline. Reconnecting - MySQL Issue

[woke]

I hadn't logged into the backend of my Cloudron in a month or so and there has been several automatic updates since then. I just tried and received the "Cloudron is offline. Reconnecting..." message constantly. I've reboot the VPS a couple times, and followed some instructions in https://cloudron.io/documentation/troubleshooting/#unreachable-dashboard with no luck yet.

Here's what I get from console after entering "systemctl status mysql.service":
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-07-03 19:24:16 UTC; 2s ago
Process: 15803 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 15780 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)

Any suggestions?

[girish]

@woke Can you try systemctl restart mysql and check the file /var/log/mysql/error.log in parallel ?

[woke]

"Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details."

2020-07-03T18:43:09.363989Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation$
2020-07-03T18:43:09.364473Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30-0ubuntu0.18.04.1) starting as process 23208 ...
2020-07-03T18:43:09.369544Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-07-03T18:43:09.369592Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-07-03T18:43:09.369601Z 0 [Note] InnoDB: Uses event mutexes
2020-07-03T18:43:09.369608Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-07-03T18:43:09.369614Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-07-03T18:43:09.369626Z 0 [Note] InnoDB: Using Linux native AIO
2020-07-03T18:43:09.369931Z 0 [Note] InnoDB: Number of pools: 1
2020-07-03T18:43:09.370205Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-07-03T18:43:09.372281Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-07-03T18:43:09.385584Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-07-03T18:43:09.392729Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-07-03T18:43:09.423982Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-07-03T18:43:09.425909Z 0 [ERROR] InnoDB: Ignoring the redo log due to missing MLOG_CHECKPOINT between the checkpoint 517725298 and the end 517725184.
2020-07-03T18:43:09.425934Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-07-03T18:43:10.026710Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-07-03T18:43:10.026809Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-07-03T18:43:10.026828Z 0 [ERROR] Failed to initialize builtin plugins.
2020-07-03T18:43:10.026840Z 0 [ERROR] Aborting

[girish]

@woke The MySQL db is corrupt. After creating a copy of the database cp -r /var/lib/mysql /var/lib/mysql.old, try the recovery steps in http://mytechmembank.blogspot.com/2017/10/recovering-crashed-mysql-instance.html ? The steps in that post are similar to https://cloudron.io/documentation/troubleshooting/#recover-mysql except that it first tries to remove the redo log files and see if that works.

[girish]

@woke Did you manage to get it working?

[woke]

@girish Unfortunately I have not... I tried the steps and continue to receive the same error. Thinking about just wiping the server at this point, not sure what else to try.

[girish]

@woke If you don't mind, I would like to debug further if possible and fix up the docs as needed. Can you enable SSH support and send a mail to support@cloudron.io with the IP address? Thanks