MySQL not starting

[V8AAAAAA]

Hello, recently I have tried accessing my cloudron dashboard without any success, when looking at the Troubleshoot category in the documentation and I have found that my MySQL is not running. I looked at MySQL's logs and I have no clue of what's happening.

MySQL Logs :

2021-11-08T19:26:35.502258Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-0ubuntu0.20.04.1) starting as process 15687
2021-11-08T19:26:35.507041Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-11-08T19:26:35.653552Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-11-08T19:26:35.788592Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2021-11-08T19:26:35.788619Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2021-11-08T19:26:35.789156Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-11-08T19:26:35.789180Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-11-08T19:26:35.808999Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-11-08T19:26:35.809023Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-0ubuntu0.20.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2021-11-08T19:28:09.690209Z 0 [ERROR] [MY-010083] [Server] --verbose is for use with --help; did you mean --log-error-verbosity?
2021-11-08T19:28:09.690297Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-0ubuntu0.20.04.1) starting as process 15794
2021-11-08T19:28:09.695391Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-11-08T19:28:09.714256Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:10.714535Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:11.714833Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:12.715340Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:13.715753Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:14.716173Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:15.716532Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:16.717018Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:17.717557Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:18.717821Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:19.718107Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:20.718737Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:21.719389Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:22.720009Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:23.720670Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:24.721210Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:25.721877Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:26.722177Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:27.722813Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:28.723292Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:29.723943Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:30.724564Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:31.725228Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:32.725604Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:33.725965Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:34.726197Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:35.726488Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:36.726783Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:37.727242Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:38.727537Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:39.727921Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:40.728432Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:41.728939Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:42.729239Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:43.729853Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:44.730419Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:45.730867Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:46.731321Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:47.731679Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:48.732168Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:49.732797Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:50.733348Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:51.733699Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:52.734210Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:53.734778Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:54.735292Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:55.735721Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:56.736243Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:57.736750Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:58.737212Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:28:59.737571Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:00.737995Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:01.738262Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:02.738545Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:03.738880Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:04.739231Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:05.739867Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:06.740507Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:07.740946Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:08.741400Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:09.741745Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:10.742064Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:11.742362Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:12.742851Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:13.743216Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:14.743603Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:15.743900Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:16.744413Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:17.744854Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:18.745306Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:19.745622Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:20.745965Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:21.746264Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:22.746559Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:23.746796Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:24.747062Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:25.747478Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:26.747926Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:27.748263Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:28.748790Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:29.749340Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:30.749682Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:31.749939Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:32.750217Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:33.750496Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:34.750712Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:35.750957Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:36.751588Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:37.752067Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:38.752407Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:39.752805Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:40.753105Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:41.753770Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:42.754149Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:43.754465Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:44.754752Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:45.755431Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:46.755899Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:47.756471Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:48.756908Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:49.757352Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2021-11-08T19:29:49.757532Z 1 [ERROR] [MY-012592] [InnoDB] Operating system error number 11 in a file operation.
2021-11-08T19:29:49.757591Z 1 [ERROR] [MY-012596] [InnoDB] Error number 11 means 'Resource temporarily unavailable'
2021-11-08T19:29:49.757659Z 1 [ERROR] [MY-012215] [InnoDB] Cannot open datafile './ibdata1'
2021-11-08T19:29:49.757733Z 1 [ERROR] [MY-012959] [InnoDB] Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2021-11-08T19:29:49.757773Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Cannot open a file.
2021-11-08T19:29:50.258060Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-11-08T19:29:50.258374Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2021-11-08T19:29:50.259504Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-11-08T19:29:50.260662Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.27-0ubuntu0.20.04.1)  (Ubuntu).

OS Specs (in case) :

OS : Ubuntu Server 20.04
Kernel : 5.4.0-89-generic
CPU : Intel i5-8500
RAM : 16Gb

[girish]

@v8aaaaaa It seems MySQL got corrupt. Are these the logs from the MySQL on the host or on the addon inside docker?

[V8AAAAAA]

@girish Those logs are from MySQL on the host however MySQL (addon) works fine

[nebulon]

@v8aaaaaa can you check if you have disk space left on your server and also see if you have inodes left with running df -i ?

[V8AAAAAA]

@nebulon df -i generates this output :

Filesystem        Inodes  IUsed     IFree IUse% Mounted on                                                                                                              udev             1960725    609   1960116    1% /dev
tmpfs            1981019   1529   1979490    1% /run
/dev/nvme0n1p2  15597568 587662  15009906    4% /
tmpfs            1981019      4   1981015    1% /dev/smh
tmpfs            1981019      7   1981012    1% /run/lock
tmpfs            1981019     18   1981001    1% /sys/fs/cgroup
/dev/nvme0n1p1         0      0         0     - /boot/efi
/dev/md127     122085376     23 122085353    1% /run/mount/raid
tmpfs            1981019     48   1980971    1% /run/user/133                                                                                                tmpfs            1981019     38   1980981    1% /run/user/1000                                                                                                         

(I removed every snap packages mount because it's kinda useless to list them)

[nebulon]

@v8aaaaaa so that does not seem to be the issue. Can you also verify that files like ibdata1 exist under /var/lib/mysql/ as well as being owned by the mysql user?

[V8AAAAAA]

@nebulon ibdata1 does exists and belongs to the mysql user
-rw-r----- 1 mysql mysql 12582912 Nov 8 20:27 ibdata1

[nebulon]

@v8aaaaaa given that those things look ok, it looks like there was disk/filesystem corruption affecting this file. Looks like the database then has to be rebuilt, but given this affects the host database, the easiest way is likely to just restore the whole Cloudron from a backup onto a new server, also to get a fresh start on the disk/filesystem. See https://docs.cloudron.io/backups/#restore-cloudron

[V8AAAAAA]

@nebulon Is there any way to restore cloudron without reinstalling my OS? It's the only "server" I have and it's selfhosted and I have other services running on it that would be bothering to reinstall

[girish]

@v8aaaaaa Have you tried recovering mysql already? https://docs.cloudron.io/troubleshooting/#recover-mysql . In case, you are not able to recover the existing mysql, you can just start afresh. The database dump file is a file named box.mysqldump inside your latest box_xxxxx.tar.gz backup.

[V8AAAAAA]

@girish I couldn't recover MySQL so I just decided to install Debian on my server and run Cloudron inside of a Ubuntu VM so I can take snapshots incase if that happens to me again.
Thanks for the help tho