Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved MySQL not starting

    Support
    3
    11
    203
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V8AAAAAA
      V8AAAAAA last edited by

      Hello, recently I have tried accessing my cloudron dashboard without any success, when looking at the Troubleshoot category in the documentation and I have found that my MySQL is not running. I looked at MySQL's logs and I have no clue of what's happening.

      MySQL Logs :

      2021-11-08T19:26:35.502258Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-0ubuntu0.20.04.1) starting as process 15687
      2021-11-08T19:26:35.507041Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
      2021-11-08T19:26:35.653552Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
      2021-11-08T19:26:35.788592Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
      2021-11-08T19:26:35.788619Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
      2021-11-08T19:26:35.789156Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
      2021-11-08T19:26:35.789180Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
      2021-11-08T19:26:35.808999Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
      2021-11-08T19:26:35.809023Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-0ubuntu0.20.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
      2021-11-08T19:28:09.690209Z 0 [ERROR] [MY-010083] [Server] --verbose is for use with --help; did you mean --log-error-verbosity?
      2021-11-08T19:28:09.690297Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-0ubuntu0.20.04.1) starting as process 15794
      2021-11-08T19:28:09.695391Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
      2021-11-08T19:28:09.714256Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:10.714535Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:11.714833Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:12.715340Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:13.715753Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:14.716173Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:15.716532Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:16.717018Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:17.717557Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:18.717821Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:19.718107Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:20.718737Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:21.719389Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:22.720009Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:23.720670Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:24.721210Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:25.721877Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:26.722177Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:27.722813Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:28.723292Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:29.723943Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:30.724564Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:31.725228Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:32.725604Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:33.725965Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:34.726197Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:35.726488Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:36.726783Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:37.727242Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:38.727537Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:39.727921Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:40.728432Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:41.728939Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:42.729239Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:43.729853Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:44.730419Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:45.730867Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:46.731321Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:47.731679Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:48.732168Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:49.732797Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:50.733348Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:51.733699Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:52.734210Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:53.734778Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:54.735292Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:55.735721Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:56.736243Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:57.736750Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:58.737212Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:28:59.737571Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:00.737995Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:01.738262Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:02.738545Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:03.738880Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:04.739231Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:05.739867Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:06.740507Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:07.740946Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:08.741400Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:09.741745Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:10.742064Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:11.742362Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:12.742851Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:13.743216Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:14.743603Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:15.743900Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:16.744413Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:17.744854Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:18.745306Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:19.745622Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:20.745965Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:21.746264Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:22.746559Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:23.746796Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:24.747062Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:25.747478Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:26.747926Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:27.748263Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:28.748790Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:29.749340Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:30.749682Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:31.749939Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:32.750217Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:33.750496Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:34.750712Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:35.750957Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:36.751588Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:37.752067Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:38.752407Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:39.752805Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:40.753105Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:41.753770Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:42.754149Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:43.754465Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:44.754752Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:45.755431Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:46.755899Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:47.756471Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:48.756908Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:49.757352Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
      2021-11-08T19:29:49.757532Z 1 [ERROR] [MY-012592] [InnoDB] Operating system error number 11 in a file operation.
      2021-11-08T19:29:49.757591Z 1 [ERROR] [MY-012596] [InnoDB] Error number 11 means 'Resource temporarily unavailable'
      2021-11-08T19:29:49.757659Z 1 [ERROR] [MY-012215] [InnoDB] Cannot open datafile './ibdata1'
      2021-11-08T19:29:49.757733Z 1 [ERROR] [MY-012959] [InnoDB] Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
      2021-11-08T19:29:49.757773Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Cannot open a file.
      2021-11-08T19:29:50.258060Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
      2021-11-08T19:29:50.258374Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
      2021-11-08T19:29:50.259504Z 0 [ERROR] [MY-010119] [Server] Aborting
      2021-11-08T19:29:50.260662Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.27-0ubuntu0.20.04.1)  (Ubuntu).
      

      OS Specs (in case) :

      OS : Ubuntu Server 20.04
      Kernel : 5.4.0-89-generic
      CPU : Intel i5-8500
      RAM : 16Gb
      
      girish 1 Reply Last reply Reply Quote 0
      • girish
        girish Staff @V8AAAAAA last edited by

        @v8aaaaaa It seems MySQL got corrupt. Are these the logs from the MySQL on the host or on the addon inside docker?

        V8AAAAAA 1 Reply Last reply Reply Quote 0
        • V8AAAAAA
          V8AAAAAA @girish last edited by

          @girish Those logs are from MySQL on the host however MySQL (addon) works fine

          nebulon 1 Reply Last reply Reply Quote 0
          • nebulon
            nebulon Staff @V8AAAAAA last edited by

            @v8aaaaaa can you check if you have disk space left on your server and also see if you have inodes left with running df -i ?

            V8AAAAAA 1 Reply Last reply Reply Quote 0
            • V8AAAAAA
              V8AAAAAA @nebulon last edited by V8AAAAAA

              @nebulon df -i generates this output :

              Filesystem        Inodes  IUsed     IFree IUse% Mounted on                                                                                                              udev             1960725    609   1960116    1% /dev
              tmpfs            1981019   1529   1979490    1% /run
              /dev/nvme0n1p2  15597568 587662  15009906    4% /
              tmpfs            1981019      4   1981015    1% /dev/smh
              tmpfs            1981019      7   1981012    1% /run/lock
              tmpfs            1981019     18   1981001    1% /sys/fs/cgroup
              /dev/nvme0n1p1         0      0         0     - /boot/efi
              /dev/md127     122085376     23 122085353    1% /run/mount/raid
              tmpfs            1981019     48   1980971    1% /run/user/133                                                                                                tmpfs            1981019     38   1980981    1% /run/user/1000                                                                                                         
              

              (I removed every snap packages mount because it's kinda useless to list them)

              nebulon 1 Reply Last reply Reply Quote 0
              • nebulon
                nebulon Staff @V8AAAAAA last edited by

                @v8aaaaaa so that does not seem to be the issue. Can you also verify that files like ibdata1 exist under /var/lib/mysql/ as well as being owned by the mysql user?

                V8AAAAAA 1 Reply Last reply Reply Quote 0
                • V8AAAAAA
                  V8AAAAAA @nebulon last edited by

                  @nebulon ibdata1 does exists and belongs to the mysql user
                  -rw-r----- 1 mysql mysql 12582912 Nov 8 20:27 ibdata1

                  nebulon 1 Reply Last reply Reply Quote 0
                  • nebulon
                    nebulon Staff @V8AAAAAA last edited by

                    @v8aaaaaa given that those things look ok, it looks like there was disk/filesystem corruption affecting this file. Looks like the database then has to be rebuilt, but given this affects the host database, the easiest way is likely to just restore the whole Cloudron from a backup onto a new server, also to get a fresh start on the disk/filesystem. See https://docs.cloudron.io/backups/#restore-cloudron

                    V8AAAAAA 1 Reply Last reply Reply Quote 0
                    • V8AAAAAA
                      V8AAAAAA @nebulon last edited by

                      @nebulon Is there any way to restore cloudron without reinstalling my OS? It's the only "server" I have and it's selfhosted and I have other services running on it that would be bothering to reinstall

                      girish 1 Reply Last reply Reply Quote 1
                      • girish
                        girish Staff @V8AAAAAA last edited by girish

                        @v8aaaaaa Have you tried recovering mysql already? https://docs.cloudron.io/troubleshooting/#recover-mysql . In case, you are not able to recover the existing mysql, you can just start afresh. The database dump file is a file named box.mysqldump inside your latest box_xxxxx.tar.gz backup.

                        V8AAAAAA 1 Reply Last reply Reply Quote 0
                        • V8AAAAAA
                          V8AAAAAA @girish last edited by

                          @girish I couldn't recover MySQL so I just decided to install Debian on my server and run Cloudron inside of a Ubuntu VM so I can take snapshots incase if that happens to me again.
                          Thanks for the help tho

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Powered by NodeBB