mysql not responding (with vaultwarden installed)
-
I get currently an msql error for all apps on my cloudron.
mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) Mar 28 11:05:08 2023-03-28 09:05:08,737 INFO exited: mysql (exit status 2; not expected)
I already increased the memory limit. but still get the error after a reboot in the mysql logs/infos.
it is a production server.
help would be much appreciated! -
This was hit a few times now. For some reason some db tables got corrupted.
The fix here is to ssh into the server, then exec into the docker container and set mysql recovery to 4:docker exec -ti mysql /bin/bash
Then edit
/etc/mysql/my.cnf
and addinnodb_force_recovery=4
to the[mysqld]
section. The mysql service will start fine then. Afterwards remove that line again. -
-
@nebulon thx
but this does not help.more infos:
2023-03-28T10:00:08.374631Z 8 [ERROR] [MY-012153] [InnoDB] Trying to access page number 1667445293 in space 1781, space name aea049681ed9c3bb/users, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server. 2023-03-28T10:00:08.374717Z 8 [ERROR] [MY-012154] [InnoDB] Server exits. 2023-03-28T10:00:08.374736Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7531 thread 140018124093184
-
I had the same issue.. for me the problem was some sql query that would make the mysql server crash all the time...
2023-03-28T04:07:53.080199Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.20.04.2) starting as process 5166 2023-03-28T04:07:53.099776Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2023-03-28T04:07:54.219247Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2023-03-28T04:07:54.711046Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2023-03-28T04:07:54.711179Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. 2023-03-28T04:07:54.768261Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock 2023-03-28T04:07:54.768378Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.20.04.2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu). 2023-03-28T04:07:54.829889Z 8 [ERROR] [MY-012153] [InnoDB] Trying to access page number 1664378413 in space 656, space name 602a424a87c078e5/users, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server. 2023-03-28T04:07:54.830033Z 8 [ERROR] [MY-012154] [InnoDB] Server exits. 2023-03-28T04:07:54.830094Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7531 thread 140669377869568 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 04:07:54 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x7fefb0000fc0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7ff0244c6d10 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x5563eaa8e4c1] /usr/sbin/mysqld(print_fatal_signal(int)+0x2fb) [0x5563e992dfbb] /usr/sbin/mysqld(my_server_abort()+0x76) [0x5563e992e106] /usr/sbin/mysqld(my_abort()+0xe) [0x5563eaa884be] /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x349) [0x5563ead18f59] /usr/sbin/mysqld(+0x26d1ae6) [0x5563eae9cae6] /usr/sbin/mysqld(Fil_shard::do_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*)+0xc12) [0x5563eaec3352] /usr/sbin/mysqld(fil_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*)+0x77) [0x5563eaec33f7] /usr/sbin/mysqld(buf_read_page_low(dberr_t*, bool, unsigned long, unsigned long, page_id_t const&, page_size_t const&, bool)+0x171) [0x5563eadc4271] /usr/sbin/mysqld(buf_read_page(page_id_t const&, page_size_t const&)+0x4a) [0x5563eadc48ea] /usr/sbin/mysqld(Buf_fetch<Buf_fetch_normal>::read_page()+0x38) [0x5563ead804d8] /usr/sbin/mysqld(Buf_fetch_normal::get(buf_block_t*&)+0x657) [0x5563ead8c017] /usr/sbin/mysqld(Buf_fetch<Buf_fetch_normal>::single_page()+0x59) [0x5563ead8c109] /usr/sbin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, ut::Location, mtr_t*, bool)+0x233) [0x5563ead8d713] /usr/sbin/mysqld(btr_cur_open_at_index_side(bool, dict_index_t*, unsigned long, btr_cur_t*, unsigned long, ut::Location, mtr_t*)+0x56a) [0x5563ead66c0a] /usr/sbin/mysqld(+0x26a842b) [0x5563eae7342b] /usr/sbin/mysqld(+0x26aa78b) [0x5563eae7578b] /usr/sbin/mysqld(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0xeff) [0x5563eae7991f] /usr/sbin/mysqld(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0xef2) [0x5563eae79912] /usr/sbin/mysqld(ha_innobase::open(char const*, int, unsigned int, dd::Table const*)+0xa14) [0x5563eaae4fc4] /usr/sbin/mysqld(handler::ha_open(TABLE*, char const*, int, int, dd::Table const*)+0x59) [0x5563e9a3f139] /usr/sbin/mysqld(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, dd::Table const*)+0x1238) [0x5563e98d8728] /usr/sbin/mysqld(open_table(THD*, TABLE_LIST*, Open_table_context*)+0x121f) [0x5563e972b9af] /usr/sbin/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*)+0x3c7) [0x5563e972d2d7] /usr/sbin/mysqld(mysql_alter_table(THD*, char const*, char const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*)+0x5fb) [0x5563e987046b] /usr/sbin/mysqld(Sql_cmd_alter_table::execute(THD*)+0x4cc) [0x5563e9c9f5cc] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xa00) [0x5563e97c0c30] /usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x434) [0x5563e97c5a94] /usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1d17) [0x5563e97c7cd7] /usr/sbin/mysqld(do_command(THD*)+0x207) [0x5563e97c8fd7] /usr/sbin/mysqld(+0x1153e48) [0x5563e991ee48] /usr/sbin/mysqld(+0x282ea4d) [0x5563eaff9a4d] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7ff02a365609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7ff0295d2293] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7fefb00ed880): ALTER TABLE users ADD COLUMN avatar_color VARCHAR(7) Connection ID (thread ID): 8 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
The fix from the troubleshooting guide worked fine:
docker stop mysql mv /home/yellowtent/platformdata/mysql /home/yellowtent/platformdata/mysql-copy mkdir /home/yellowtent/platformdata/mysql docker restart mysql
Then I had to restore the affected applications from the backup.
-
@Grienauer actually this should be the same issue, have you tried my suggested solution? As @opensourced explained, one can also of course remove the table store and restore all apps.
-
This same thing just happened to my box as well. I have vaultwarden installed. My bookstack instance crashed as well.
*Edit
I restored from a backup and I was able to get everything working - vault warden is on v1.11.0 now and i disabled updates for it after reading this thread
-
It's not a given that mysql crashes. The latest vaultwarden works on my cloudron as well. It seems to happen only on some servers but we got atleast 5-10 support tickets so far with the same mysql crash.
I think we have to narrow it down further. Maybe kernel/ubuntu version related as well.
-
Since 7.4 is out and also a new vaultwarden, I have released a package for manual updates on 7.4 today. Maybe there is someone who was affected by the mysql issue, who is brave enough to test this.
If you do manually update and your Cloudron breaks again, let us know, so we fix it up again and can revoke that vaultwarden version.
The reason to pull you in for help here is, that we had no Cloudron where we were able to reproduce this for debugging.