mysql not responding (with vaultwarden installed)

[Grienauer]

I get currently an msql error for all apps on my cloudron.

mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Mar 28 11:05:08 2023-03-28 09:05:08,737 INFO exited: mysql (exit status 2; not expected)

I already increased the memory limit. but still get the error after a reboot in the mysql logs/infos.

it is a production server.
help would be much appreciated!

[nebulon]

This was hit a few times now. For some reason some db tables got corrupted.
The fix here is to ssh into the server, then exec into the docker container and set mysql recovery to 4:

docker exec -ti mysql /bin/bash

Then edit /etc/mysql/my.cnf and add innodb_force_recovery=4 to the [mysqld] section. The mysql service will start fine then. Afterwards remove that line again.

[Grienauer]

@nebulon thx
but this does not help.

more infos:

2023-03-28T10:00:08.374631Z 8 [ERROR] [MY-012153] [InnoDB] Trying to access page number 1667445293 in space 1781, space name aea049681ed9c3bb/users, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server.
2023-03-28T10:00:08.374717Z 8 [ERROR] [MY-012154] [InnoDB] Server exits.
2023-03-28T10:00:08.374736Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7531 thread 140018124093184

[Grienauer]

@nebulon can you point me to the direction, where i can fin the docker mysql backup? thx!

[opensourced]

I had the same issue.. for me the problem was some sql query that would make the mysql server crash all the time...

2023-03-28T04:07:53.080199Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.20.04.2) starting as process 5166
2023-03-28T04:07:53.099776Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-03-28T04:07:54.219247Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-03-28T04:07:54.711046Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-03-28T04:07:54.711179Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-03-28T04:07:54.768261Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2023-03-28T04:07:54.768378Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.20.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2023-03-28T04:07:54.829889Z 8 [ERROR] [MY-012153] [InnoDB] Trying to access page number 1664378413 in space 656, space name 602a424a87c078e5/users, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server.
2023-03-28T04:07:54.830033Z 8 [ERROR] [MY-012154] [InnoDB] Server exits.
2023-03-28T04:07:54.830094Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7531 thread 140669377869568
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
04:07:54 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7fefb0000fc0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff0244c6d10 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x5563eaa8e4c1]
/usr/sbin/mysqld(print_fatal_signal(int)+0x2fb) [0x5563e992dfbb]
/usr/sbin/mysqld(my_server_abort()+0x76) [0x5563e992e106]
/usr/sbin/mysqld(my_abort()+0xe) [0x5563eaa884be]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x349) [0x5563ead18f59]
/usr/sbin/mysqld(+0x26d1ae6) [0x5563eae9cae6]
/usr/sbin/mysqld(Fil_shard::do_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*)+0xc12) [0x5563eaec3352]
/usr/sbin/mysqld(fil_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*)+0x77) [0x5563eaec33f7]
/usr/sbin/mysqld(buf_read_page_low(dberr_t*, bool, unsigned long, unsigned long, page_id_t const&, page_size_t const&, bool)+0x171) [0x5563eadc4271]
/usr/sbin/mysqld(buf_read_page(page_id_t const&, page_size_t const&)+0x4a) [0x5563eadc48ea]
/usr/sbin/mysqld(Buf_fetch<Buf_fetch_normal>::read_page()+0x38) [0x5563ead804d8]
/usr/sbin/mysqld(Buf_fetch_normal::get(buf_block_t*&)+0x657) [0x5563ead8c017]
/usr/sbin/mysqld(Buf_fetch<Buf_fetch_normal>::single_page()+0x59) [0x5563ead8c109]
/usr/sbin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, ut::Location, mtr_t*, bool)+0x233) [0x5563ead8d713]
/usr/sbin/mysqld(btr_cur_open_at_index_side(bool, dict_index_t*, unsigned long, btr_cur_t*, unsigned long, ut::Location, mtr_t*)+0x56a) [0x5563ead66c0a]
/usr/sbin/mysqld(+0x26a842b) [0x5563eae7342b]
/usr/sbin/mysqld(+0x26aa78b) [0x5563eae7578b]
/usr/sbin/mysqld(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0xeff) [0x5563eae7991f]
/usr/sbin/mysqld(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0xef2) [0x5563eae79912]
/usr/sbin/mysqld(ha_innobase::open(char const*, int, unsigned int, dd::Table const*)+0xa14) [0x5563eaae4fc4]
/usr/sbin/mysqld(handler::ha_open(TABLE*, char const*, int, int, dd::Table const*)+0x59) [0x5563e9a3f139]
/usr/sbin/mysqld(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, dd::Table const*)+0x1238) [0x5563e98d8728]
/usr/sbin/mysqld(open_table(THD*, TABLE_LIST*, Open_table_context*)+0x121f) [0x5563e972b9af]
/usr/sbin/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*)+0x3c7) [0x5563e972d2d7]
/usr/sbin/mysqld(mysql_alter_table(THD*, char const*, char const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*)+0x5fb) [0x5563e987046b]
/usr/sbin/mysqld(Sql_cmd_alter_table::execute(THD*)+0x4cc) [0x5563e9c9f5cc]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xa00) [0x5563e97c0c30]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x434) [0x5563e97c5a94]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1d17) [0x5563e97c7cd7]
/usr/sbin/mysqld(do_command(THD*)+0x207) [0x5563e97c8fd7]
/usr/sbin/mysqld(+0x1153e48) [0x5563e991ee48]
/usr/sbin/mysqld(+0x282ea4d) [0x5563eaff9a4d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7ff02a365609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7ff0295d2293]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fefb00ed880): ALTER TABLE users ADD COLUMN avatar_color VARCHAR(7)
Connection ID (thread ID): 8
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

The fix from the troubleshooting guide worked fine:

docker stop mysql
mv /home/yellowtent/platformdata/mysql /home/yellowtent/platformdata/mysql-copy
mkdir /home/yellowtent/platformdata/mysql
docker restart mysql

Then I had to restore the affected applications from the backup.

[nebulon]

@Grienauer actually this should be the same issue, have you tried my suggested solution? As @opensourced explained, one can also of course remove the table store and restore all apps.

[nebulon]

Does everyone here hitting this issue have vaultwarden installed?

[opensourced]

@nebulon yes, i have vaultwarden installed...

[rlowevoiply]

This same thing just happened to my box as well. I have vaultwarden installed. My bookstack instance crashed as well.

*Edit

I restored from a backup and I was able to get everything working - vault warden is on v1.11.0 now and i disabled updates for it after reading this thread

[nebulon]

I have revoked vaultwarden package update by now, as it is strongly related to this issue. Still not sure how a mysql client can cause such issues for the database server!

[Grienauer]

@nebulon also have vaultwarden in use

[scooke]

Crap, I literally just pushed the Update button on my bitwarden. Is it a given that something will go wrong? Should I just restore now to the previous bitwarden before things start breaking?

[Grienauer]

@scooke I think ou should restore to the version from yesterday, to prevent problems and deactivate auto update for now. just in case and follow this thread.

[fbartels]

How exactly did this happen for you? So far i have the latest vault warden installed and i did not yet notice another app not responding.

Maybe I don't have many mariadb apps?

[girish]

It's not a given that mysql crashes. The latest vaultwarden works on my cloudron as well. It seems to happen only on some servers but we got atleast 5-10 support tickets so far with the same mysql crash.

I think we have to narrow it down further. Maybe kernel/ubuntu version related as well.

[darkben]

We had the problem on a Cloudron (V 7.3.6) with Ubuntu 20.04 and the last Vaultwarden version. It was 2 days ago. After restoring a previous version of Vaultwarden the problem with MYSQL was solved. 36 hours without the problem at the moment.

[girish]

Maybe MySQL has already fixed this. When you guys have time, please update to 7.4.0 and let's check the situation there. Atleast, on our servers, the issue is not reproducible (not even with 7.3.).

[nebulon]

Since 7.4 is out and also a new vaultwarden, I have released a package for manual updates on 7.4 today. Maybe there is someone who was affected by the mysql issue, who is brave enough to test this.

If you do manually update and your Cloudron breaks again, let us know, so we fix it up again and can revoke that vaultwarden version.

The reason to pull you in for help here is, that we had no Cloudron where we were able to reproduce this for debugging.

[Neluser]

@nebulon My cloudron is broken. Applications work but the board does not start.

[Neluser]

@Neluser The mysql service for Board was getting an error.
Otherwise, all applications worked. I tried downloading some of the logs. But I did not see that somewhere it would be written about the problem.