Vaultwarden fails to start after update – DB migration error (SSO)
-
J james has marked this topic as solved
-
Hello @vaultwarden users
Since users have voiced their concerns about manually editing the database I have created a bash script that does it for you.
Please follow these steps if you have this issue:- create an app backup of your @vaultwarden Cloudron app
- put the erroring @vaultwarden app in recovery mode
- open the File Manager of your @vaultwarden app and create a
fix_db.shfile - copy and paste the following script into the just created
fix_db.shfile:#!/bin/bash echo "=> Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci" CURRENT_DEFAULT_CHARACTER_SET_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) CURRENT_DEFAULT_COLLATION_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) FIX_NEEDED=0 if [ "${CURRENT_DEFAULT_CHARACTER_SET_NAME}" != "utf8mb4" ] || [ "${CURRENT_DEFAULT_COLLATION_NAME}" != "utf8mb4_unicode_ci" ]; then echo "=> DEFAULT_CHARACTER_SET_NAME or DEFAULT_COLLATION_NAME is not set to utf8mb4 or utf8mb4_unicode_ci." echo "==> Current values are: DEFAULT_CHARACTER_SET_NAME=${CURRENT_DEFAULT_CHARACTER_SET_NAME}, DEFAULT_COLLATION_NAME=${CURRENT_DEFAULT_COLLATION_NAME}" FIX_NEEDED=1 else echo "=> DEFAULT_CHARACTER_SET_NAME is ${CURRENT_DEFAULT_CHARACTER_SET_NAME} and DEFAULT_COLLATION_NAME is ${CURRENT_DEFAULT_COLLATION_NAME}" echo "=> Nothing to do - quitting." FIX_NEEDED=0 exit 0 fi if [ ${FIX_NEEDED} -eq 1 ]; then read -p "Have you created a backup of your Cloudron Vaultwarden app? (y/N): " yn case $yn in [Yy]* ) echo "=> Setting database character set and collation to utf8mb4 and utf8mb4_unicode_ci." mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "ALTER DATABASE \`${CLOUDRON_MYSQL_DATABASE}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" 2>/dev/null echo "=> Converting all tables to character set utf8mb4 and collation utf8mb4_unicode_ci." mysql --silent --skip-column-names \ --user="${CLOUDRON_MYSQL_USERNAME}" \ --password="${CLOUDRON_MYSQL_PASSWORD}" \ --host="${CLOUDRON_MYSQL_HOST}" \ "${CLOUDRON_MYSQL_DATABASE}" \ -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,'\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS CharSetConvert FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='${CLOUDRON_MYSQL_DATABASE}' AND TABLE_TYPE=\"BASE TABLE\";" 2>/dev/null | while read -r sql_command; do echo "==> Executing: ${sql_command}" mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SET foreign_key_checks=0; ${sql_command} SET foreign_key_checks=1;" 2>/dev/null done ;; [Nn]* ) echo "=> Please create a backup of your database before proceeding with the fix. Exiting." exit 1 ;; * ) echo "=> Invalid response. Please answer with y or n. Exiting." exit 1 ;; esac fi - open the Web Terminal of your @vaultwarden Cloudron app
- execute the following command:
bash /app/data/fix_db.sh - Execute the following command to start @vaultwarden:
/app/pkg/start.sh - Validate if your @vaultwarden is now working correctly
- If validated working, you can delete the
fix_db.shfile and disable the recovery mode of your @vaultwarden Cloudron app
If you run into any issues, copy the output of the terminal, post it here and restore your app from the backup created.
@james it doesnt work for me, here is the output
root@b2a9826b-7631-435b-91a0-d0a505476ee9:/app/code# bash /app/data/fix_db.sh /app/data/fix_db.sh: line 2: $'\r': command not found => Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci /app/data/fix_db.sh: line 4: $'\r': command not found /app/data/fix_db.sh: line 8: $'\r': command not found /app/data/fix_db.sh: line 22: syntax error near unexpected token `$'in\r'' 'app/data/fix_db.sh: line 22: ` case $yn in root@b2a9826b-7631-435b-91a0-d0a505476ee9:/app/code# /app/pkg/start.sh => Exporting env vars expected by Vaultwarden => Starting Bitwarden /--------------------------------------------------------------------\ | Starting Vaultwarden | |--------------------------------------------------------------------| | This is an *unofficial* Bitwarden implementation, DO NOT use the | | official channels to report bugs/features, regardless of client. | | Send usage/configuration questions or feature requests to: | | https://github.com/dani-garcia/vaultwarden/discussions or | | https://vaultwarden.discourse.group/ | | Report suspected bugs/issues in the software itself at: | | https://github.com/dani-garcia/vaultwarden/issues/new | \--------------------------------------------------------------------/ [INFO] Using saved config from `/app/data/config.json` for configuration. [2026-01-03 17:24:33.038][panic][ERROR] thread 'main' panicked at 'Error running migrations: QueryError(DieselMigrationName { name: "2024-03-06-170000_add_sso_users", version: MigrationVersion("20240306170000") }, DatabaseError(Unknown, "Referencing column 'user_uuid' and referenced column 'uuid' in foreign key constraint 'sso_users_ibfk_1' are incompatible."))': src/db/mod.rs:505 0: vaultwarden::init_logging::{{closure}} 1: std::panicking::panic_with_hook 2: std::panicking::panic_handler::{{closure}} 3: std::sys::backtrace::__rust_end_short_backtrace 4: __rustc::rust_begin_unwind 5: core::panicking::panic_fmt 6: core::result::unwrap_failed 7: vaultwarden::db::DbPool::from_config 8: vaultwarden::main::{{closure}} 9: vaultwarden::main 10: std::sys::backtrace::__rust_begin_short_backtrace 11: main 12: <unknown> 13: __libc_start_main 14: _start root@b2a9826b-7631-435b-91a0-d0a505476ee9:/app/code#PS: i restored to 1.22.2 to get it working for now.
-
Hello @factord
This looks like a copy-paste issue from Windows.
\rin line 2 is a blank line, so a line break.
When you copy and pasted, it pasted the line breaks windows style.Please describe how you have copy and pasted the code into the
/app/data/fix_db.shfile. -
I have the same issue as @factord I am on windows 11 and created the file through the file manager, pasted the contents copied from the post into the file using the built in file editor that is a part of the file manager, saved the file, opened the web terminal and ran the
bash /app/data/fix_db.shcommand. -
I have run the command
sed -i 's/\r$//g' /app/data/fix_db.shto fix the new line characters and then runbash /app/data/fix_db.shagain and it ran as expected. Disabled recover mode and confirmed things are back to working as expected. -
So everytime I upgrade from Vaultwarden 1.34.3 to 1.35.0 the instance becomes unresponsive, saying connection failure in the terminal and nothing really in the logs apart from connection failure again.
Jan 06 12:13:28 0: vaultwarden::init_logging::{{closure}} Jan 06 12:13:28 10: std::sys::backtrace::__rust_begin_short_backtrace Jan 06 12:13:28 11: main Jan 06 12:13:28 12: <unknown> Jan 06 12:13:28 13: __libc_start_main Jan 06 12:13:28 14: _start Jan 06 12:13:28 1: std::panicking::panic_with_hook Jan 06 12:13:28 2: std::panicking::panic_handler::{{closure}} Jan 06 12:13:28 3: std::sys::backtrace::__rust_end_short_backtrace Jan 06 12:13:28 4: __rustc::rust_begin_unwind Jan 06 12:13:28 5: core::panicking::panic_fmt Jan 06 12:13:28 6: core::result::unwrap_failed Jan 06 12:13:28 7: vaultwarden::db::DbPool::from_config Jan 06 12:13:28 8: vaultwarden::main::{{closure}} Jan 06 12:13:28 9: vaultwarden::main Jan 06 12:13:28 Please generate a secure Argon2 PHC string by using `vaultwarden hash` or `argon2`. Jan 06 12:13:28 See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token Jan 06 12:13:28 [2026-01-06 12:13:28.418][panic][ERROR] thread 'main' panicked at 'Error running migrations: QueryError(DieselMigrationName { name: "2024-03-06-170000_add_sso_users", version: MigrationVersion("20240306170000") }, DatabaseError(Unknown, "Referencing column 'user_uuid' and referenced column 'uuid' in foreign key constraint 'sso_users_ibfk_1' are incompatible."))': src/db/mod.rs:505 Jan 06 12:13:28 [INFO] Using saved config from `/app/data/config.json` for configuration. Jan 06 12:13:28 [NOTICE] You are using a plain text `ADMIN_TOKEN` which is insecure. Jan 06 12:13:28 \--------------------------------------------------------------------/ Jan 06 12:13:28 | https://github.com/dani-garcia/vaultwarden/discussions or | Jan 06 12:13:28 | https://github.com/dani-garcia/vaultwarden/issues/new | Jan 06 12:13:28 | https://vaultwarden.discourse.group/ | Jan 06 12:13:28 | Report suspected bugs/issues in the software itself at: | Jan 06 12:13:28 | Send usage/configuration questions or feature requests to: | Jan 06 12:13:28 | official channels to report bugs/features, regardless of client. | Jan 06 12:13:37 => Healthcheck error: AbortError: The operation was aborted Jan 06 12:13:41 => Healthcheck error: Error: connect ECONNREFUSED 172.18.16.123:3000 Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 /--------------------------------------------------------------------\ Jan 06 12:13:41 0: vaultwarden::init_logging::{{closure}} Jan 06 12:13:41 10: std::sys::backtrace::__rust_begin_short_backtrace Jan 06 12:13:41 11: main Jan 06 12:13:41 12: <unknown> Jan 06 12:13:41 13: __libc_start_main Jan 06 12:13:41 14: _start Jan 06 12:13:41 1: std::panicking::panic_with_hook Jan 06 12:13:41 2: std::panicking::panic_handler::{{closure}} Jan 06 12:13:41 3: std::sys::backtrace::__rust_end_short_backtrace Jan 06 12:13:41 4: __rustc::rust_begin_unwind Jan 06 12:13:41 5: core::panicking::panic_fmt Jan 06 12:13:41 6: core::result::unwrap_failed Jan 06 12:13:41 7: vaultwarden::db::DbPool::from_config Jan 06 12:13:41 8: vaultwarden::main::{{closure}} Jan 06 12:13:41 9: vaultwarden::main Jan 06 12:13:41 => Exporting env vars expected by Vaultwarden Jan 06 12:13:41 => Starting Bitwarden Jan 06 12:13:41 Please generate a secure Argon2 PHC string by using `vaultwarden hash` or `argon2`. Jan 06 12:13:41 See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token Jan 06 12:13:41 [2026-01-06 12:13:41.691][panic][ERROR] thread 'main' panicked at 'Error running migrations: QueryError(DieselMigrationName { name: "2024-03-06-170000_add_sso_users", version: MigrationVersion("20240306170000") }, DatabaseError(Unknown, "Referencing column 'user_uuid' and referenced column 'uuid' in foreign key constraint 'sso_users_ibfk_1' are incompatible."))': src/db/mod.rs:505 Jan 06 12:13:41 [INFO] Using saved config from `/app/data/config.json` for configuration. Jan 06 12:13:41 [NOTICE] You are using a plain text `ADMIN_TOKEN` which is insecure. Jan 06 12:13:41 \--------------------------------------------------------------------/ Jan 06 12:13:41 | Starting Vaultwarden | Jan 06 12:13:41 | https://github.com/dani-garcia/vaultwarden/discussions or | Jan 06 12:13:41 | https://github.com/dani-garcia/vaultwarden/issues/new | Jan 06 12:13:41 | https://vaultwarden.discourse.group/ | Jan 06 12:13:41 | Report suspected bugs/issues in the software itself at: | Jan 06 12:13:41 | Send usage/configuration questions or feature requests to: | Jan 06 12:13:41 | This is an *unofficial* Bitwarden implementation, DO NOT use the | Jan 06 12:13:41 | official channels to report bugs/features, regardless of client. | Jan 06 12:13:41 |--------------------------------------------------------------------| Jan 06 12:13:52 => Healthcheck error: Error: connect EHOSTUNREACH 172.18.16.123:3000I then have to restore the 1.34.3 version for vaultwarden to work again. I have also had to disable auto update. Please note that I know there are two updates from 1.34.3, but both updates have the same connection issue.
-
So everytime I upgrade from Vaultwarden 1.34.3 to 1.35.0 the instance becomes unresponsive, saying connection failure in the terminal and nothing really in the logs apart from connection failure again.
Jan 06 12:13:28 0: vaultwarden::init_logging::{{closure}} Jan 06 12:13:28 10: std::sys::backtrace::__rust_begin_short_backtrace Jan 06 12:13:28 11: main Jan 06 12:13:28 12: <unknown> Jan 06 12:13:28 13: __libc_start_main Jan 06 12:13:28 14: _start Jan 06 12:13:28 1: std::panicking::panic_with_hook Jan 06 12:13:28 2: std::panicking::panic_handler::{{closure}} Jan 06 12:13:28 3: std::sys::backtrace::__rust_end_short_backtrace Jan 06 12:13:28 4: __rustc::rust_begin_unwind Jan 06 12:13:28 5: core::panicking::panic_fmt Jan 06 12:13:28 6: core::result::unwrap_failed Jan 06 12:13:28 7: vaultwarden::db::DbPool::from_config Jan 06 12:13:28 8: vaultwarden::main::{{closure}} Jan 06 12:13:28 9: vaultwarden::main Jan 06 12:13:28 Please generate a secure Argon2 PHC string by using `vaultwarden hash` or `argon2`. Jan 06 12:13:28 See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token Jan 06 12:13:28 [2026-01-06 12:13:28.418][panic][ERROR] thread 'main' panicked at 'Error running migrations: QueryError(DieselMigrationName { name: "2024-03-06-170000_add_sso_users", version: MigrationVersion("20240306170000") }, DatabaseError(Unknown, "Referencing column 'user_uuid' and referenced column 'uuid' in foreign key constraint 'sso_users_ibfk_1' are incompatible."))': src/db/mod.rs:505 Jan 06 12:13:28 [INFO] Using saved config from `/app/data/config.json` for configuration. Jan 06 12:13:28 [NOTICE] You are using a plain text `ADMIN_TOKEN` which is insecure. Jan 06 12:13:28 \--------------------------------------------------------------------/ Jan 06 12:13:28 | https://github.com/dani-garcia/vaultwarden/discussions or | Jan 06 12:13:28 | https://github.com/dani-garcia/vaultwarden/issues/new | Jan 06 12:13:28 | https://vaultwarden.discourse.group/ | Jan 06 12:13:28 | Report suspected bugs/issues in the software itself at: | Jan 06 12:13:28 | Send usage/configuration questions or feature requests to: | Jan 06 12:13:28 | official channels to report bugs/features, regardless of client. | Jan 06 12:13:37 => Healthcheck error: AbortError: The operation was aborted Jan 06 12:13:41 => Healthcheck error: Error: connect ECONNREFUSED 172.18.16.123:3000 Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 2026-01-06T12:13:41Z Jan 06 12:13:41 /--------------------------------------------------------------------\ Jan 06 12:13:41 0: vaultwarden::init_logging::{{closure}} Jan 06 12:13:41 10: std::sys::backtrace::__rust_begin_short_backtrace Jan 06 12:13:41 11: main Jan 06 12:13:41 12: <unknown> Jan 06 12:13:41 13: __libc_start_main Jan 06 12:13:41 14: _start Jan 06 12:13:41 1: std::panicking::panic_with_hook Jan 06 12:13:41 2: std::panicking::panic_handler::{{closure}} Jan 06 12:13:41 3: std::sys::backtrace::__rust_end_short_backtrace Jan 06 12:13:41 4: __rustc::rust_begin_unwind Jan 06 12:13:41 5: core::panicking::panic_fmt Jan 06 12:13:41 6: core::result::unwrap_failed Jan 06 12:13:41 7: vaultwarden::db::DbPool::from_config Jan 06 12:13:41 8: vaultwarden::main::{{closure}} Jan 06 12:13:41 9: vaultwarden::main Jan 06 12:13:41 => Exporting env vars expected by Vaultwarden Jan 06 12:13:41 => Starting Bitwarden Jan 06 12:13:41 Please generate a secure Argon2 PHC string by using `vaultwarden hash` or `argon2`. Jan 06 12:13:41 See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token Jan 06 12:13:41 [2026-01-06 12:13:41.691][panic][ERROR] thread 'main' panicked at 'Error running migrations: QueryError(DieselMigrationName { name: "2024-03-06-170000_add_sso_users", version: MigrationVersion("20240306170000") }, DatabaseError(Unknown, "Referencing column 'user_uuid' and referenced column 'uuid' in foreign key constraint 'sso_users_ibfk_1' are incompatible."))': src/db/mod.rs:505 Jan 06 12:13:41 [INFO] Using saved config from `/app/data/config.json` for configuration. Jan 06 12:13:41 [NOTICE] You are using a plain text `ADMIN_TOKEN` which is insecure. Jan 06 12:13:41 \--------------------------------------------------------------------/ Jan 06 12:13:41 | Starting Vaultwarden | Jan 06 12:13:41 | https://github.com/dani-garcia/vaultwarden/discussions or | Jan 06 12:13:41 | https://github.com/dani-garcia/vaultwarden/issues/new | Jan 06 12:13:41 | https://vaultwarden.discourse.group/ | Jan 06 12:13:41 | Report suspected bugs/issues in the software itself at: | Jan 06 12:13:41 | Send usage/configuration questions or feature requests to: | Jan 06 12:13:41 | This is an *unofficial* Bitwarden implementation, DO NOT use the | Jan 06 12:13:41 | official channels to report bugs/features, regardless of client. | Jan 06 12:13:41 |--------------------------------------------------------------------| Jan 06 12:13:52 => Healthcheck error: Error: connect EHOSTUNREACH 172.18.16.123:3000I then have to restore the 1.34.3 version for vaultwarden to work again. I have also had to disable auto update. Please note that I know there are two updates from 1.34.3, but both updates have the same connection issue.
Hello @apesorguk
I have merged your new topic into this one, since it is a duplication.
The fix has already been provided, see above https://forum.cloudron.io/post/117779 -
Hello @apesorguk
I have merged your new topic into this one, since it is a duplication.
The fix has already been provided, see above https://forum.cloudron.io/post/117779@james I get this problem.
root@bf8db951-7632-4ddf-bf9b-8b58cd65ffe5:/app/code# bash /app/data/fix_db.sh /app/data/fix_db.sh: line 2: $'\r': command not found => Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci /app/data/fix_db.sh: line 4: $'\r': command not found /app/data/fix_db.sh: line 8: $'\r': command not found /app/data/fix_db.sh: line 22: syntax error near unexpected token `$'in\r'' 'app/data/fix_db.sh: line 22: ` case $yn in root@bf8db951-7632-4ddf-bf9b-8b58cd65ffe5:/app/code# -
Hello @factord
This looks like a copy-paste issue from Windows.
\rin line 2 is a blank line, so a line break.
When you copy and pasted, it pasted the line breaks windows style.Please describe how you have copy and pasted the code into the
/app/data/fix_db.shfile.@apesorguk looks like you've hit the same copy-paste issue @james described here:
@james said in Vaultwarden fails to start after update – DB migration error (SSO):
This looks like a copy-paste issue from Windows.
\rin line 2 is a blank line, so a line break.
When you copy and pasted, it pasted the line breaks windows style.Please describe how you have copy and pasted the code into the
/app/data/fix_db.shfile. -
Thanks all fixed now. I found this worked even with the line brake.
I have run the command sed -i 's/\r$//g' /app/data/fix_db.sh to fix the new line characters and then run bash /app/data/fix_db.sh again and it ran as expected. Disabled recover mode and confirmed things are back to working as expected. -
-
@overholt Very likely you've hit the same thing as this thread. Solution here https://forum.cloudron.io/post/117779
-
I used this issue to improve my aidevops cloudron subagent, and it managed to fix the issue on all instances. More info here for anyone interested to add this to your toolkit:
-
I have run the command
sed -i 's/\r$//g' /app/data/fix_db.shto fix the new line characters and then runbash /app/data/fix_db.shagain and it ran as expected. Disabled recover mode and confirmed things are back to working as expected.@ChristopherMag Thanks, this fixed my issue.
