Vaultwarden fails to start after update – DB migration error (SSO)
-
Hello @vaultwarden users
Since users have voiced their concerns about manually editing the database I have created a bash script that does it for you.
Please follow these steps if you have this issue:- create an app backup of your @vaultwarden Cloudron app
- put the erroring @vaultwarden app in recovery mode
- open the File Manager of your @vaultwarden app and create a
fix_db.shfile - copy and paste the following script into the just created
fix_db.shfile:#!/bin/bash echo "=> Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci" CURRENT_DEFAULT_CHARACTER_SET_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) CURRENT_DEFAULT_COLLATION_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) FIX_NEEDED=0 if [ "${CURRENT_DEFAULT_CHARACTER_SET_NAME}" != "utf8mb4" ] || [ "${CURRENT_DEFAULT_COLLATION_NAME}" != "utf8mb4_unicode_ci" ]; then echo "=> DEFAULT_CHARACTER_SET_NAME or DEFAULT_COLLATION_NAME is not set to utf8mb4 or utf8mb4_unicode_ci." echo "==> Current values are: DEFAULT_CHARACTER_SET_NAME=${CURRENT_DEFAULT_CHARACTER_SET_NAME}, DEFAULT_COLLATION_NAME=${CURRENT_DEFAULT_COLLATION_NAME}" FIX_NEEDED=1 else echo "=> DEFAULT_CHARACTER_SET_NAME is ${CURRENT_DEFAULT_CHARACTER_SET_NAME} and DEFAULT_COLLATION_NAME is ${CURRENT_DEFAULT_COLLATION_NAME}" echo "=> Nothing to do - quitting." FIX_NEEDED=0 exit 0 fi if [ ${FIX_NEEDED} -eq 1 ]; then read -p "Have you created a backup of your Cloudron Vaultwarden app? (y/N): " yn case $yn in [Yy]* ) echo "=> Setting database character set and collation to utf8mb4 and utf8mb4_unicode_ci." mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "ALTER DATABASE \`${CLOUDRON_MYSQL_DATABASE}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" 2>/dev/null echo "=> Converting all tables to character set utf8mb4 and collation utf8mb4_unicode_ci." mysql --silent --skip-column-names \ --user="${CLOUDRON_MYSQL_USERNAME}" \ --password="${CLOUDRON_MYSQL_PASSWORD}" \ --host="${CLOUDRON_MYSQL_HOST}" \ "${CLOUDRON_MYSQL_DATABASE}" \ -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,'\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS CharSetConvert FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='${CLOUDRON_MYSQL_DATABASE}' AND TABLE_TYPE=\"BASE TABLE\";" 2>/dev/null | while read -r sql_command; do echo "==> Executing: ${sql_command}" mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SET foreign_key_checks=0; ${sql_command} SET foreign_key_checks=1;" 2>/dev/null done ;; [Nn]* ) echo "=> Please create a backup of your database before proceeding with the fix. Exiting." exit 1 ;; * ) echo "=> Invalid response. Please answer with y or n. Exiting." exit 1 ;; esac fi - open the Web Terminal of your @vaultwarden Cloudron app
- execute the following command:
bash /app/data/fix_db.sh - Execute the following command to start @vaultwarden:
/app/pkg/start.sh - Validate if your @vaultwarden is now working correctly
- If validated working, you can delete the
fix_db.shfile and disable the recovery mode of your @vaultwarden Cloudron app
If you run into any issues, copy the output of the terminal, post it here and restore your app from the backup created.
@james this won’t fix the issue of logging everyone out though I assume?
Edit: just saw the newest update.
-
Hello @vaultwarden users
Since users have voiced their concerns about manually editing the database I have created a bash script that does it for you.
Please follow these steps if you have this issue:- create an app backup of your @vaultwarden Cloudron app
- put the erroring @vaultwarden app in recovery mode
- open the File Manager of your @vaultwarden app and create a
fix_db.shfile - copy and paste the following script into the just created
fix_db.shfile:#!/bin/bash echo "=> Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci" CURRENT_DEFAULT_CHARACTER_SET_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) CURRENT_DEFAULT_COLLATION_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) FIX_NEEDED=0 if [ "${CURRENT_DEFAULT_CHARACTER_SET_NAME}" != "utf8mb4" ] || [ "${CURRENT_DEFAULT_COLLATION_NAME}" != "utf8mb4_unicode_ci" ]; then echo "=> DEFAULT_CHARACTER_SET_NAME or DEFAULT_COLLATION_NAME is not set to utf8mb4 or utf8mb4_unicode_ci." echo "==> Current values are: DEFAULT_CHARACTER_SET_NAME=${CURRENT_DEFAULT_CHARACTER_SET_NAME}, DEFAULT_COLLATION_NAME=${CURRENT_DEFAULT_COLLATION_NAME}" FIX_NEEDED=1 else echo "=> DEFAULT_CHARACTER_SET_NAME is ${CURRENT_DEFAULT_CHARACTER_SET_NAME} and DEFAULT_COLLATION_NAME is ${CURRENT_DEFAULT_COLLATION_NAME}" echo "=> Nothing to do - quitting." FIX_NEEDED=0 exit 0 fi if [ ${FIX_NEEDED} -eq 1 ]; then read -p "Have you created a backup of your Cloudron Vaultwarden app? (y/N): " yn case $yn in [Yy]* ) echo "=> Setting database character set and collation to utf8mb4 and utf8mb4_unicode_ci." mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "ALTER DATABASE \`${CLOUDRON_MYSQL_DATABASE}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" 2>/dev/null echo "=> Converting all tables to character set utf8mb4 and collation utf8mb4_unicode_ci." mysql --silent --skip-column-names \ --user="${CLOUDRON_MYSQL_USERNAME}" \ --password="${CLOUDRON_MYSQL_PASSWORD}" \ --host="${CLOUDRON_MYSQL_HOST}" \ "${CLOUDRON_MYSQL_DATABASE}" \ -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,'\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS CharSetConvert FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='${CLOUDRON_MYSQL_DATABASE}' AND TABLE_TYPE=\"BASE TABLE\";" 2>/dev/null | while read -r sql_command; do echo "==> Executing: ${sql_command}" mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SET foreign_key_checks=0; ${sql_command} SET foreign_key_checks=1;" 2>/dev/null done ;; [Nn]* ) echo "=> Please create a backup of your database before proceeding with the fix. Exiting." exit 1 ;; * ) echo "=> Invalid response. Please answer with y or n. Exiting." exit 1 ;; esac fi - open the Web Terminal of your @vaultwarden Cloudron app
- execute the following command:
bash /app/data/fix_db.sh - Execute the following command to start @vaultwarden:
/app/pkg/start.sh - Validate if your @vaultwarden is now working correctly
- If validated working, you can delete the
fix_db.shfile and disable the recovery mode of your @vaultwarden Cloudron app
If you run into any issues, copy the output of the terminal, post it here and restore your app from the backup created.
-
@james said in Vaultwarden fails to start after update – DB migration error (SSO):
From version Cloudron package version 1.22.2 => Vaultwarden Version 1.34.3 to Cloudron package version 1.23.0 => Vaultwarden Version 1.35.0
Despite my worries and concerns, I never actually hit this problem. Guess I was lucky. But I wonder why?
My Vaultwarden started fine after doing that update. And then I immediately updated to the next version without issue too.
-
Hello @vaultwarden users
Since users have voiced their concerns about manually editing the database I have created a bash script that does it for you.
Please follow these steps if you have this issue:- create an app backup of your @vaultwarden Cloudron app
- put the erroring @vaultwarden app in recovery mode
- open the File Manager of your @vaultwarden app and create a
fix_db.shfile - copy and paste the following script into the just created
fix_db.shfile:#!/bin/bash echo "=> Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci" CURRENT_DEFAULT_CHARACTER_SET_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) CURRENT_DEFAULT_COLLATION_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) FIX_NEEDED=0 if [ "${CURRENT_DEFAULT_CHARACTER_SET_NAME}" != "utf8mb4" ] || [ "${CURRENT_DEFAULT_COLLATION_NAME}" != "utf8mb4_unicode_ci" ]; then echo "=> DEFAULT_CHARACTER_SET_NAME or DEFAULT_COLLATION_NAME is not set to utf8mb4 or utf8mb4_unicode_ci." echo "==> Current values are: DEFAULT_CHARACTER_SET_NAME=${CURRENT_DEFAULT_CHARACTER_SET_NAME}, DEFAULT_COLLATION_NAME=${CURRENT_DEFAULT_COLLATION_NAME}" FIX_NEEDED=1 else echo "=> DEFAULT_CHARACTER_SET_NAME is ${CURRENT_DEFAULT_CHARACTER_SET_NAME} and DEFAULT_COLLATION_NAME is ${CURRENT_DEFAULT_COLLATION_NAME}" echo "=> Nothing to do - quitting." FIX_NEEDED=0 exit 0 fi if [ ${FIX_NEEDED} -eq 1 ]; then read -p "Have you created a backup of your Cloudron Vaultwarden app? (y/N): " yn case $yn in [Yy]* ) echo "=> Setting database character set and collation to utf8mb4 and utf8mb4_unicode_ci." mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "ALTER DATABASE \`${CLOUDRON_MYSQL_DATABASE}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" 2>/dev/null echo "=> Converting all tables to character set utf8mb4 and collation utf8mb4_unicode_ci." mysql --silent --skip-column-names \ --user="${CLOUDRON_MYSQL_USERNAME}" \ --password="${CLOUDRON_MYSQL_PASSWORD}" \ --host="${CLOUDRON_MYSQL_HOST}" \ "${CLOUDRON_MYSQL_DATABASE}" \ -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,'\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS CharSetConvert FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='${CLOUDRON_MYSQL_DATABASE}' AND TABLE_TYPE=\"BASE TABLE\";" 2>/dev/null | while read -r sql_command; do echo "==> Executing: ${sql_command}" mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SET foreign_key_checks=0; ${sql_command} SET foreign_key_checks=1;" 2>/dev/null done ;; [Nn]* ) echo "=> Please create a backup of your database before proceeding with the fix. Exiting." exit 1 ;; * ) echo "=> Invalid response. Please answer with y or n. Exiting." exit 1 ;; esac fi - open the Web Terminal of your @vaultwarden Cloudron app
- execute the following command:
bash /app/data/fix_db.sh - Execute the following command to start @vaultwarden:
/app/pkg/start.sh - Validate if your @vaultwarden is now working correctly
- If validated working, you can delete the
fix_db.shfile and disable the recovery mode of your @vaultwarden Cloudron app
If you run into any issues, copy the output of the terminal, post it here and restore your app from the backup created.
-
@james Hi thx. But how to use the web terminal if it is not starting/responding? At least this is the Problem on my side currently. So I can not execute any commands there…
@Grienauer You need to put it into recovery mode, execure the script, take it out of it again
-
J james has marked this topic as solved
-
Hello @vaultwarden users
Since users have voiced their concerns about manually editing the database I have created a bash script that does it for you.
Please follow these steps if you have this issue:- create an app backup of your @vaultwarden Cloudron app
- put the erroring @vaultwarden app in recovery mode
- open the File Manager of your @vaultwarden app and create a
fix_db.shfile - copy and paste the following script into the just created
fix_db.shfile:#!/bin/bash echo "=> Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci" CURRENT_DEFAULT_CHARACTER_SET_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) CURRENT_DEFAULT_COLLATION_NAME=$(mysql --silent --skip-column-names --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SELECT DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = '${CLOUDRON_MYSQL_DATABASE}';" 2>/dev/null) FIX_NEEDED=0 if [ "${CURRENT_DEFAULT_CHARACTER_SET_NAME}" != "utf8mb4" ] || [ "${CURRENT_DEFAULT_COLLATION_NAME}" != "utf8mb4_unicode_ci" ]; then echo "=> DEFAULT_CHARACTER_SET_NAME or DEFAULT_COLLATION_NAME is not set to utf8mb4 or utf8mb4_unicode_ci." echo "==> Current values are: DEFAULT_CHARACTER_SET_NAME=${CURRENT_DEFAULT_CHARACTER_SET_NAME}, DEFAULT_COLLATION_NAME=${CURRENT_DEFAULT_COLLATION_NAME}" FIX_NEEDED=1 else echo "=> DEFAULT_CHARACTER_SET_NAME is ${CURRENT_DEFAULT_CHARACTER_SET_NAME} and DEFAULT_COLLATION_NAME is ${CURRENT_DEFAULT_COLLATION_NAME}" echo "=> Nothing to do - quitting." FIX_NEEDED=0 exit 0 fi if [ ${FIX_NEEDED} -eq 1 ]; then read -p "Have you created a backup of your Cloudron Vaultwarden app? (y/N): " yn case $yn in [Yy]* ) echo "=> Setting database character set and collation to utf8mb4 and utf8mb4_unicode_ci." mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "ALTER DATABASE \`${CLOUDRON_MYSQL_DATABASE}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" 2>/dev/null echo "=> Converting all tables to character set utf8mb4 and collation utf8mb4_unicode_ci." mysql --silent --skip-column-names \ --user="${CLOUDRON_MYSQL_USERNAME}" \ --password="${CLOUDRON_MYSQL_PASSWORD}" \ --host="${CLOUDRON_MYSQL_HOST}" \ "${CLOUDRON_MYSQL_DATABASE}" \ -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,'\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS CharSetConvert FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='${CLOUDRON_MYSQL_DATABASE}' AND TABLE_TYPE=\"BASE TABLE\";" 2>/dev/null | while read -r sql_command; do echo "==> Executing: ${sql_command}" mysql --user=${CLOUDRON_MYSQL_USERNAME} --password=${CLOUDRON_MYSQL_PASSWORD} --host=${CLOUDRON_MYSQL_HOST} ${CLOUDRON_MYSQL_DATABASE} -e "SET foreign_key_checks=0; ${sql_command} SET foreign_key_checks=1;" 2>/dev/null done ;; [Nn]* ) echo "=> Please create a backup of your database before proceeding with the fix. Exiting." exit 1 ;; * ) echo "=> Invalid response. Please answer with y or n. Exiting." exit 1 ;; esac fi - open the Web Terminal of your @vaultwarden Cloudron app
- execute the following command:
bash /app/data/fix_db.sh - Execute the following command to start @vaultwarden:
/app/pkg/start.sh - Validate if your @vaultwarden is now working correctly
- If validated working, you can delete the
fix_db.shfile and disable the recovery mode of your @vaultwarden Cloudron app
If you run into any issues, copy the output of the terminal, post it here and restore your app from the backup created.
@james it doesnt work for me, here is the output
root@b2a9826b-7631-435b-91a0-d0a505476ee9:/app/code# bash /app/data/fix_db.sh /app/data/fix_db.sh: line 2: $'\r': command not found => Checking if DEFAULT_CHARACTER_SET_NAME and DEFAULT_COLLATION_NAME are set to utf8mb4 and utf8mb4_unicode_ci /app/data/fix_db.sh: line 4: $'\r': command not found /app/data/fix_db.sh: line 8: $'\r': command not found /app/data/fix_db.sh: line 22: syntax error near unexpected token `$'in\r'' 'app/data/fix_db.sh: line 22: ` case $yn in root@b2a9826b-7631-435b-91a0-d0a505476ee9:/app/code# /app/pkg/start.sh => Exporting env vars expected by Vaultwarden => Starting Bitwarden /--------------------------------------------------------------------\ | Starting Vaultwarden | |--------------------------------------------------------------------| | This is an *unofficial* Bitwarden implementation, DO NOT use the | | official channels to report bugs/features, regardless of client. | | Send usage/configuration questions or feature requests to: | | https://github.com/dani-garcia/vaultwarden/discussions or | | https://vaultwarden.discourse.group/ | | Report suspected bugs/issues in the software itself at: | | https://github.com/dani-garcia/vaultwarden/issues/new | \--------------------------------------------------------------------/ [INFO] Using saved config from `/app/data/config.json` for configuration. [2026-01-03 17:24:33.038][panic][ERROR] thread 'main' panicked at 'Error running migrations: QueryError(DieselMigrationName { name: "2024-03-06-170000_add_sso_users", version: MigrationVersion("20240306170000") }, DatabaseError(Unknown, "Referencing column 'user_uuid' and referenced column 'uuid' in foreign key constraint 'sso_users_ibfk_1' are incompatible."))': src/db/mod.rs:505 0: vaultwarden::init_logging::{{closure}} 1: std::panicking::panic_with_hook 2: std::panicking::panic_handler::{{closure}} 3: std::sys::backtrace::__rust_end_short_backtrace 4: __rustc::rust_begin_unwind 5: core::panicking::panic_fmt 6: core::result::unwrap_failed 7: vaultwarden::db::DbPool::from_config 8: vaultwarden::main::{{closure}} 9: vaultwarden::main 10: std::sys::backtrace::__rust_begin_short_backtrace 11: main 12: <unknown> 13: __libc_start_main 14: _start root@b2a9826b-7631-435b-91a0-d0a505476ee9:/app/code#PS: i restored to 1.22.2 to get it working for now.
-
Hello @factord
This looks like a copy-paste issue from Windows.
\rin line 2 is a blank line, so a line break.
When you copy and pasted, it pasted the line breaks windows style.Please describe how you have copy and pasted the code into the
/app/data/fix_db.shfile. -
I have the same issue as @factord I am on windows 11 and created the file through the file manager, pasted the contents copied from the post into the file using the built in file editor that is a part of the file manager, saved the file, opened the web terminal and ran the
bash /app/data/fix_db.shcommand. -
I have run the command
sed -i 's/\r$//g' /app/data/fix_db.shto fix the new line characters and then runbash /app/data/fix_db.shagain and it ran as expected. Disabled recover mode and confirmed things are back to working as expected.
