Bitwarden Stops Working

[will]

@girish Correct

[msbt]

@girish said in Bitwarden Stops Working:

/run/bitwardenrs/bitwardenrs.log

happened to me as well now, this is what the log file is showing:

[2020-07-01 21:28:13][panic][ERROR] thread 'main' panicked at 'Can't connect to DB: BadConnection("Can\'t connect to MySQL server on \'mysql\' (111)")': src/main.rs:235
   0: bitwarden_rs::init_logging::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/panicking.rs:490
   2: rust_begin_unwind
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/panicking.rs:388
   3: core::panicking::panic_fmt
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libcore/panicking.rs:101
   4: core::option::expect_none_failed
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libcore/option.rs:1272
   5: bitwarden_rs::main
   6: std::rt::lang_start::{{closure}}
   7: std::rt::lang_start_internal::{{closure}}
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/rt.rs:52
      std::panicking::try::do_call
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/panicking.rs:297
      std::panicking::try
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/panicking.rs:274
      std::panic::catch_unwind
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/panic.rs:394
      std::rt::lang_start_internal
             at /rustc/74e80468347471779be6060d8d7d6d04e98e467f/src/libstd/rt.rs:51
   8: main
   9: __libc_start_main
  10: _start

[imc67]

My BW app also seemd to be crashed, bad thing is that Cloudron Healthcheck didn't noticed:

box:tasks 4110: {"percent":100,"result":null,"error":{"stack":"BoxError: Unknown install command in apptask:installed\n at /home/yellowtent/box/src/apptask.js:1048:29\n at /home/yellowtent/box/src/apps.js:520:13\n at Query.<anonymous> (/home/yellowtent/box/src/appdb.js:147:13)\n at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10)\n at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16)\n at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)\n at Query._handleFinalResultPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:149:8)\n at Query.EofPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:133:8)\n at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)\n at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)","name":"BoxError","reason":"Internal Error","details":{},"message":"Unknown install command in apptask:installed"}}
Jul 04 14:35:30 172.18.0.1 - - [04/Jul/2020:12:35:30 +0000] "GET /healthcheck HTTP/1.1" 200 173 "-" "Mozilla (CloudronHealth)"
Jul 04 14:35:40 172.18.0.1 - - [04/Jul/2020:12:35:40 +0000] "GET /healthcheck HTTP/1.1" 200 173 "-" "Mozilla (CloudronHealth)"
Jul 04 14:35:50 172.18.0.1 - - [04/Jul/2020:12:35:50 +0000] "GET /healthcheck HTTP/1.1" 200 173 "-" "Mozilla (CloudronHealth)"

[imc67]

Today, after a security reboot, the Bitwarden app didn't came up, same error and also healthcheck didn't noticed.

In-app restart solved it.

This is really annoying.

[nebulon]

Yes, I have seen that as well sometimes after a server reboot, not with bitwarden particularily but with other apps not coming up well and requiring an app restart. So far I wasn't able to point a finger but generally we may have to start containers maybe slower in sequence to not overload the system at once.

[msbt]

yea i have that with teamspeak, almost everytime it needs another restart because mysql wasn't ready yet or something.

[d19dotca]

@nebulon I've never personally seen this issue on reboot, but I have seen the original issue at times with sometimes an app stops responding but there's no notification of it and I learn of it reactively instead of proactively to restart the app. Almost like the monitor still sees it as healthy when it's not actually responding to any client requests to the website.

[girish]

When the database restarts, some apps cannot deal with the loss of connectivity. In fact, we have special code on Cloudron to restart all the corresponding apps whenever the database is updated. I guess we need similar logic for database restart. I think docker has some way to get these event notifications. Something to keep in mind when debugging this.

[nebulon]

The latest version now retries apparently. So I guess this is solved.

[d19dotca]

@nebulon Ah yes I see what you mean. Let's hope this does the trick. From the changelog: "Now the initial database connection doesn't fail instantly when there's an error, but retries for up to 15 times by default, adjustable with the DB_CONNECTION_RETRIES option."