SOLVED Bitwarden - Self-hosted password manager

[necrevistonnezr]

FYI: When you increment to build number in the Dockerfile and in CloudronManifest.json, bitwarden_rs 1.9.0 builds & installs without problems...

Maybe @fbartels can update the repo even while we're waiting for other databases being supported?

[fbartels]

@necrevistonnezr said in Bitwarden - Self-hosted password manager:

Maybe @fbartels can update the repo even while we're waiting for other databases being supported?

sure. done.

[iamthefij]

@girish any ideas on this?

[girish]

@iamthefij Sorry for the delay, got caught up with Cloudron 4. Now I have the time to investigate this a bit. From what I understand, the issue is that the scheduler container is unable to access the main container via HTTP? The scheduler container is supposed to be spawned in the same networking namespace and one is supposed to be able to directly access http://localhost:port. If that doesn't work, it's a bug. Let me test this and get back shortly.

[girish]

@iamthefij So, there is no way right now to reach out to the parent app container from the cron container. We used to use exec before but we removed it because there was actually no way to clean/delete exec containers (not sure if they have fixed this now). Those exec containers will basically hang around, so for a scheduler this means a new container keeps getting created and just accumulates garbage. IIRC, there was also a case where these scheduler containers were doing processing with files using /tmp and /run as scratchpads and then they mistakenly delete files of the parent container. This led me to change it to just spawn a completely new container. Finally, this also helps us in multi-host setups where the scheduler container can run anywhere (exec requires same pod).

I will try to make a fix tomorrow where the scheduler containers can somehow get to the app container (I guess injecting the hostname of the app container as env var will suffice).

Also, any reason why the "syncing" is not part of the main bitwarden_rs binary itself? That way the scheduler can just call bitwarden_rs ldap-sync instead of doing a http call?

[iamthefij]

@girish

We used to use exec before but we removed it because there was actually no way to clean/delete exec containers (not sure if they have fixed this now). Those exec containers will basically hang around, so for a scheduler this means a new container keeps getting created and just accumulates garbage.

I'm not sure I follow. Using docker run actually creates a new container by default. That is unless the --rm option is added. If added, it will remove the container after running. This is actually what Cloudron appears to do today.

In contrast, docker exec doesn't create any new container. It runs a process within an existing container. There is no need to clean up any containers after execution.

If the issue is that poorly written cron jobs are deleting files that should not be deleted, that sounds like a bug with the app, not with box. There are legitimate reasons to want to access the same filesystem. Maybe it's cleaning up logs or something. Periodically sending out files. Or, as in this case, accessing a SQLite database.

Also, any reason why the "syncing" is not part of the main bitwarden_rs binary itself?

That was a design decision by the original Bitwarden creator. Bitwarden_rs decided to follow the same convention.

That way the scheduler can just call bitwarden_rs ldap-sync instead of doing a http call?

Unfortunately, that would not get around this issue. Executing bitwarden_rs ldap-sync from a new container (created by docker run) would not have access to the same filesystem, and therefore it would write to a new SQLite database that would immediately be cleaned up.

[girish]

@iamthefij said in Bitwarden - Self-hosted password manager:

In contrast, docker exec doesn't create any new container. It runs a process within an existing container. There is no need to clean up any containers after execution.

If you see https://docs.docker.com/engine/api/v1.37/#operation/ContainerExec, it creates an "exec container" and returns the object id. This id is then used to start it at https://docs.docker.com/engine/api/v1.37/#operation/ExecStart. You will notice there is no API to delete this object. This object is only deleted when the main container is removed. Initially, I thought this will not be an issue but in practice, after a cron job runs more than 500 times (which is just 2-3 days), docker starts crawling and causes all sorts of strange problems. There is a github issue somewhere for this and iirc, the docker maintainers said that one should not exec too often.

@iamthefij said in Bitwarden - Self-hosted password manager:

Unfortunately, that would not get around this issue. Executing bitwarden_rs ldap-sync from a new container (created by docker run) would not have access to the same filesystem, and therefore it would write to a new SQLite database that would immediately be cleaned up.

The Scheduler run containers do have access to the filesystem/local storage by volume mounting. Otherwise, wp cron jobs cannot access wp plugins etc.

Also, regardless of above, I am working on a patch to make http access possible.

[iamthefij]

@girish wow! I had no clue that exec worked that way. TIL! Is there no garbage collection process? Seems strange. My host probably has a bunch of dangling execs. They’d seem like they’d be benign, but I wonder.

HTTP access would be a great way to solve this. Happy to help test or debug.

[necrevistonnezr]

Bitwarden_rs 1.9.1 is out
https://github.com/dani-garcia/bitwarden_rs/releases/tag/1.9.1

Fixed broken U2F in Chrome 74+
Added images to email
Updated dependencies

[fbartels]

I have updated the code in the git repo to 1.9.1 and also uploaded and submitted the app to the Cloudron app store. Maybe it will be shown as "untested" on Cloudrons on version 4.x.

[girish]

We pushed the app store release today for community apps. I will make a post tomorrow about how to get the community apps published so others can install easily.

[necrevistonnezr]

@girish said in Bitwarden - Self-hosted password manager:

We pushed the app store release today for community apps. I will make a post tomorrow about how to get the community apps published so others can install easily.

Please include how to migrate from an existing (testing) installation, if possible. Thanks!

[iamthefij]

@girish any luck with getting HTTP access to work?

[girish]

@iamthefij Oops, this got fixed a while ago. I thought I replied to this thread. You can use the env var CLOUDRON_APP_HOSTNAME now in 4.1. For example, curl http://$CLOUDRON_APP_HOSTNAME:3000 works if http is running on port 3000.

[girish]

@iamthefij btw, you can set "minBoxVersion": "4.1.4" in the manifest so that people who are below that version don't try to install the app.

[iamthefij]

Awesome! I will give this a shot.

[iamthefij]

@fbartels was there a reason for moving the bitwarden image from the COPY statement to a FROM statement at the beginning? I'm picking up LDAP support again now that the hostname is available and I'm getting the binary from a published images as well.

Was it just to avoid pulling when modifying any config values?

[iamthefij]

@girish Hmm... it doesn't seem to be working correctly.

I'm getting:

Jun 26 17:55:25 thread 'main' panicked at 'Could not authenticate with http://8e50545e-6293-459d-8aa8-5abdb13695dc-ldap_sync:3000. Error { kind: Hyper(Error { kind: Connect, cause: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" } }), url: Some("http://8e50545e-6293-459d-8aa8-5abdb13695dc-ldap_sync:3000/admin/") }', src/bw_admin.rs:62:17

It appears that the hostname is the hostname of the ldap_sync container that the cron job spawned? Is that correct? When I open a terminal for the app, it just gives the first part without ldap_sync, which seems right.

[girish]

@iamthefij This is indeed a bug CLOUDRON_APP_HOSTNAME is meant to be a constant and should be set to 8e50545e-6293-459d-8aa8-5abdb13695dc. For cron jobs, it is setting the env var as the hostname by mistake.

[girish]

@iamthefij In /home/yellowtent/box/src/docker.js at line 190,

`CLOUDRON_APP_HOSTNAME=${name}`,

Replace the above as:

`CLOUDRON_APP_HOSTNAME=${app.id}`,

Then systemctl restart box. Does that fix the problem?

[fbartels]

@iamthefij do you mean https://git.cloudron.io/fbartels/bitwardenrs-app/blob/master/Dockerfile#L1 ?

That was so that just a single line needs to be changed when bitwarden is updated.

[iamthefij]

@girish roger. That worked! Now I'm getting some new error from within Bitwarden_rs, which is good and means that it's actually hitting the server!

@fbartels got it. That could also be facilitated using an ARG.

It doesn't matter too much. The difference is really just in caching, but it doesn't look like the Cloudron build servers do caching.

[iamthefij]

@fbartels I'm not sure if something is different in my configuration... but if I visit my site at bitwarden.example.com/admin from a private window with no cookies, I'm not getting a basic auth prompt.

Is that working for you?

[iamthefij]

@girish is there anything that might prevent this container from querying LDAP?

I'm getting the following error:

Jun 27 18:30:57 thread 'main' panicked at 'rc=1 (operationsError), dn: "ou=users, dc=cloudron", text: "No such app"', src/main.rs:21:9

To verify that it's something to do with the cron container, I generate the config file and cat it to the log in the main application as well as the schedule container. I diffed the two configs and they are identical.

However, when I run the sync script from the terminal attached to the main container, it works correctly. From the scheduled container, I get this error.

Any ideas? I'm actually unable to find out where the "No such app" comes from. It is a pretty generic term, so searching online isn't much help. I did check the bitwarden_ldap_sync codebase, the Rust ldap3 codebase, and the box codebase, but no luck.

[girish]

@iamthefij I think that's the LDAP server not allowing scheduler containers from accessing it. Can you try this - https://git.cloudron.io/cloudron/box/commit/22d731f06da98af196c43e6713bf0ce551107fa6 ?

[iamthefij]

@girish woo!!! That did it! I'll just clean up all the stuff I did for debugging and I can publish it after the next Cloudron update.

[iamthefij]

@fbartels in addition to the admin page issue described above, I'm running into email sending issues again.

This time I'm not having success with any setting.

Explicit TLS only: Bitwarden hangs and times out when trying to invite/send with a panic:

Jun 28 03:35:56 thread 'main' panicked at 'Could not call with http://90859b46-d863-4609-a83a-79c5f010254c:3000/admin/invite. Error { kind: Io(Custom { kind: TimedOut, error: StringError("timed out") }), url: Some("http://90859b46-d863-4609-a83a-79c5f010254c:3000/admin/invite") }', src/bw_admin.rs:134:21
Jun 28 03:35:56 stack backtrace:
Jun 28 03:35:56 0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
Jun 28 03:35:56 at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
Jun 28 03:35:56 1: std::sys_common::backtrace::_print
Jun 28 03:35:56 at src/libstd/sys_common/backtrace.rs:70
Jun 28 03:35:56 2: std::panicking::default_hook::{{closure}}
Jun 28 03:35:56 at src/libstd/sys_common/backtrace.rs:58
Jun 28 03:35:56 at src/libstd/panicking.rs:200
Jun 28 03:35:56 3: std::panicking::default_hook
Jun 28 03:35:56 at src/libstd/panicking.rs:215
Jun 28 03:35:56 4: std::panicking::rust_panic_with_hook
Jun 28 03:35:56 at src/libstd/panicking.rs:478
Jun 28 03:35:56 5: std::panicking::continue_panic_fmt
Jun 28 03:35:56 at src/libstd/panicking.rs:385
Jun 28 03:35:56 6: std::panicking::begin_panic_fmt
Jun 28 03:35:56 at src/libstd/panicking.rs:340
Jun 28 03:35:56 7: bitwarden_rs_ldap::bw_admin::Client::invite
Jun 28 03:35:56 8: bitwarden_rs_ldap::invite_from_ldap
Jun 28 03:35:56 9: bitwarden_rs_ldap::main
Jun 28 03:35:56 10: std::rt::lang_start::{{closure}}
Jun 28 03:35:56 11: std::panicking::try::do_call
Jun 28 03:35:56 at src/libstd/rt.rs:49
Jun 28 03:35:56 at src/libstd/panicking.rs:297
Jun 28 03:35:56 12: __rust_maybe_catch_panic
Jun 28 03:35:56 at src/libpanic_unwind/lib.rs:87
Jun 28 03:35:56 13: std::rt::lang_start_internal
Jun 28 03:35:56 at src/libstd/panicking.rs:276
Jun 28 03:35:56 at src/libstd/panic.rs:388
Jun 28 03:35:56 at src/libstd/rt.rs:48
Jun 28 03:35:56 14: main

Enable SSL only: Bitwarden hangs and times out when trying to invite/send

Jun 28 03:37:54 [Fri Jun 28 03:37:54.553168 2019] [proxy_http:error] [pid 34] (70007)The timeout specified has expired: [client 172.18.0.1:36626] AH01102: error reading status line from remote server 127.0.0.1:3000, referer: https://bitwarden.iamthefij.com/admin

With both enabled: I get the handshake error.

Jun 28 03:40:52 [2019-06-28 03:40:52][bitwarden_rs::error][ERROR] Error sending email. handshake error

[fbartels]

@iamthefij yes, http auth is working in my branch.

I think I should add that I am not using the app for my production bitwarden install. This is still running on a different system.

I have also seen quite some weird behaviour in regards to the mail integration, but did not yet have the time or interest in really digging into it.

[girish]

Given that the email error is Timeout, could it be that the mail server hostname or port configuration is not set correctly?

[d19dotca]

Sorry if this is a newbie question (somewhat new to Cloudron) but how does one install this as a community app? I have the option for unstable apps showing enabled, but I do not see Bitwarden in the list of apps I can install. Am I missing something?

[girish]

@d19dotca Apps that are not published yet have to be built by hand. It's easy to build though:

git clone <app repo>
npm install -g cloudron-cli
cloudron build # this will ask your cloudron.io login
cloudron install # this will ask you for your cloudron's login

[murgero]

@girish Is the cloudron build command usable to all users or just developers right now?

[necrevistonnezr]

@murgero It's available to anyone, using the cli tool (https://cloudron.io/blog/2017-03-08-cli-part2.html)

[murgero]

@necrevistonnezr in theory sure, but last I tried (about a month ago) it said my user account was denied access (error 503)

[iamthefij]

Started digging back into email sending and got closer.

I just noticed that the Cloudron docs say that the STARTLS uses a self signed cert, which makes sense given that the host I'm using is mail and not my.example.com.

So I switched back to unencrypted and enabled more verbose logging. Ended up with:

Jul 25 16:34:22 [2019-07-25 16:34:22][rocket::rocket][INFO] POST /admin//invite/ application/json:
Jul 25 16:34:22 [2019-07-25 16:34:22][_][INFO] Matched: POST /admin/invite (invite_user)
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] connecting to 172.18.0.6:2525
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Read: 220 my.example.com ESMTP Haraka/2.8.23 ready<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp][INFO] connection established to 172.18.0.6:2525
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Wrote: EHLO 90859b46-d863-4609-a83a-79c5f010254c<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Read: 250-my.example.com Hello 90859b46-d863-4609-a83a-79c5f010254c.cloudron [172.18.0.12]Haraka is at your service.<CRLF>250-PIPELINING<CRLF>250-8BITMIME<CRLF>250-SMTPUTF8<CRLF>250-SIZE 26214400<CRLF>250 AUTH LOGIN PLAIN<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp][DEBUG] server my.example.com with {Authentication(Login), Authentication(Plain), SmtpUtfEight, EightBitMime}
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp][INFO] No supported authentication mechanisms available
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Wrote: MAIL FROM:<bitwarden.app@example.com> BODY=8BITMIME SMTPUTF8<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Read: 550 Mail from domain 'example.com' is not allowed from your host<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Wrote: QUIT<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][lettre::smtp::client][DEBUG] Read: 221 my.example.com closing connection. Have a jolly good day.<CRLF>
Jul 25 16:34:22 [2019-07-25 16:34:22][bitwarden_rs::error][ERROR] Error sending email. Mail from domain 'example.com' is not allowed from your host
Jul 25 16:34:22 [2019-07-25 16:34:22][_][INFO] Outcome: Success
Jul 25 16:34:22 172.18.0.1 - User [25/Jul/2019:16:34:22 +0000] "POST /admin/invite/ HTTP/1.1" 400 990 "https://bitwarden.example.com/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0"
Jul 25 16:34:22 [2019-07-25 16:34:22][_][INFO] Response succeeded.
Jul 25 16:34:24 [Thu Jul 25 16:34:24.254775 2019] [access_compat:error] [pid 32] [client 172.18.0.1:37122] AH01797: client denied by server configuration: proxy:http://127.0.0.1:3000/admin/

So this is definitely more descriptive. Seems like the server is rejecting the request.

[girish]

: 550 Mail from domain 'example.com' is not allowed from your host<CRLF>

Can you check if the email for example.com is not set to Disabled under the outbound relay email settings?

[necrevistonnezr]

Tried updating the bitwardenrs-app today to its current domain; usually it would just install (and upgrade) the app. Today I got the following error:

➜ cloudron install
ERROR (node:9463) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification. [ internal/process/warning.js:27:3 ]
Location: bit
ERROR Failed to install app. Domain 'bitwarden.domain.com' is in use [ /Users/user/.nvm/versions/node/v12.7.0/lib/node_modules/cloudron/src/helper.js:68:29 ]

[necrevistonnezr]

Tried with the LTS node version today - still the same error "Domain is in use". Anyone have an idea why?

[girish]

@necrevistonnezr You have to delete the DNS entry manually in the DNS provider. What's hapenned is that the dns already has an entry for the subdomain. Cloudron will never overwrite existing DNS records.

This is getting fixed with https://git.cloudron.io/cloudron/box/issues/644. Cloudron will then overwrite DNS entries with that flag set.

If the above is not the issue, maybe there is another app on that subdomain? (Also, check any redirects you have to set for other apps).

[necrevistonnezr]

@girish said in Bitwarden - Self-hosted password manager:

@necrevistonnezr You have to delete the DNS entry manually in the DNS provider. What's hapenned is that the dns already has an entry for the subdomain. Cloudron will never overwrite existing DNS records.

Where am I supposed to delete the DNS entry? Is the "DNS provider" something in the cloudron control panel?

I have no other apps / redirects regarding that subdomain.

[d19dotca]

@necrevistonnezr - He means delete the DNS entry from your DNS service provider itself for the domain. So if you already have bitwarden.domain.com in your DNS listing for your domain, then remove the bitwarden entry and then re-create with Cloudron.

[necrevistonnezr]

@d19dotca said in Bitwarden - Self-hosted password manager:

@necrevistonnezr - He means delete the DNS entry from your DNS service provider itself for the domain. So if you already have bitwarden.domain.com in your DNS listing for your domain, then remove the bitwarden entry and then re-create with Cloudron.

That was not the issue since I did not have a subdomain entry in my DNS settings (I have a wildcard DNS entry). The issue was that cloudron-cli was not able to connect to my cloudron properly (TLS, https issues) for reasons I don't know (I told support about it).

After using cloudron login --allow-selfsigned I was able to build and update the bitwarden...

[d19dotca]

@necrevistonnezr Ah interesting, good to know you figured that out then. Normally when that error is seen (I came across it quite a lot when setting up new environments on Cloudron as I learned the platform), it was always because I had already defined a "www" subdomain in my DNS provider, for example, so it then needed me to remove it before it could create it automatically by itself.

[iamthefij]

Ok. So I think I've just figured out the email issue, but don't have a fix yet.

When using the non-smtps connection, the logs show that lettre, the Rust email library that Bitwarden_rs uses, recognizes that the server supports several authentication methods, including Authentication(Plain), which is what swacks uses when successfully sending on the same port. However, lettre refuses to use it because we aren't using an encrypted connection and it doesn't want to send credentials in the clear. (Checked here, defined as empty here).

Unfortunately, using smtps is not a solution either because it appears that lettre is trying to validate the certificate, but that certificate is invalid.

A patch to Bitwarden_rs may be required to tell lettre to allow sending credentials over an insecure connection using this method or by telling it to accept insecure certs.

I'm working on a patch for Bitwarden_rs to allow insecure SSL connections so that it will accept a self signed cert.

[iamthefij]

Also, it looks like MySQL support is available now. Once I get all the email support sorted out, I'll try to roll that in as well.

[iamthefij]

@iamthefij Ok. My bitwarden_rs patch works. I'm going to submit that upstream.

[girish]

@iamthefij Talking to yourself again ?

[iamthefij]

On another note... There is no Docker image built for MySQL support yet, so I'm writing a new Dockerfile that uses a multi stage built to compile Bitwarden_rs with MySQL support.

[fbartels]

@iamthefij there is one already at https://github.com/dani-garcia/bitwarden_rs/tree/master/docker/amd64/mysql

[iamthefij]

@fbartels yes. But not an image on Docker Hub that I can pull from. Instead I'm building a multistage build that looks a lot like that Dockerfile but with the an LDAP layer and the Cloudron layer.

[fbartels]

@iamthefij it's here https://hub.docker.com/r/bitwardenrs/server-mysql

[iamthefij]

@fbartels well, I guess I didn't look hard enough! (Edit: Looks like the 1.10.0 release was only 2 hours ago. No wonder I missed it. ) It would be good to roll those all into one image with multiple tags, but I can chat with them about that on Matrix.

Anyway, adding the multi-stage build wasn't terribly hard and cuts out an extra dependency on that build pipeline, which is probably a good thing for a security sensitive project.

Latest on my master is now fully operational. Working email, working ldap sync, working MySQL.

BIG WARNING! There is no migration path from SQLite to MySQL. You should export your vault to CSV or something and then re-import it after migration.

https://git.cloudron.io/iamthefij/bitwardenrs-app

[necrevistonnezr]

@iamthefij Maybe the migration scripts by the author work that are mentioned here?

* Start bitwarden_rs with and empty mysql database, so diesel can run migrations and set up the schema properly. Do not do anything else.
* Stop bitwarden_rs.
* Dump your existing sqlite database: sqlite3 db.sqlite3 .dump > sqlitedump.sql
* Drop schema creation and diesel metadata from your dump, leaving only your actual data: grep "INSERT INTO" sqlitedump.sql | grep -v "__diesel_schema_migrations" > mysqldump.sql

* Load your MySQL dump: mysql -u bitwarden -p bitwarden < mysqldump.sql

* Start bitwarden_rs
ˋˋˋ

[iamthefij]

@necrevistonnezr I saw that. Yea, it's possible to do, but it's not something that I think is worth automating given that it could be a bit finicky.

If someone wants to do this, they are welcome to try using the console.

I'm planning to just export and re-import myself.

[necrevistonnezr]

@iamthefij but import / export does not work for attachments, right? Or are these somehow migrateable?

[iamthefij]

@necrevistonnezr hmm. I've got no clue. I don't have any attachments and only use Bitwarden to manage passwords.

[lukaszj]

@iamthefij I've used your build with fixed emails and ldap and it installed well however I'm not sure how to get in.

Bitwarden asks for email so I supply email corresponding to my Cloudron login and Cloudron's password but it says username or password is invalid. Can you suggest something?

[iamthefij]

@lukaszj you should receive an invite email within a few min of the sync task running. I think it runs every 5 min. You'll need to click the link in that email to create your account.

[jimcavoli]

With all the activity on this thread, is it likely that Bitwarden will see release in the App Store any time soon? Even as an "unstable" package for easier trial by others?

[girish]

@jimcavoli We will take this up next week since we are working on getting 4.2 out this week. Will follow up with @iamthefij on what the status is.