Software Engineer turned Director at a Silicon Valley company.
After the switch to management I was eager for a technical project and decided to take on self hosting for privacy and ethical reasons as well as the technical challenge. I’ve made several attempts at managing all my self hosted services from scratch and eventually found Cloudron.
I've got a branch where I've almost gotten LDAP syncing fully working, but invites don't seem to be working properly.
Even if I try to send an invite directly through the web interface, it just hangs. The user shows in the list, however the logs never show a success or failure response for the request. I've checked the SMTP settings and they appear to be correct. I'll keep debugging though.
https://git.cloudron.io/iamthefij/bitwardenrs-app/tree/ldap-sync
@nicolas Are your wordpress instances up to date? Do you have plugins installed in those instances?
That's where my mind goes immediately.
@yusf What are you wanting to see incorporated? The directory sync connector?
That diff that @girish linked is to add experimental support for the upstream Directory Connector APIs to allow you to use the upstream connector.
The directory connector could probably be added as a separate app much like ONLYOFFICE is with Nextcloud.
Alternately, I wrote the original bitwarden_rs_ldap connector, which was supported from within the single install. It was auto configured and then triggered by a timer every 5 min to auto send invites. The reason it wasn't included in the final Cloudron release was because the LDAP connector doesn't in the same way as other Cloudron apps and it was confusing to the users who were testing.
As @girish said, it works by sending users invites. Passwords cannot be synced because the Bitwarden server never even gets to know your password.
It looks like it has been removed, but we could probably patch back in the old LDAP sync at least and make it something that could be configured using file manager or the terminal as an advanced feature.
LDAP syncing is now available for bitwarden_rs. Check it out on the wiki.
Do you have your app shared on git.cloudron.io yet? If so I can help contribute.
It would seem that supporting Keycloak would be a great way to still only really have to maintain LDAP on the Cloudron side and then add support for OpenID Connect, OAuth 2.0
and SAML 2.0.
I've never set up Keycloak though, so I can't speak to it's ease of use or maintaining, but it is often recommended when people talk about FOSS Identity and Access Management tools.
Got it working! Turns out I needed to enable SMTP_EXPLICIT_TLS.
Now I just have to schedule the sync task and do some cleanup. Should have a fully ready app soon.
@will just a note, I don't believe fbartels version supports a using a dump for backing up the database. This means that if the backup is taken while the db is in a transaction, it could be corrupted.
Bitwarden_rs now supports an admin API for making sqlite backups, but does not have any cron embedded. Similar to the way the LDAP sync tool works, an additional script could be added to periodically make dumps of the sqlite database so that it can be properly backed up.
Instead, the version I have is using MySQL, which leverages the native Cloudron backup and restore functionality.
That and the LDAP invite service are the real differences between the two forks. If you do not wish to use automated LDAP invites on my fork, you can select to opt out when installing. This is covered in the readme.
@yusf Yea, the Readme describe the reasoning.
There is no way to actually do true SSO without breaking the model for Bitwarden. The only thing that we can do is automatically invite users to sign up.
The Bitwarden_rs project doesn't have a way to invite without sending an email as when an SMTP server is configured, it will generate unique invite links for each user.
If you disable SSO, you only disable the auto-invite feature. You will then need to invite yourself via the Admin panel (admin token is echoed in the logs and in /app/data/admin_token). You can then invite anyone else you wish manually.
I'd prefer to restrict a Cloudron instance to a particular zone rather than use the Global API Key. Whenever I do so, I get an error from Cloudron. What should the account be scoped to? Or is it even possible to use this?
@iamthefij Ok. Just restored matomo from backup again and it's all good.
@nebulon Ever since I brought up my system this way, backups have failed.
Unexpected response code when piping https://172.18.0.4:3000/databases/0e9ccf544025299e/backup?access_token=<token>: 404 message: Not Found filename: /home/yellowtent/appsdata/<id>/mysqldump
When I check that path, there is indeed a file there, but it's an empty one.
It looks like it may be one app (Matomo) failing and canceling the entire backup operation since one app (Gitea) does get backed up before the failure.
I just disabled backups for Matomo and I'm trying to back up the entire box again now to narrow it down.
Oh, interesting. Matomo says "Database access denied". So something is up there.
@nebulon yea. I put apps into recovery and stopped them, then wiped the data, started restored, turned off recovery for each one by one. Not sure if there’s a root cause for this. It happened right after some app updates.
I'm following this guide to wipe away and restore: https://docs.cloudron.io/troubleshooting/#mysql-addon
Hoping that works.
After some automatic app upgrades, my MySQL service is no longer starting.
My Event Log shows apps stopping responding after updates to Ghost, Wekan, and Collabora office.
Yesterday healthmonitor wekan.example.com (Wekan) is down
Yesterday cron Ghost at blog.example.com was updated to v3.189.3
Yesterday cron Update of Ghost at blog.iamthefij.com started from v3.189.3 to v3.190.0
Yesterday healthmonitor blog.example.com (Ghost) is down
Yesterday cron Wekan at wekan.example.com was updated to v4.16.2
Yesterday cron Ghost at blog.example.com was updated to v3.189.2
Yesterday cron Collabora Office at docs.example.com was updated to v1.10.18
Yesterday cron Update of Wekan at wekan.example.com started from v4.16.2 to v4.16.3
Yesterday cron Update of Collabora Office at docs.example.com started from v1.10.18 to v1.10.19
Yesterday cron Update of Ghost at blog.example.com started from v3.189.2 to v3.189.3
One odd thing I see is that it appears to apply updates in odd orders... Eg. Chost from v3.189.2 to v3.189.3 started, then it says it completed update to v3.189.2 (old version). The same thing happens again shortly after showing update of Ghost at blog.example.com started from v3.189.3 to v3.190.0 and then saying that it's been updated to v3.189.3. Maybe that is just an error in the Event Log though posting the old version when the update is completed.
When I inspect the mysql container I find the following in the logs:
root@mysql:/# cat /tmp/mysqld.err
2021-11-07T17:18:12.374786Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.23-0ubuntu0.20.04.1) starting as process 13
2021-11-07T17:18:12.399005Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-11-07T17:18:13.560419Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-11-07T17:18:14.130206Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2021-11-07T17:18:14.224525Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: trx0purge.cc:174:purge_sys->iter.trx_no <= purge_sys->rseg->last_trx_no thread 139872211805952
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:18:14 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7f3668000b60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2021-11-07T17:18:14.227292Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-11-07T17:18:14.227358Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
stack_bottom = 7f36897f9ba0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x5628ef0ac961]
/usr/sbin/mysqld(handle_fatal_signal+0x31b) [0x5628eded7e0b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f36cfcd73c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f36cf33518b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f36cf314859]
/usr/sbin/mysqld(+0xe89b63) [0x5628edc38b63]
/usr/sbin/mysqld(TrxUndoRsegsIterator::set_next()+0xa48) [0x5628ef392a18]
/usr/sbin/mysqld(+0x25e3d1e) [0x5628ef392d1e]
/usr/sbin/mysqld(+0x25e6931) [0x5628ef395931]
/usr/sbin/mysqld(trx_purge(unsigned long, unsigned long, bool)+0x13d) [0x5628ef39603d]
/usr/sbin/mysqld(srv_purge_coordinator_thread()+0x5be) [0x5628ef35ddae]
/usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Runnable, void (*)()> > >::_M_run()+0xbd) [0x5628ef25483d]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6d84) [0x7f36cf721d84]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f36cfccb609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f36cf411293]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
I've done some research on recovering from database corruption and I'll try to follow that, but since all apps are backed up, I'm wondering if there's a way to blow away this database and just restore all apps that depend on MySQL from backup.
@girish creds in mysql and env both match. Does the mail server authenticate using mysql directly?
@girish I didn't see any relevant logs there when attempting to send an email. Just a bunch of health monitor checks.
Just verified that my Wekan instance can send emails just fine. So something to do with my Vaultwarden instance in particular.
Not sure if this is effecting other apps than Vaultwarden right now, but when I send an email I'm getting an error. If I send a test email from the admin interface, the logs on the Email server show:
Aug 09 09:35:58 [NOTICE] [CD43F71D-77D4-4615-8955-474CFD81836B] [core] connect ip=x.x.x.x port=41932 local_ip=x.x.x.x local_port=2525
Aug 09 09:35:58 [INFO] [CD43F71D-77D4-4615-8955-474CFD81836B] [helo.checks] multi: true, skip:proto_mismatch(private), host_mismatch(private)
Aug 09 09:35:58 [INFO] [-] [cloudron] (cn=bitwarden@example.com,ou=sendmail,dc=cloudron) cn=bitwarden@example.com, ou=sendmail, dc=cloudron
Aug 09 09:35:58 [NOTICE] [CD43F71D-77D4-4615-8955-474CFD81836B] [cloudron] delaying for 1 seconds
Aug 09 09:35:59 [INFO] [CD43F71D-77D4-4615-8955-474CFD81836B] [core] hook=unrecognized_command plugin=cloudron function=hook_unrecognized_command params=AUTH retval=OK msg=""
Aug 09 09:35:59 [INFO] [CD43F71D-77D4-4615-8955-474CFD81836B] [core] client half closed connection ip=x.x.x.x
Aug 09 09:35:59 [NOTICE] [CD43F71D-77D4-4615-8955-474CFD81836B] [core] disconnect ip=x.x.x.x rdns=ccec216e-cf35-4df3-ad1b-9e18b59502c5.cloudron helo=ccec216e-cf35-4df3-ad1b-9e18b59502c5 relay=N early=N esmtp=Y tls=N pipe=N errors=0 txns=0 rcpts=0/0/0 msgs=0/0/0 bytes=0 lr="535 5.7.8 Authentication failed" time=1.022
To debug, I tried to send using swaks and got an error as well:
root@ccec216e-cf35-4df3-ad1b-9e18b59502c5:/app# swaks --server "${CLOUDRON_MAIL_SMTP_SERVER}" -p "${CLOUDRON_MAIL_SMTP_PORT}" --from "${CLOUDRON_MAIL_FROM}" --body "Test mail from cloudron app at $(hostname -f)" --auth-user "${CLOUDRON_MAIL_SMTP_USERNAME}" --auth-password "${CLOUDRON_MAIL_SMTP_PASSWORD}"
To: ian@example.com
=== Trying mail:2525...
=== Connected to mail.
<- 220 my.example.com ESMTP Haraka/2.8.27 ready
-> EHLO ccec216e-cf35-4df3-ad1b-9e18b59502c5
<- 250-my.iamthefij.com Hello ccec216e-cf35-4df3-ad1b-9e18b59502c5.cloudron [x.x.x.x]Haraka is at your service.
<- 250-PIPELINING
<- 250-8BITMIME
<- 250-SMTPUTF8
<- 250-SIZE 26214400
<- 250 AUTH LOGIN PLAIN
-> AUTH LOGIN
<- 334 VX<somehash>WU6
-> Kml0<somehash>oZWZpai5jb20=
<- 334 UGFzc3dvcmQ6
-> ATkOT<somehash>M3Y2ExMWMzMTk1
<** 535 5.7.8 Authentication failed
-> AUTH PLAIN UEOpdHdhcmRlbkBpY<somehash>MjUxZGM0OWFjN2NhMTFjMzE5NQ==
<** 535 5.7.8 Authentication failed
*** No authentication type succeeded
-> QUIT
<- 221 my.example.com closing connection. Have a jolly good day.
=== Connection closed with remote host.
Email addon logs
Aug 09 09:38:21 [NOTICE] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [core] connect ip=x.x.x.x port=42596 local_ip=x.x.x.x local_port=2525
Aug 09 09:38:21 [INFO] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [helo.checks] multi: true, skip:proto_mismatch(private), host_mismatch(private)
Aug 09 09:38:21 [INFO] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [core] hook=unrecognized_command plugin=cloudron function=hook_unrecognized_command params=AUTH retval=OK msg=""
Aug 09 09:38:21 [INFO] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [core] hook=unrecognized_command plugin=cloudron function=hook_unrecognized_command params="Kml0<somehash>oZWZpai5jb20=" retval=OK msg=""
Aug 09 09:38:21 [INFO] [-] [cloudron] (cn=bitwarden@example.com,ou=sendmail,dc=cloudron) cn=bitwarden@example.com, ou=sendmail, dc=cloudron
Aug 09 09:38:21 [NOTICE] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [cloudron] delaying for 1 seconds
Aug 09 09:38:22 [INFO] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [core] hook=unrecognized_command plugin=cloudron function=hook_unrecognized_command params=ATkOT<somehash>M3Y2ExMWMzMTk1 retval=OK msg=""
Aug 09 09:38:22 [INFO] [-] [cloudron] (cn=bitwarden@example.com,ou=sendmail,dc=cloudron) cn=bitwarden@example.com, ou=sendmail, dc=cloudron
Aug 09 09:38:22 [NOTICE] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [cloudron] delaying for 2 seconds
Aug 09 09:38:24 [INFO] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [core] hook=unrecognized_command plugin=cloudron function=hook_unrecognized_command params=AUTH retval=OK msg=""
Aug 09 09:38:24 [NOTICE] [AE97B67D-E5F8-47A8-9E07-B664151018A6] [core] disconnect ip=x.x.x.x rdns=ccec216e-cf35-4df3-ad1b-9e18b59502c5.cloudron helo=ccec216e-cf35-4df3-ad1b-9e18b59502c5 relay=N early=N esmtp=Y tls=N pipe=N errors=0 txns=0 rcpts=0/0/0 msgs=0/0/0 bytes=0 lr="535 5.7.8 Authentication failed" time=3.043
It looks to me like maybe the credentials that are being passed to my application are potentially invalid. I'm not sure how to cycle this or test another way.
@yusf yea, that was the feedback from the other thread too. Unfortunately, if email is enabled, Bitwarden_rs will automatically send emails for all invited users. An upstream change to provide an API option to skip sending emails would need to be added.