Mosquitto and TLS certs question

[jadudm]

Hi all,

I'm working on a mosquitto (MQTT) package. I'm... wrestling with testing a variety of clients, and am uncertain where I have a failure with an SSL handshake from one of my devices. Hence, I'm fishing.

When I configure mosquitto, I need to give it the following for TLS to work (in theory?):

certfile /etc/letsencrypt/live/mqtt.example.com/cert.pem
cafile /etc/letsencrypt/live/mqtt.example.com/chain.pem
keyfile /etc/letsencrypt/live/mqtt.example.com/privkey.pem

Using the TLS addon, I seem to be able to provide... two of these three (certfile and keyfile)? If I'm using the cloudron base image, is there a path I can pass to mosquitto in its config to get a valid cafile?

Why I'm fishing up this stream: I'm wondering if the absence of the cafile is something that could break a TLS handshake? (Some clients I test work fine; some don't. I'm wondering if there's a cipher mismatch or TLS v1.2 vs v1.3 problem here...)

Many thanks,
Matt

[girish]

@jadudm the CA file I think is the intermediate certificate. this is already embedded in the cert file. Is it not optional in MQTT config?

[girish]

You can get the intermediate certs here - https://letsencrypt.org/certificates/#intermediate-certificates , if it's really needed. I think "Let’s Encrypt R3" is the one which is active now (ideally, this should not be hardcoded though).

[jadudm]

@girish Many thanks; that was useful to know about/something I did not know existed (the LE cert page), but sadly didn't solve my problem. (I was hoping for magic.)

I'm going to call it a day; it is clear that mosquitto should support the cipher that tasmota is offering, but I keep getting

error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher

on the broker side. I hoped that this might be a "missing piece."

When I have this figured out, I'll feel better about suggesting there possibility of a viable MQTT container.

(My goal here is to pair it with Home Assistant, and know that Tasmota, an open source firmware for home automation devices, will work with the MQTT broker.)

[jadudm]

@girish I have learned more things.

I know/understand certs and PKI at a high level, not at a here's now it works level. I'm in the process of learning more of here's how it works to solve this.

I was able to stand up a container with mosquitto running just fine, and as long as the clients that talked to it were running on full-featured hosts, everything "just worked" (e.g. running mosquitto_sub on another Linux host). This works for a cloudron-packaged version of Home Assistant, which bundles an MQTT client, and performs auto-discovery. It has happily discovered my MQTT container, which is good, and sends messages to it.

However, my mosquitto container did not work with my (one) piece of IoT test hardware. Specifically, it did not work for my little "smart switch," based on an ESP8266 (a Sonoff Basic). This is an 80MHz device with KB of RAM and 1MB of flash. As a result, the Tasmota team uses BearSSL as the SSL implementation, and it only deals with one encryption algorithm and TLSv1.2. It is known to work with mosquitto (being an Eclipse foundation "standard" of sorts in the MQTT broker world). As proof that it works, I can build an insecure container (meaning I configure mosquitto to use comms unencrypted by TLS), and the Tasmota device will talk to the container just fine. That confirmed for me that it wasn't a network issue... it is definitely a cert/encryption question.

But, it looks like there's a cert question w.r.t. the LetsEncrypt certs that cloudron pulls down. Or, that's where I'm going to investigate next.

From a developer on the Tasmota end of things:

I used the laptop version of BearSSL and I got the same error:

build/brssl client mqtt.example.com:443 -cs ECDHE_RSA_WITH_AES_128_GCM_SHA256
WARNING: no configured trust anchor
connecting to: 1.2.3.4 connected.
Algorithms:
    RNG:           rdrand
    AES/CTR:       pwr8
    GHASH (GCM):   pclmul
    EC:            all_m31
    RSA (vrfy):    i62
 ERROR: SSL error 296 (received alert 40)

Using OpenSSL shows that Server public key is 384 bit
We don't support EC keys for servers, they need to be RSA 2048 or 4096. In general we don't support ECDSA. You need to generate a new certificate with RSA private/public key

Now, this is where I need to do more learning. Does cloudron pull an elliptic-curve-based cert (as opposed to an RSA-based cert), and that's going to be something I need to work around? (For example, I could bundle self-signed certs into the mosquitto container, or generate them when the container is launched, or...). Or, is there something I don't (yet) know about cloudron that will let me solve/work around this?

If y'all have insight, I welcome it. In the meantime, I'm going to go digging into the box source, and find out what happens when cloudron requests LE certs on my behalf.

When I'm done, I'll document all of this somewhere, so that people following after me don't have to walk the same path.

Cheers,
M

[jadudm]

@girish, Based on

https://git.cloudron.io/cloudron/box/-/blob/master/src/acme2.js#L279

which is part of the function Acme2.prototype.createKeyAndCsr, suggesting... it is where the key and CSR are created...

let key = safe.child_process.execSync('openssl ecparam -genkey -name secp384r1');

which suggests a key based on an EC.

But, this is a few minutes of code spelunking. I'm going to hold for a bit, and see if you have any thoughts. I'll do some more reading, but I have a suspicion solving this is going to require more access to box than an app packager currently has.

I wonder if I can include a self-signed cert in the app package that matches the domain as a stop-gap?

Anyway. Thoughts welcome, and thank you.

[jadudm]

And, now that I've thought about it some more.

1. Being able to run a TLS-protected MQTT server is good. It allows full-featured clients to talk to the sever. That said,
2. Embedded devices will need the CA compiled in. (Or, I might be able to get away with a fingerprint. That said...)
3. It would require an RSA cert, no larger than 2048 bits.

This could be more annoying than it is worth. Certainly, having an RSA cert that would play with BearSSL would be nice, but it might not be in the cards.

I'll have to think about whether there are other options/approaches available.

[jadudm]

For anyone following along at home... I have, in "draft" form:

1. A Home Assistant app, and
2. An MQTT instance that:
   • uses self-signed certs, and
   • listens on 8883 (the default secure port) with TLS

The self-signed certs are because I'm working against the constraints of BearSSL, which means I need 2048-bit RSA CA/keys and SHA256.

However, the test is that I now have a Sonoff Basic switch that is chattering away at my Home Assistant instance over TLS-secured MQTT.

There's a bunch of cleanup I'd like to do in order to make these apps usable for others. The MQTT container alone might be generally usable... perhaps I can put the LetsEncrypt certs on the default port, and put the self-signed certs on a different port (e.g. 8883 for LE, and 9993 for the self-signed certs...)

[girish]

@jadudm said in Mosquitto and TLS certs question:

We don't support EC keys for servers, they need to be RSA 2048 or 4096. In general we don't support ECDSA. You need to generate a new certificate with RSA private/public key

Ah, yet another service not supporting ECDSA certs. I wonder why this. IIRC, zapier also had the same limitation. I wonder if this is because those certs don't work well on older devices?

[jadudm]

@girish In the case, it has to do with embedded systems.

Tasmota is a FOSS firmware that runs on a variety of ESP8266 and ESP32-based devices. (https://en.wikipedia.org/wiki/ESP8266) Think "cheap, wifi-enabled switches and sensors." The 8266 is a wifi-enabled chip (2.4GHz only) that runs at 80MHz, has around 80K of RAM, and typically between 512K and 4MB of external flash (for static code and data storage).

As a result, fitting a full SSL implementation is not an option. BearSSL is used in this context, and a limited number of algorithms are therefore made available from that library. There's only one RSA-based cipher that is included in Tasmota (IIRC).

Hence, I'm going to see if I can package Mosquitto so that one of the TLS ports can use the system-wide LE certs (which could, for example, be used by server-side home orchestration/automation packages like Home Assistant to talk to the MQTT server), but also allow a second port to use the self-signed certs (that can interact with low-spec embedded devices).

Long story short, the SSL limitations are more about the resource constraints of the systems involved (e.g. small IoT devices) than it is about out-of-date software. (mosquitto is an Eclipse foundation project, and effectively the reference standard for MQTT as a protocol, and BearSSL as a library implements more ciphers; MQTT is not the problem, Bear's limitations, somewhat. The limitations of the ESP8266-based IoT hardware is definitely a problem for doing EC ciphers.)

[girish]

@jadudm I re-opened https://git.cloudron.io/cloudron/box/-/issues/799 . Should be easy to add

[jadudm]

@girish For reference, these are the constraints if we're trying to support Tasmota-based IoT devices using BearSSL:

Your SSL/TLS server must support TLS 1.2 and the RSA_WITH_AES_128_GCM_SHA256 cipher - which is the case with the default Mosquitto configuration
 
The server certificate must have an RSA private key (max 2048 bits) and the certificate must be signed with RSA and SHA256 hash. This is the case with default LetsEncrypt certificates.

The docs are obviously dated, because I don't know that those LE defaults still apply.

[bartmathijssen]

@jadudm I'm looking for a mosquitto Cloudron package and found this topic. Do you have any plans to open-source the package you created?