Mail: spf=temperror on a large share of inbound mail, embedded DNS drops concurrent queries

[imc67]

I went looking for why a spoofed sender got delivered to an inbox here, and ended up somewhere I did not expect. On three of my Cloudrons, counting the Authentication-Results of everything delivered in the last 30 days, spf=temperror comes back on 14%, 21% and 31% of the messages that got an SPF verdict at all. For comparison, fail and softfail together never reach three messages per server. So temperror is by far the most common non-pass outcome, not a rare hiccup.

It is not the senders. The temperror messages are spread over 31 to 34 different sending domains per server, including icloud.com, overheid.nl, apple.com, github.com and proton.me. Those are not domains with broken SPF.

It is not a broken resolver either, at least not when you test it the obvious way. From inside the mail container, thirty lookups in a row all succeed in 23 to 40 ms, and the whole include chain resolves fine. It only falls apart when queries for different names run at the same time, which is exactly what evaluating an SPF record does:

# serial, and 30 parallel queries for the same name: fine
docker exec mail bash -c 'for i in $(seq 1 30); do dig +time=2 +tries=1 +short TXT kvk.nl >/dev/null 2>&1 && echo ok || echo fail; done' | sort | uniq -c

# 60 parallel queries across six different names
docker exec mail bash -c 'for d in kvk.nl icloud.com overheid.nl apple.com github.com amazon.nl; do for i in $(seq 1 10); do (dig +time=2 +tries=1 +short TXT $d >/dev/null 2>&1 && echo ok || echo fail) & done; done; wait' | sort | uniq -c

On both servers the last one returns 20 ok and 40 fail. resolv.conf in the container points at 127.0.0.11, Docker's embedded DNS.

Fair warning about that test: I used +tries=1 with a 2 second timeout, so no retries. With retries the loss is smaller. The 14 to 31 percent measured on real delivered mail is the part that matters, and that is independent of how I tested.

What makes this more than a cosmetic issue is that temperror carries no penalty, and rightly so, since it means "could not be checked" rather than "failed". The effect is that SPF and DMARC are quietly not enforced on a large slice of mail, including mail from senders who publish -all and p=quarantine and do everything right. That is how a spoof of such a domain reached an inbox here at score 2.3. The same resolver is also used for the DNSBL, URIBL, DKIM and rDNS lookups, and a dropped query there just means a rule silently does not fire.

Would it be possible to give the mail container a real resolver instead of the embedded one, or is there a reason it has to be 127.0.0.11?

[imc67]

@girish Small update with 24 hour numbers from the three servers, plus a couple of things I ruled out on my own side.

Over the last 24 hours, the share of delivered mail that got an SPF verdict of temperror was 48%, 34% and 18% on the three servers. Their own 30 day averages are 21%, 31% and 14.5%, so all three are sitting above their baseline at the moment.

Ruled out locally: conntrack is at roughly 0.5% of the table on all three with no table-full events in dmesg, and UDP InErrors, RcvbufErrors and SndbufErrors are all zero, both on the host and inside the mail container. Nothing is being dropped by the kernel, so it really does look like the embedded resolver just does not answer part of the concurrent queries.

One thing worth adding for balance, because it changes how urgent this looks. It mostly hits large legitimate senders with heavy SPF records. Of the 24 messages delivered with temperror in those 24 hours, 23 were from the likes of Dutch Government, GitHub, WeTransfer, etc. So the practical effect is not that spam suddenly floods in but the risk of doing so is higher. It is that SPF and DMARC quietly stop being enforced for a sizeable share of mail, including for the senders who do everything right.

[girish]

Great find. So the issue here was that some of those domains have large TXT records. So, the UDP DNS response is truncated and then a TCP DNS is attempted. However, the firewall is blocking DNS over TCP.

Can you try this one liner on your server - https://git.cloudron.io/platform/box/-/commit/d33febe056c5667fac53777575e8ec8a45027a1b ? You have to edit the file /home/yellowtent/platform/box/setup/start/cloudron-firewall.sh . Then systemctl restart cloudron-firewall.

Works consistently after that:

root@my:/home/yellowtent/box# docker exec mail bash -c 'for d in kvk.nl icloud.com overheid.nl apple.com github.com amazon.nl; do for i in $(seq 1 10); do (dig +time=2 +tries=1 +short TXT $d >/dev/null 2>&1 && echo ok || echo fail) & done; done; wait' | sort | uniq -c
     60 ok
root@my:/home/yellowtent/box# docker exec mail bash -c 'for d in kvk.nl icloud.com overheid.nl apple.com github.com amazon.nl; do for i in $(seq 1 10); do (dig +time=2 +tries=1 +short TXT $d >/dev/null 2>&1 && echo ok || echo fail) & done; done; wait' | sort | uniq -c
     60 ok
root@my:/home/yellowtent/box# docker exec mail bash -c 'for d in kvk.nl icloud.com overheid.nl apple.com github.com amazon.nl; do for i in $(seq 1 10); do (dig +time=2 +tries=1 +short TXT $d >/dev/null 2>&1 && echo ok || echo fail) & done; done; wait' | sort | uniq -c
     60 ok
root@my:/home/yellowtent/box# docker exec mail bash -c 'for d in kvk.nl icloud.com overheid.nl apple.com github.com amazon.nl; do for i in $(seq 1 10); do (dig +time=2 +tries=1 +short TXT $d >/dev/null 2>&1 && echo ok || echo fail) & done; done; wait' | sort | uniq -c
     60 ok

[imc67]

@girish don't know what is changed but I can't login to git.cloudron.io anymore (previously a separate account, later I remember it's changed to OIDC?) with/after OIDC it keeps asking for "Enter verification code" but the one for the Cloudron.io-account doesn't work (I can login there). Any idea?

update: strange, when I click your URL I have to login, if I surf to git. and your commit I can read it ....

[girish]

@imc67 can you try again?

(I have been playing around with all these visibility settings to reduce bots but making mistakes along the way. Finally, I have thrown anubis at the problem, and it seems to quite work well! Lets see may be we should integrate something like this into Cloudron).

[girish]

@imc67 i think it might just be a race, I just made it manually public again. I had a bug in my set visibility script.

[imc67]

yes, direct url is working again.

That was it! Applied the one line change on all three servers here and the 60 query test goes from 20/60 to 60/60 on each of them.

For reference, temperror was running at 48%, 34% and 18% of inbound mail over the last 24 hours on those three. I will report back once enough new mail has come in to see where it lands.

Thanks for digging into this so quickly.