Bug: Network blocklist also drops replies to the server's own outbound connections (DNS, SPF temperror)

[imc67]

We use the network blocklist (Network → Firewall → Blocklist, filled through /api/v1/network/blocklist) with a country block plus some public lists. While looking into why SPF checks on incoming mail ended in temperror for a few percent of all messages, we found that the blocklist does not only stop incoming connections: it also drops the replies to connections the server itself opens.

What happens

Cloudron 10.1.5. In setup/start/cloudron-firewall.sh the blocklist DROP is appended to the CLOUDRON chain before the rules that accept established traffic:

$iptables -t filter -A CLOUDRON -m set --match-set cloudron_blocklist src -j DROP            # line 36
ipxtables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT                # line 44
ipxtables -t filter -A CLOUDRON -p udp --sport 53 -j ACCEPT                                    # line 114

and in DOCKER-USER (and FORWARD for IPv6) it is inserted at position 1 without any state check. So any packet from a blocklisted IP is dropped, including the answer to a request the server or an app sent itself.

The visible effect for us was DNS. The nameservers of .br (a.dns.br, b.dns.br, c.dns.br) fall inside a blocked range, so unbound can no longer resolve anything under .br:

dig +tries=1 +time=4 @127.0.0.150 TXT <any-domain>.gov.br    -> timed out
dig +tries=1 +time=4 @1.1.1.1     TXT <any-domain>.gov.br    -> NOERROR

Haraka then records spf=temperror and DMARC is not applied. That is harmless for Brazilian spam, but the same mechanism hits any domain whose nameservers (or SPF includes, DKIM keys, DNSBL zones) sit in a blocked range. We had two forged messages pretending to come from a well-known Dutch public register (whose domain has -all and a DMARC quarantine policy) land in an inbox with spf=temperror. Over the last months 2–12 % of incoming mail on one server had an SPF temperror; on the other two servers it was lower. The same applies to outbound connections from apps (for example delivering mail to an MX in a blocked range), although we did not measure that separately.

Workaround we use now

A rule above the blocklist DROP in all four chains, re-added every few minutes by a systemd timer (because the chains are rebuilt on platform updates and docker restarts):

iptables  -I CLOUDRON    1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT
iptables  -I DOCKER-USER 1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT
ip6tables -I CLOUDRON    1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT
ip6tables -I FORWARD     1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT

--ctdir REPLY only matches the reply direction of a connection that originated on the server, so anything a blocklisted IP starts itself (new or already running) still hits the DROP. After adding it, .br resolves again on all three servers and the blocklist DROP counters keep rising as before.

Suggestion

Put an ESTABLISHED,RELATED --ctdir REPLY accept (or simply the existing RELATED,ESTABLISHED accept) above the blocklist DROP in cloudron-firewall.sh, for both IPv4 and IPv6 and in both the host and the container chains. The blocklist is meant to keep unwanted visitors out; it should not cut the server off from DNS servers, MX hosts or APIs that happen to sit in a blocked range. The existing --sport 53 accept suggests DNS replies were meant to pass anyway.

Happy to test a fix.

[imc67]

@girish ?

[nebulon]

we will catch up on things, but have to recover a bit from the recent incidents which were quite a lot of work

[girish]

OK, I have changed the behavior of the firewall. The blocked list applies primarily to incoming connections and outbound is still allowed (it seems ufw, fail2ban have similar behavior). Fixed for next release.