SSH rate limit on ports 22/202/222 also throttles *outgoing* connections from apps (breaks Termix and other SSH clients)

[fbartels]

Cloudron version: 10.1.5

Summary

The brute-force rate limit on ports 22, 202 and 222 doesn't only apply to incoming connections. Apps hit it too when they open SSH connections to external servers. Once an app container opens more than 4 new connections to port 22 within 10 seconds, its SYN packets are dropped. Every retry refreshes the recent entry, so an app that keeps retrying never gets out.

Impact

I run Termix (web SSH client / server management) as a custom app. It runs periodic status checks and metrics collection for every configured host. With 6 hosts, two of them reached through a jump host, it opens enough SSH connections per round to hit the limit. After that, every outgoing SSH connection from the Termix container times out, and the app can't reach any server:

Error: Timed out while waiting for handshake
[ERROR] Jump host 1/1 connection failed
[ERROR] Stats collector connection failed

Inside the container, ss -tan shows connections to :22 stuck in SYN-SENT. The same destinations work fine from other app containers and from outside the server.

Steps to reproduce

From any app's web terminal (I used n8n), open a few quick connections to an external port 22 and compare with another port:

for i in 1 2 3 4 5 6 7; do
  timeout 3 bash -c "exec 3<>/dev/tcp/portquiz.net/22" 2>/dev/null && echo "$i open" || echo "$i FAIL"
done
# 1-4 open, 5-7 FAIL

for i in 1 2 3 4 5 6 7; do
  timeout 3 bash -c "exec 3<>/dev/tcp/portquiz.net/2222" 2>/dev/null && echo -n "ok " || echo -n "FAIL "
done
# ok ok ok ok ok ok ok

sleep 15; timeout 3 bash -c "exec 3<>/dev/tcp/portquiz.net/22" && echo open
# open again after 10+ s without new attempts

Destination	Termix container (stays throttled)	Other app containers
github.com:22	timeout	open
portquiz.net:22	timeout	open
portquiz.net:2222	open	open

Cause

In setup/start/cloudron-firewall.sh:

for port in 22 202 222; do
    ipxtables -A CLOUDRON_RATELIMIT -p tcp --dport ${port} -m state --state NEW -m recent --set --name "public-${port}"
    ipxtables -A CLOUDRON_RATELIMIT -p tcp --dport ${port} -m state --state NEW -m recent --update --name "public-${port}" --seconds 10 --hitcount 5 -j CLOUDRON_RATELIMIT_LOG
done
...
ipxtables -I FORWARD 1 -j CLOUDRON_RATELIMIT

CLOUDRON_RATELIMIT is hooked into FORWARD, which makes sense for incoming SSH that gets DNAT'ed to app containers (e.g. Gitea's SSH port). The rules have no source restriction, though, so traffic leaving containers towards the internet also matches. recent keys on the source address, so each app container gets its own counter, shared across all destinations.

Suggested fix

Exclude container-originated traffic from these rules, e.g.:

$iptables -A CLOUDRON_RATELIMIT -p tcp ! -s 172.18.0.0/16 --dport ${port} -m state --state NEW -m recent --set ...
$iptables -A CLOUDRON_RATELIMIT -p tcp ! -s 172.18.0.0/16 --dport ${port} -m state --state NEW -m recent --update ...

(plus the equivalent for the Docker IPv6 subnet in the ip6tables rules, since containers can also go out over IPv6). Alternatively, restrict the match to packets coming in on the external interface.

Current workaround

On the host:

iptables -I CLOUDRON_RATELIMIT 1 -s 172.18.0.0/16 ! -d 172.18.0.0/16 -p tcp --dport 22 -j RETURN

This is lost whenever Cloudron rebuilds its firewall rules, so it isn't a lasting fix.

cloudron-support --troubleshoot

═══ Host ═══
Vendor: netcup Product: KVM Server
Linux: 6.8.0-147-generic
Ubuntu: noble 24.04
Cloudron: 10.1.5
Execution environment: kvm
Processor: AMD EPYC 7702P 64-Core Processor
BIOS pc-i440fx-6.2  CPU @ 2.0GHz x 6
RAM: 16376312KB
Disk: /dev/vda3       256G

═══ System ═══
[OK]	Root disk usage is OK (58%)
[OK]	Memory usage is OK (77%)
[WARN]	Disk I/O on /home/yellowtent (vda, HDD): 180 MB/s write, iowait 30% — sustained iowait at or above 20% means the system is I/O bound. Investigate with 'iotop' or 'pidstat -d 2'.
[OK]	Clock is NTP-synchronized
[OK]	node version is correct
[OK]	IPv6 is enabled in kernel. Public IPv6 address detected
[OK]	docker is running
[OK]	docker version is correct

═══ Networking ═══
[OK]	no incompatible persistent-firewall packages installed
[OK]	DNS is resolving via systemd-resolved
[OK]	unbound is running
[OK]	IPv4 HTTPS to api.cloudron.io/api/v1/helper/public_ip
[OK]	IPv6 HTTPS to api.cloudron.io/api/v1/helper/public_ip
[OK]	IPv4 HTTPS to auth.docker.io/token
[OK]	IPv6 HTTPS to auth.docker.io/token
[OK]	IPv4 HTTPS to acme-v02.api.letsencrypt.org
[OK]	IPv6 HTTPS to acme-v02.api.letsencrypt.org
[OK]	netplan is good

═══ Cloudron Platform ═══
[OK]	MySQL is running
[OK]	nginx is running
[OK]	Dashboard cert is valid (33 days remaining, expires Nov 21 05:12:04 2026 GMT)
[OK]	dashboard is reachable via loopback
[OK]	No pending database migrations
[OK]	Service 'mysql' is running and healthy
[OK]	Service 'postgresql' is running and healthy
[OK]	Service 'mongodb' is lazy-stopped (no installed Cloudron app uses the 'mongodb' addon)
[OK]	Service 'mail' is running and healthy
[OK]	Service 'graphite' is running and healthy
[OK]	Service 'sftp' is running and healthy
[OK]	box v10.1.5 is running
[OK]	Dashboard is reachable via IPv4 (https://my.9wd.eu)
[OK]	Dashboard is reachable via IPv6 (https://my.9wd.eu)
[SKIP]	Domain expiry check — RDAP lookup for 9wd.eu failed (TLD may not support RDAP, or network unreachable)

======== Summary ========
PASS:  31
WARN:  1
FAIL:  0
SKIP:  1

Logs

/home/yellowtent/platformdata/logs/box.log: no errors; the drops happen in the kernel firewall, not in box.

The kernel log shows a drop by CLOUDRON_RATELIMIT_LOG of a SYN going out of an app container (IN=br-… OUT=eth0, SRC = the Termix container, DST = an external SSH server). The log rule is limited to 2 entries/min shared with all the inbound scanner drops, so only a few container drops make it into the log:

Oct 18 17:38:00 kermit kernel: IPTables RateLimit: IN=br-6a947611c07d OUT=eth0 PHYSIN=vethc6c0e7b SRC=172.18.17.61 DST=188.90.223.109 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=14768 DF PROTO=TCP SPT=59040 DPT=22 WINDOW=42340 RES=0x00 SYN URGP=0

/proc/net/xt_recent/public-22 lists container addresses next to the internet scanners (172.18.17.61 is the Termix container):

src=172.18.17.61 ttl: 64 last_seen: 4302953152 oldest_pkt: 7 4302952079, 4302952128, 4302952128, 4302953088, 4302953088, 4302953152, 4302953152, ...

The hook in FORWARD:

-A FORWARD -j CLOUDRON_RATELIMIT
-A CLOUDRON_RATELIMIT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name public-22 ...
-A CLOUDRON_RATELIMIT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 5 --name public-22 ...

[james]

Hello @fbartels
Thanks for the report. This will be fixed with the next release.