Bug: network blocklist is flushed and refilled line by line — the firewall is partly open for minutes on every update (fix included)

[imc67]

While investigating hourly API timeouts on POST /api/v1/network/blocklist I found that every blocklist update leaves the server partly unprotected for several minutes. I measured it, read the code (10.1.5, unchanged in master), and wrote a drop-in replacement for setblocklist.sh that fixes it. I think this is also the root cause behind several older threads.

What happens

src/scripts/setblocklist.sh first runs ipset flush cloudron_blocklist and then re-adds every line with a separate ipset add process. With a list of ~110,000 entries that is ~110,000 process starts in a row. Until the loop is done, everything not yet re-added is simply not blocked — and the entries at the end of the file come back last.

Measured (ipset size sampled every 15 s during a normal update on a 10-core server):

10:59:51  89,645 entries
11:00:21   3,474   <- flushed
11:01:21  28,960
11:02:22  54,211
11:03:22  71,989
11:04:07  89,658   <- complete again

So roughly 4 minutes with on average half the list missing — on a quiet server. When the server is busy the loop takes longer than 10 minutes, the API call times out on the client side, and the hole lasts accordingly. With an hourly sync that is 7 % of the time or more with a partly open firewall.

Two more problems in the same code path

1. No serialization. network.setBlocklist() writes blocklist.txt and starts setblocklist.sh without any lock. If a second call arrives while the first script is still running (easy when the first one takes 10+ minutes, or when a client gives up and retries), the second script flushes the set in the middle of the first one's loop, while the first one keeps reading a file that has just been rewritten.
2. Partial apply on error. The script runs with set -e. If one ipset add fails, the script stops and the rest of the list is never loaded until the next update — while the database says the full list is active. That matches what was described in #10568 ("Blocked IP addresses in Cloudron but still able to access WordPress sites": "the block list never completely applied").

Fix: load into a temporary set and swap atomically

ipset has exactly the tools for this: build the new list in a temporary set with a single ipset restore (one process instead of 110,000), then ipset swap it with the live set. The swap is atomic, the iptables rules keep pointing at the same set name, and the live set is never empty. A flock serializes concurrent calls. If any entry is rejected, ipset restore fails as a whole and the old list stays fully active instead of a partial one.

Measured on the same server with the same blocklist.txt (114,330 entries, 24,670 of them IPv6), loaded into separate test sets: ipset restore 2.2 seconds, ipset swap 3 milliseconds — instead of 4 to 10+ minutes with a partly empty firewall. The resulting set had exactly the same number of entries as the live one.

Drop-in replacement for src/scripts/setblocklist.sh (same input file, same set names, same --check) :

#!/bin/bash

set -eu -o pipefail

if [[ ${EUID} -ne 0 ]]; then
    echo "This script should be run as root." > /dev/stderr
    exit 1
fi

if [[ $# == 1 && "$1" == "--check" ]]; then
    echo "OK"
    exit 0
fi

[[ "${BOX_ENV}" == "test" ]] && exit

exec 9>/run/cloudron-setblocklist.lock
flock 9 # one blocklist update at a time

user_firewall_json="/home/yellowtent/platformdata/firewall/blocklist.txt"

has_ipv6=0
ipset list -n cloudron_blocklist6 >/dev/null 2>&1 && has_ipv6=1 # kernel may not have ipv6

# same type and maxelem as in cloudron-firewall.sh, otherwise swap refuses
ipset destroy cloudron_blocklist_new 2>/dev/null || true
ipset create cloudron_blocklist_new hash:net maxelem 262144
if [[ ${has_ipv6} == 1 ]]; then
    ipset destroy cloudron_blocklist6_new 2>/dev/null || true
    ipset create cloudron_blocklist6_new hash:net family inet6 maxelem 262144
fi

if [[ -f "${user_firewall_json}" ]]; then
    # without the -n block, any last line without a new line won't be read it!
    while read -r line || [[ -n "$line" ]]; do
        [[ -z "${line}" ]] && continue # ignore empty lines
        [[ "$line" =~ ^#.*$ ]] && continue # ignore lines starting with #
        if [[ "$line" == *":"* ]]; then
            if [[ ${has_ipv6} == 1 ]]; then echo "add cloudron_blocklist6_new ${line}"; fi
        else
            echo "add cloudron_blocklist_new ${line}"
        fi
    done < "${user_firewall_json}" | ipset restore -exist # -exist ignores duplicates
fi

ipset swap cloudron_blocklist_new cloudron_blocklist
ipset destroy cloudron_blocklist_new
if [[ ${has_ipv6} == 1 ]]; then
    ipset swap cloudron_blocklist6_new cloudron_blocklist6
    ipset destroy cloudron_blocklist6_new
fi

Background — this has come up for years

• #1838 (2020): firewall restart takes 15–20 s — "the time the kernel is taking to add your blocklist to ipset".
• #3154 (2020): country lists cause response timeouts; fixed by removing the API timeout (req.clearTimeout()), but the loading itself stayed line by line.
• #3795 (2020–2024): many users keep blocklists up to date with scripts against the API ("1.5 minutes per API call").
• #6010 (2021): server crash after adding FireHOL level1.
• #10547 / #10568 (2023): "hash is full", and blocked IPs still getting through because the list never completely applied.
• #15536 (2026): body size limit raised to 20 MB — thanks! — which makes large lists possible, and makes the slow, non-atomic apply more visible.
• #15702 (2026): robi on large IP sets; the FSF/reaction approach he quotes is exactly this: export/restore instead of rule by rule, "nearly instantaneous".

A possible next step (not needed for the fix)

With an atomic apply that takes seconds, frequent full updates become cheap. An API to add or remove single entries (without resending the whole list) would be a nice follow-up for tools that react to attacks in real time, but the fix above is the important part: it closes the hole on every update, removes the timeouts and makes concurrent updates safe, without any change to the API, database or UI.

Happy to test a patched version on my servers.

[girish]

Thanks for the report. I have fixed this now.