Cloudron Forum

@girish ?

While investigating hourly API timeouts on POST /api/v1/network/blocklist I found that every blocklist update leaves the server partly unprotected for several minutes. I measured it, read the code (10.1.5, unchanged in master), and wrote a drop-in replacement for setblocklist.sh that fixes it. I think this is also the root cause behind several older threads.

What happens

src/scripts/setblocklist.sh first runs ipset flush cloudron_blocklist and then re-adds every line with a separate ipset add process. With a list of ~110,000 entries that is ~110,000 process starts in a row. Until the loop is done, everything not yet re-added is simply not blocked — and the entries at the end of the file come back last.

Measured (ipset size sampled every 15 s during a normal update on a 10-core server):

10:59:51  89,645 entries
11:00:21   3,474   <- flushed
11:01:21  28,960
11:02:22  54,211
11:03:22  71,989
11:04:07  89,658   <- complete again

So roughly 4 minutes with on average half the list missing — on a quiet server. When the server is busy the loop takes longer than 10 minutes, the API call times out on the client side, and the hole lasts accordingly. With an hourly sync that is 7 % of the time or more with a partly open firewall.

Two more problems in the same code path

1. No serialization. network.setBlocklist() writes blocklist.txt and starts setblocklist.sh without any lock. If a second call arrives while the first script is still running (easy when the first one takes 10+ minutes, or when a client gives up and retries), the second script flushes the set in the middle of the first one's loop, while the first one keeps reading a file that has just been rewritten.
2. Partial apply on error. The script runs with set -e. If one ipset add fails, the script stops and the rest of the list is never loaded until the next update — while the database says the full list is active. That matches what was described in #10568 ("Blocked IP addresses in Cloudron but still able to access WordPress sites": "the block list never completely applied").

Fix: load into a temporary set and swap atomically

ipset has exactly the tools for this: build the new list in a temporary set with a single ipset restore (one process instead of 110,000), then ipset swap it with the live set. The swap is atomic, the iptables rules keep pointing at the same set name, and the live set is never empty. A flock serializes concurrent calls. If any entry is rejected, ipset restore fails as a whole and the old list stays fully active instead of a partial one.

Measured on the same server with the same blocklist.txt (114,330 entries, 24,670 of them IPv6), loaded into separate test sets: ipset restore 2.2 seconds, ipset swap 3 milliseconds — instead of 4 to 10+ minutes with a partly empty firewall. The resulting set had exactly the same number of entries as the live one.

Drop-in replacement for src/scripts/setblocklist.sh (same input file, same set names, same --check) :

#!/bin/bash

set -eu -o pipefail

if [[ ${EUID} -ne 0 ]]; then
    echo "This script should be run as root." > /dev/stderr
    exit 1
fi

if [[ $# == 1 && "$1" == "--check" ]]; then
    echo "OK"
    exit 0
fi

[[ "${BOX_ENV}" == "test" ]] && exit

exec 9>/run/cloudron-setblocklist.lock
flock 9 # one blocklist update at a time

user_firewall_json="/home/yellowtent/platformdata/firewall/blocklist.txt"

has_ipv6=0
ipset list -n cloudron_blocklist6 >/dev/null 2>&1 && has_ipv6=1 # kernel may not have ipv6

# same type and maxelem as in cloudron-firewall.sh, otherwise swap refuses
ipset destroy cloudron_blocklist_new 2>/dev/null || true
ipset create cloudron_blocklist_new hash:net maxelem 262144
if [[ ${has_ipv6} == 1 ]]; then
    ipset destroy cloudron_blocklist6_new 2>/dev/null || true
    ipset create cloudron_blocklist6_new hash:net family inet6 maxelem 262144
fi

if [[ -f "${user_firewall_json}" ]]; then
    # without the -n block, any last line without a new line won't be read it!
    while read -r line || [[ -n "$line" ]]; do
        [[ -z "${line}" ]] && continue # ignore empty lines
        [[ "$line" =~ ^#.*$ ]] && continue # ignore lines starting with #
        if [[ "$line" == *":"* ]]; then
            if [[ ${has_ipv6} == 1 ]]; then echo "add cloudron_blocklist6_new ${line}"; fi
        else
            echo "add cloudron_blocklist_new ${line}"
        fi
    done < "${user_firewall_json}" | ipset restore -exist # -exist ignores duplicates
fi

ipset swap cloudron_blocklist_new cloudron_blocklist
ipset destroy cloudron_blocklist_new
if [[ ${has_ipv6} == 1 ]]; then
    ipset swap cloudron_blocklist6_new cloudron_blocklist6
    ipset destroy cloudron_blocklist6_new
fi

Background — this has come up for years

• #1838 (2020): firewall restart takes 15–20 s — "the time the kernel is taking to add your blocklist to ipset".
• #3154 (2020): country lists cause response timeouts; fixed by removing the API timeout (req.clearTimeout()), but the loading itself stayed line by line.
• #3795 (2020–2024): many users keep blocklists up to date with scripts against the API ("1.5 minutes per API call").
• #6010 (2021): server crash after adding FireHOL level1.
• #10547 / #10568 (2023): "hash is full", and blocked IPs still getting through because the list never completely applied.
• #15536 (2026): body size limit raised to 20 MB — thanks! — which makes large lists possible, and makes the slow, non-atomic apply more visible.
• #15702 (2026): robi on large IP sets; the FSF/reaction approach he quotes is exactly this: export/restore instead of rule by rule, "nearly instantaneous".

A possible next step (not needed for the fix)

With an atomic apply that takes seconds, frequent full updates become cheap. An API to add or remove single entries (without resending the whole list) would be a nice follow-up for tools that react to attacks in real time, but the fix above is the important part: it closes the hole on every update, removes the timeouts and makes concurrent updates safe, without any change to the API, database or UI.

Happy to test a patched version on my servers.

We use the network blocklist (Network → Firewall → Blocklist, filled through /api/v1/network/blocklist) with a country block plus some public lists. While looking into why SPF checks on incoming mail ended in temperror for a few percent of all messages, we found that the blocklist does not only stop incoming connections: it also drops the replies to connections the server itself opens.

What happens

Cloudron 10.1.5. In setup/start/cloudron-firewall.sh the blocklist DROP is appended to the CLOUDRON chain before the rules that accept established traffic:

$iptables -t filter -A CLOUDRON -m set --match-set cloudron_blocklist src -j DROP            # line 36
ipxtables -t filter -A CLOUDRON -m state --state RELATED,ESTABLISHED -j ACCEPT                # line 44
ipxtables -t filter -A CLOUDRON -p udp --sport 53 -j ACCEPT                                    # line 114

and in DOCKER-USER (and FORWARD for IPv6) it is inserted at position 1 without any state check. So any packet from a blocklisted IP is dropped, including the answer to a request the server or an app sent itself.

The visible effect for us was DNS. The nameservers of .br (a.dns.br, b.dns.br, c.dns.br) fall inside a blocked range, so unbound can no longer resolve anything under .br:

dig +tries=1 +time=4 @127.0.0.150 TXT <any-domain>.gov.br    -> timed out
dig +tries=1 +time=4 @1.1.1.1     TXT <any-domain>.gov.br    -> NOERROR

Haraka then records spf=temperror and DMARC is not applied. That is harmless for Brazilian spam, but the same mechanism hits any domain whose nameservers (or SPF includes, DKIM keys, DNSBL zones) sit in a blocked range. We had two forged messages pretending to come from a well-known Dutch public register (whose domain has -all and a DMARC quarantine policy) land in an inbox with spf=temperror. Over the last months 2–12 % of incoming mail on one server had an SPF temperror; on the other two servers it was lower. The same applies to outbound connections from apps (for example delivering mail to an MX in a blocked range), although we did not measure that separately.

Workaround we use now

A rule above the blocklist DROP in all four chains, re-added every few minutes by a systemd timer (because the chains are rebuilt on platform updates and docker restarts):

iptables  -I CLOUDRON    1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT
iptables  -I DOCKER-USER 1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT
ip6tables -I CLOUDRON    1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT
ip6tables -I FORWARD     1 -m conntrack --ctstate ESTABLISHED,RELATED --ctdir REPLY -j ACCEPT

--ctdir REPLY only matches the reply direction of a connection that originated on the server, so anything a blocklisted IP starts itself (new or already running) still hits the DROP. After adding it, .br resolves again on all three servers and the blocklist DROP counters keep rising as before.

Suggestion

Put an ESTABLISHED,RELATED --ctdir REPLY accept (or simply the existing RELATED,ESTABLISHED accept) above the blocklist DROP in cloudron-firewall.sh, for both IPv4 and IPv6 and in both the host and the container chains. The blocklist is meant to keep unwanted visitors out; it should not cut the server off from DNS servers, MX hosts or APIs that happen to sit in a blocked range. The existing --sport 53 accept suggests DNS replies were meant to pass anyway.

Happy to test a fix.

@huehner thanks, I checked it: cloudron/php-base:8.5-20261021 indeed ships ghostscript 10.06.0~dfsg-3ubuntu1.1, which is the fixed version for CVE-2026-39919 on resolute (the 20260920 image had ubuntu1). So the PHP rebuilds take care of this one as a side effect. Agreed, the general timing question can wait until the PHP dust has settled.

Sharing what I did on my three servers while waiting for the patched images, in case it helps someone.

The leaked analyses (Hacker News, Sansec, Huntress) all point at session.upload_progress, and the circulating PoC sends a multipart POST with a PHP_SESSION_UPLOAD_PROGRESS field. None of my apps use that feature, so I switched it off everywhere. On Cloudron this is easy for most PHP apps, because /app/data/php.ini is linked as 99-cloudron.ini into both the Apache and the PHP-FPM conf.d:

; php2shell mitigation, remove after the patched image
session.upload_progress.enabled = Off

followed by apache2ctl -k graceful inside the container (and a USR2 to the php-fpm master for LimeSurvey, which uses FPM). That covered 47 installs (WordPress, Nextcloud, FreeScout, EspoCRM, Matomo, LimeSurvey, HumHub, Roundcube, LAMP) without any downtime. YOURLS has no /app/data/php.ini, but it reads /app/data/htaccess, so php_flag session.upload_progress.enabled off works there.

Two things I learned on the way: OIDC login doesn't protect a PHP app here, because the request is parsed by PHP before the app gets to the login check. proxyAuth does, because nginx stops the request before it reaches the app. And the WordPress maintenance mode doesn't help either, for the same reason.

There are also unverified claims of a second path through the general multipart parser, which the setting above wouldn't cover. Like many sites worldwide that went offline as a precaution today, I stopped 17 of my 53 PHP installs until the patched images are out: the ones nobody misses for a few hours (MainWP, Matomo, a staging site, personal tools) and all EspoCRM instances, because they hold personal data. The public websites, helpdesk and intranet stay up with the mitigation in place.

Looking forward to the patched php-base images. Is there already an idea of the timing?

@girish a concrete follow-up to the layer discussion above, because I ran into a case where the "shared base gets old" effect matters more than the usual medium CVEs.

Ubuntu fixed a critical Ghostscript bug on 21 September (CVE-2026-39919, CVSS 9.8, a crafted PDF with a JPEG 2000 image can run code; USN-8791-1, fixed in 10.06.0~dfsg-3ubuntu1.1 on 26.04). On my servers the fix is in some images and not in others, also in images that were built weeks after the fix:

Image (build date)	Ghostscript
Nextcloud (16 Oct), Paperless-ngx (19 Oct), LAMP	10.06.0~dfsg-3ubuntu1.1 (fixed)
WordPress Developer (7 Oct), FreeScout (17 Oct), EspoCRM, HumHub, LimeSurvey, Matomo	10.06.0~dfsg-3ubuntu1
Roundcube, YOURLS, AdGuard Home (24.04)	10.02.1~dfsg1-0ubuntu7.8 (fix is 7.9)

The difference seems to be the shared PHP layer (the sury PHP_SERIES step with Imagick), which in these images was built on 20 September, one day before the fix. Apps that install Ghostscript again in their own layer (Nextcloud does apt-get install ghostscript ...) get the current version; apps that rely on the shared layer keep the old one, even when the app itself is rebuilt later.

For most of these apps it doesn't matter much, because they never feed a PDF to ImageMagick. WordPress is the exception: the policy.xml in the image allows PDF (the restrictive lines are in commented-out examples), and WordPress renders a preview for every PDF uploaded to the media library. I checked it in the container: a PDF read through Imagick goes straight to Ghostscript. So anyone with upload rights on a site (author and up) can reach it. Not anonymous, but on a platform with many editors that's still a real path.

This fits nicely with what you said about sharing layers: refreshing that one shared PHP layer when Ubuntu ships a fix for something in it would fix all of these apps at once, without each app needing its own upgrade step. Is there a cadence or trigger for rebuilding those shared layers, or is it tied to app releases for now?

For anyone who wants to close the WordPress path in the meantime: returning an empty array from the fallback_intermediate_image_sizes filter stops WordPress from rendering PDF previews, so uploads never reach Ghostscript (you only lose the PDF thumbnail in the media library).

Confirming this on Cloudron 10.1.5 (mail 4.4.4, Dovecot 2.4.2). On our three servers the nightly learning run has crashed with the same write EPIPE / exited: mail-service on 23 of 27 nights since 11 October; two of the servers have not completed a single run in that period ("learning done" never logged). Every time it stops at the first candidate above 500 KB: last night at messages of 2.5 MB, 3.4 MB and 14.8 MB. Spam/ham training on IMAP moves (IMAPSieve) is not affected, only the nightly run.

Follow-up to topic 15801, where you turned off RDB persistence for the WordPress object cache (thanks again, the disk I/O dropped right away). We ran into a side effect of that change this morning, and it took nine WordPress apps down for almost three hours.

What happened

Cloudron 10.1.5. The box update ran at 00:00–00:10 UTC and went fine; all apps were healthy afterwards. At 03:31 UTC the host rebooted for a new kernel. From that moment nine of our ~20 WordPress apps (org.wordpress.unmanaged) restarted every ~13 seconds, 791 times for one of them, until we intervened at 06:13 UTC. The other WordPress apps on the same server started normally.

Why

Each redis container logs this on start:

==> Disabling RDB persistence
* Loading RDB produced by version 8.10.2
* RDB age 12401 seconds

So persistence is off, but redis still loads /home/yellowtent/platformdata/redis/<appid>/dump.rdb when it starts. That file is only written when the box takes a backup (the one loaded at 03:32 was 12401 s old, i.e. from the pre-update backup at ~00:05). Everything the app wrote to its cache between that backup and the reboot is gone, and the cache goes back three and a half hours, out of sync with the database.

For WordPress that broke the start script. start.sh runs with set -eu and does wp option update smtp_mailer_options <json> on every start. The cached smtp_mailer_options was the old value (12 keys; the SMTP Mailer plugin adds force_from_name/force_from_email), while the database already held the new value with the 10 keys start.sh writes. wp-cli compares against the cached value, sees a difference and calls update_option(); the UPDATE changes 0 rows because the database already has that value, so update_option() returns false, wp-cli prints Error: Could not update option 'smtp_mailer_options'., the script exits and the container is restarted. Same result on every start. The sites where cache and database happened to match started fine.

I checked the database side to rule it out: no read-only, full grants, CHECK TABLE OK, and the same UPDATE inside a transaction (rolled back) affects 1 row without error.

Workaround

wp --path=/app/data/public cache flush in each affected app container (during the seconds it is up). On the next start wp-cli reads the database value, reports Value passed for 'smtp_mailer_options' option is unchanged. and Apache starts. Afterwards a BGSAVE on each app's redis, because the dump from this morning's 06:00 backup still held the stale value and the next reboot would have repeated the loop.

Suggestions

• Don't load the dump at start for non-persistent redis. It's a cache, so starting empty is correct; the dump is only needed for a restore. Removing or renaming dump.rdb before redis starts (or having the backup write it outside the data dir and load it explicitly on restore) would fix it. As it stands, every host reboot rolls every WordPress cache back to the last backup.
• Make the WordPress start.sh tolerant here. A wp option update that has nothing to change shouldn't stop the container. For example || true on that line, or a wp cache flush early in start.sh (the cache is not persistent anyway, so flushing it at start costs nothing).

Happy to test a fix on our side.

Thanks @girish, that's clear. Good to know that an app password login doesn't send a mail, and bringing impersonate back makes it a lot easier to explain to other admins. The option to create an app password for another user is a nice addition too, I hadn't spotted that yet.

Thanks @girish and @james, that was quick, and it's good to see the fixes already in the box repo (quoting identifiers in the dump, patching an existing Taiga dump, skipping apps in error instead of blocking the platform, and the importer no longer hanging).

I ran the same check on our other two servers: per database, function argument names and type names against the words that are stricter in PG 18 than in 16 (json plus the new json_* keywords and merge_action). Same result as yours: only Taiga. Dawarich has a function json(geometry), but that belongs to PostGIS, so pg_dump leaves it out. We archived the Taiga on the second server, after renaming the parameter in the live database (json → j, Taiga calls the function positionally), so that backup restores cleanly on PG 18.

One question: does the Taiga patch also apply when restoring an app backup or archive that was taken on PostgreSQL 16? That's the case someone hits months from now when they bring an archived Taiga back.

We'll keep the other two servers on 10.0.5 until the patch release is out.

On one of my servers I added a second backup site of type filesystem (rsync format, local disk) next to the regular sshfs one. That local site fails now and then with an error like this, always inside a WP Rocket page cache folder:

ENOENT: no such file or directory, unlink '/var/backups/cloudron/snapshot/app_<id>/data/public/wp-content/cache/wp-rocket/<site>/evenement/<post-slug>/index-https.html_gzip'

It failed 2 out of about 11 runs. One of them was the backup before the platform update to 10.1.3 (the site is enabled for updates), so that update attempt was aborted and only went through an hour later. The sshfs site on the same server succeeded 150 out of 150 runs in the same period.

I traced it to src/syncer.js and could reproduce it with Cloudron's own code. Everything below is from 10.1.3; the file is identical in 10.0.5.

What goes wrong

The sync cache is written in traversal order: depth first, with the names in each folder sorted. For a folder romeo-julia with a sibling romeo-julia-2 that order is:

evenement/romeo-julia
evenement/romeo-julia/index-https.html
evenement/romeo-julia/index-https.html_gzip
evenement/romeo-julia-2
evenement/romeo-julia-2/index-https.html

traverse() and advanceCache() then compare cache entries and the current entry with plain string comparison (cache[curCacheIndex].path < entryPath on lines 99 and 124, cachePath > entryPath on line 131). In plain string order - (0x2D) and . (0x2E) sort before / (0x2F), so romeo-julia-2 < romeo-julia/index-https.html. That does not match the cache order.

When romeo-julia disappears (WP Rocket purges the cache of that page) and romeo-julia-2 still exists, the next run produces this delete queue:

removedir evenement/romeo-julia                     (missing)
remove    evenement/romeo-julia/index-https.html    (missing)
remove    evenement/romeo-julia/index-https.html_gzip   (missing)
removedir evenement/romeo-julia-2                   (missing)

and this add queue:

add       evenement/romeo-julia-2/index-https.html  (new)

So there are two problems:

1. The files inside the removed folder are queued again as separate remove operations, because lastRemovedDir is local to each advanceCache() call and the comparison stops halfway. backupformat/rsync.js processes the delete queue with async.eachLimit(…, concurrency). The removedir (rm -rf) therefore runs in parallel with the remove of a file inside that folder. storage/filesystem.js remove() does a statSync followed by an unlinkSync and throws on the error. If rm -rf deletes the file between the two calls, the whole backup fails with ENOENT. On sshfs, removeDir runs rm -rf over a separate ssh connection, which is much slower, so the individual removes almost always finish first. That is presumably why I only see the failure on the local target.
2. The sibling romeo-julia-2, which did not change, is removed from the snapshot and uploaded again as "new". That is not an error, but it is unnecessary work on every backup where this happens, on every target.

With a sibling without a dash (romeo-julia2) the delete queue is a single removedir, as expected.

This is a common pattern on WordPress: posts and events with the same title get the slugs name, name-2, name-3, and every caching plugin writes a folder per slug.

Reproduction

The script uses the real syncer.js, datalayout.js and storage/filesystem.js from /home/yellowtent/box and works only in /tmp/syncer-repro. Run it as yellowtent with the box environment:

sudo -u yellowtent env HOME=/home/yellowtent BOX_ENV=cloudron NODE_ENV=production /usr/local/node-24.19.0/bin/node /tmp/syncer-repro.mjs

import { createRequire } from 'node:module';
import fs from 'node:fs';
import path from 'node:path';

const BOX = '/home/yellowtent/box';
const async = createRequire(BOX + '/package.json')('async');
const { default: syncer } = await import(process.env.SYNCER || (BOX + '/src/syncer.js'));
const { default: DataLayout } = await import(BOX + '/src/datalayout.js');
const { default: storage } = await import(BOX + '/src/storage/filesystem.js');

const ROOT = '/tmp/syncer-repro', SRC = ROOT + '/src', DEST = ROOT + '/dest', CACHE = ROOT + '/test.sync.cache';
const reset = () => { fs.rmSync(ROOT, { recursive: true, force: true }); fs.mkdirSync(SRC, { recursive: true }); fs.mkdirSync(DEST, { recursive: true }); };
const write = (rel) => { for (const base of [ SRC, DEST ]) { const p = path.join(base, rel); fs.mkdirSync(path.dirname(p), { recursive: true }); fs.writeFileSync(p, 'x'); } };
const sync = () => syncer.sync(new DataLayout(SRC, []), CACHE);
async function syncAndFinalize() {
    const r = await sync();
    for (const a of r.addQueue) r.integrityMap.set(a.path, { size: 1, sha256: 'x' });
    await syncer.finalize(r.integrityMap, CACHE);
}

// 1. the queues
reset();
write('evenement/romeo-julia/index-https.html');
write('evenement/romeo-julia/index-https.html_gzip');
write('evenement/romeo-julia-2/index-https.html');
await syncAndFinalize();
fs.rmSync(path.join(SRC, 'evenement/romeo-julia'), { recursive: true });
const { delQueue, addQueue } = await sync();
console.log(delQueue, addQueue);

// 2. the failure: process the delete queue like backupformat/rsync.js does
const config = { _provider: 'filesystem', backupDir: DEST, prefix: '' };
let failed = 0;
for (let run = 0; run < 50; run++) {
    reset();
    for (let i = 0; i < 200; i++) { write(`e/p${i}/index-https.html`); write(`e/p${i}/index-https.html_gzip`); write(`e/p${i}-2/index-https.html`); }
    await syncAndFinalize();
    for (let i = 0; i < 200; i++) fs.rmSync(path.join(SRC, `e/p${i}`), { recursive: true });
    const { delQueue } = await sync();
    await async.eachLimit(delQueue, 10, async (c) => {
        if (c.operation === 'removedir') await storage.removeDir(config, {}, c.path, () => {});
        else if (c.operation === 'remove') await storage.remove(config, c.path);
    }).catch((e) => { failed++; if (failed === 1) console.log(e.message); });
}
console.log(`failed runs: ${failed}/50`);
fs.rmSync(ROOT, { recursive: true, force: true });

Result on 10.1.3: part 1 prints exactly the queues shown above. Part 2 failed in 46, 47 and 50 of the 50 runs (three attempts), with errors like ENOENT: no such file or directory, unlink '/tmp/syncer-repro/dest/e/p72/index-https.html'. With p${i}2 instead of p${i}-2 as the sibling name it fails 0 out of 50 times.

Suggested fix

Compare paths per component, so that the comparison follows the same order as the traversal:

// compare paths per component, so that the order matches the depth-first traversal order
// ("a" < "a/x" < "a-2"), instead of plain string order ("a" < "a-2" < "a/x")
function comparePaths(a, b) {
    const x = a.split('/'), y = b.split('/');
    for (let i = 0; i < Math.min(x.length, y.length); i++) {
        if (x[i] !== y[i]) return x[i] < y[i] ? -1 : 1;
    }
    return x.length - y.length;
}

and use it in the three places:

-        for (; curCacheIndex !== cache.length && (entryPath === '' || cache[curCacheIndex].path < entryPath); ++curCacheIndex) {
+        for (; curCacheIndex !== cache.length && (entryPath === '' || comparePaths(cache[curCacheIndex].path, entryPath) < 0); ++curCacheIndex) {
-            if (curCacheIndex !== cache.length && cache[curCacheIndex].path < entryPath) { // files disappeared. first advance cache as needed
+            if (curCacheIndex !== cache.length && comparePaths(cache[curCacheIndex].path, entryPath) < 0) { // files disappeared. first advance cache as needed
-            if (cachePath === null || cachePath > entryPath) { // new files appeared
+            if (cachePath === null || comparePaths(cachePath, entryPath) > 0) { // new files appeared

With this change, on a copy of syncer.js, part 1 gives a single removedir evenement/romeo-julia and an empty add queue, and part 2 fails 0 out of 50 times (both with and without the dash). Existing cache files are already in traversal order, so they don't need to change.

As extra hardening it might also make sense to treat ENOENT from unlinkSync in filesystem.js remove() as success, since the file being gone is the desired end state.

Last night one of our three Cloudrons auto-updated from 10.0.5 to 10.1.3. It ended with 24 of our 30 apps down for over five hours, from about 01:10 to 06:24 UTC, and the server did not recover by itself, not even after a restart of the box. The trigger was a single app (Taiga) whose database dump PostgreSQL 18 refuses. What turned that into a full outage is how the platform handles that failure. Below is the root cause, the workaround that got us back, and some suggestions, because this will hit every Cloudron with Taiga installed.

What happened

After the update, startPostgresql exports the PostgreSQL addon databases (Taiga and Pretix), removes the old data dir (rmaddondir.sh postgresql), starts the new cloudron/postgresql:7.0.0 (PostgreSQL 18) and imports the dumps again. The Taiga import fails:

ERROR:  syntax error at or near "json" at character 53
STATEMENT:  CREATE FUNCTION public.json_object_delete_keys(json json, VARIADIC keys_to_delete text[]) RETURNS json
restore: failed to restore database db45c50160… Error: process exited with code 3

The box marks Taiga as errored ("PostgreSQL restore failed. This may require more memory." — memory had nothing to do with it) and moves on to Pretix, and there it hangs: importing addon postgresql of app 58ebed9d… / Setting up postgresql, then nothing. The request never reached the addon (nothing in the postgresql log); the box's socket to 172.18.30.2:3000 had 3.7 MB stuck in the send queue, which looks like the rest of the Taiga dump that was still being piped when the addon gave up. onInfraReady never ran, so 23 apps stayed in pending_restart (plus Taiga in error) with "Waiting for platform to initialize".

The dashboard meanwhile showed the banner "Starting PostgreSQL service" while every service on the Services page was green. The first run (01:10) even ended in an uncaughtException (Unexpected status code 500 from pipework) and a box restart; that restart reproduced the hang exactly, and so did a manual docker restart postgresql + systemctl restart box this morning.

Root cause

Taiga creates its own helper function json_object_delete_keys with a parameter that is literally named json (of type json, and a jsonb variant). The dump header says:

-- Dumped from database version 16.14
-- Dumped by pg_dump version 16.14

For PostgreSQL 16 an unquoted json is a valid parameter name, so pg_dump 16 sees no reason to quote it. PostgreSQL 18 rejects it. I verified this in the new 7.0.0 container: the original statement gives the syntax error above, the same function with "json" json is created and works. The data in Taiga is irrelevant; the function exists from the moment Taiga is installed, so an unused Taiga breaks just as well as a busy one.

So the migration dumps with the old major version's pg_dump and restores into the new major version. That is the classic way to get bitten by keyword changes between majors; the usual practice is to dump with the new version's pg_dump, or at least use pg_dump --quote-all-identifiers, which exists exactly for dumps that go to a different major version.

Workaround that brought the server back

1. Copy the dumps somewhere safe first. After rmaddondir.sh the files in appsdata/<appid>/postgresqldump (and the pre-update backup) are the only copies of these databases.
2. Quote the parameter name in Taiga's dump (four lines: CREATE and ALTER for both variants), keep the owner:

F=/home/yellowtent/appsdata/<taiga-app-id>/postgresqldump
sed -i 's/json_object_delete_keys(json json,/json_object_delete_keys("json" json,/; s/json_object_delete_keys(json jsonb,/json_object_delete_keys("json" jsonb,/' "$F"
chown yellowtent:yellowtent "$F"
systemctl restart box

The box then sees "already exported addon postgresql in previous run", imports both databases in two seconds, recreates the redis containers and reports "platform is ready"; the apps start by themselves. Taiga and Pretix stayed in error though: their pending task fails with "Unknown install command in apptask:error", so both need a manual Repair. Pretix's data was complete afterwards (order count identical to the dump).

Suggestions

• Dump with the target version (or with --quote-all-identifiers) when migrating between PostgreSQL majors. This one is probably a one-line fix and would have prevented all of the above.
• One app's database must not be able to block the platform. If an import fails, mark that app as errored, close the connection properly and continue with the next app and with onInfraReady. Here a single unused app kept 23 other apps, among them a dozen WordPress sites, a ticket shop and two helpdesks, offline for hours.
• Put a timeout on the addon requests during the import, so a stuck socket ends in an error instead of waiting forever.
• Don't remove the old data dir before the import has succeeded, or keep the export in a place that survives a retry. As it is, the only way back is the dumps or a full restore.
• Make the state visible. "Starting PostgreSQL service" with all services green gives an admin nothing to go on at 7 in the morning. The import error and the app it concerns belong in the notification and on the Services page, and the error message should not suggest a memory problem when PostgreSQL reported a syntax error.
• Mention it in the release notes: a major PostgreSQL migration happens in 10.1, apps with their own SQL functions are at risk, and admins with Taiga should expect this.

I get that a major database migration on many installs is not trivial, and in general updates have been smooth for us. But this was an automatic night-time update, and the result was a server that stayed down until someone dug through the logs. Our other two Cloudrons stay on 10.0.5 for now. Can you confirm whether other appstore apps ship SQL functions or identifiers that PostgreSQL 18 no longer accepts unquoted, and whether a fix will land before the next release?

My first server updates to 10.1.3 tonight, and I noticed this in the changelog: "users: impersonate is removed. Sign in as a user with an app password for the dashboard or for apps".

I sometimes sign in as a user to help them or to check what they see. Two questions about the new way:

1. Does signing in with an app password send the user the "new login from a new location" email? If so, users will get an alarming mail every time an admin helps them, which may cause confusion (or worse, teach them to ignore those mails).
2. How does an admin get that app password: can an admin create one for another user, or does the user have to create it and share it?

Thanks!

Hi @nebulon , I've been using the new Mail app on one of our Cloudrons for a few weeks and it has quickly become my daily driver: sign in once with Cloudron and every mailbox I have access to is right there. Before rolling it out to more users I read through the code that runs in the container (1.1.1) with security in mind. Short version: it's very nicely done. A few notes and small suggestions below.

What I really like

• No master credential. The app has no password for the mail server at all: it uses the user's own OIDC token with a dedicated mailclient scope as the IMAP/SMTP/Sieve password, and the mail server checks per mailbox whether that token may access it. Even a bug in the app can't open a mailbox the user has no rights to. That is a better model than classic webmail, where the user types their mail password into the web app.
• Login through Cloudron OIDC with PKCE and state, so Cloudron's 2FA applies automatically and nobody ever types a mail password into the webmail.
• Mail HTML is rendered in a sandboxed iframe without allow-scripts, and the frame document gets its own CSP with script-src 'none'. Two locks, either one is enough.
• Remote images and other external content are blocked by default, with a per-sender/per-domain allowlist. Great for tracking pixels.
• Session cookie is HttpOnly and SameSite=Lax, and HSTS is on.

Small suggestions

1. Tokens in the session store end up in backups. Sessions are kept by tegel's file store in /app/data/.session.store, and each session file holds the access token, the refresh token (offline_access) and the mailclient token in plain text, valid for 14 rolling days. Since /app/data is backed up, anyone with access to a backup (unencrypted backups are not unusual) could use those tokens until they expire or are revoked. Sessions are disposable, so the simplest fix might be to keep the session store outside the backup (e.g. /run or a path excluded from backup) — after a restore users just sign in again. Encrypting the tokens at rest would also work.
2. The session cookie has no Secure flag. tegel sets secure: process.env.NODE_ENV === 'production', and NODE_ENV is not set in the container. HSTS covers most of it, but setting NODE_ENV=production (together with trust proxy, since TLS ends at nginx) would close it.
3. No CSP or frame protection on the app itself. The message frame has a strict CSP, but the main page sends no Content-Security-Policy and no X-Frame-Options/frame-ancestors, so the webmail can be framed by another site (clickjacking). A frame-ancestors 'self' would already help, and the app looks like it could live with a fairly strict CSP.
4. Minor, defence in depth: sanitizeHtml() in MessageView only strips external content; scripts and event handlers stay in the HTML and are neutralised by the sandbox + CSP. That is fine as long as those stay in place; a DOMPurify pass would be a third lock if the frame setup ever changes.

A question about 1.2.0

1.2.0 mentions "token exchange authentication" and needs Cloudron 10.1.1, so I can't try it yet. Does it change what is kept in the session (e.g. no refresh token any more, or a short-lived exchanged token only)? If so, point 1 may already be (partly) solved.

Thanks for building this, it's a really pleasant client to use already.

When deleting a mailbox in the dashboard with the "Also delete mails" checkbox ticked, the mailbox is removed but its mail stays on disk under /home/yellowtent/boxdata/mail/vmail/<address>. I noticed it on two servers (both Cloudron 10.0.5, cloudron/mail:4.3.9), for two different mailboxes.

The eventlog only shows mail.box.remove {"name":"info","domain":"..."}, and the directory's modification time is unchanged afterwards.

Looking at the mail container, the route passes the flag on correctly: routes/mail.js checks typeof req.body.deleteMails === 'boolean' and calls mailboxes.delMailbox(name, domain, { deleteMails: req.body.deleteMails }, auditSource). But in mailboxes.js the options argument is named _options and never used: delMailbox() deletes the database rows, the FTS index, the spamd dir and the sieve forwards, but nothing removes the maildir.

Expected: with the checkbox ticked, the maildir of that address is removed (or the checkbox is hidden if this is intentional). For now I remove the leftover directory manually.

It's not the open handles, as far as I can tell. I tried it with a node process that has nothing open at all (node -e "setInterval(()=>{},1000)" as PID 1 in a throwaway container from the Surfer image) and it behaves the same: it survives the SIGTERM and is killed after 10 s.

What happens is that node's default SIGTERM handler doesn't shut anything down itself, it resets the handler and raises the signal again with the default action. For a normal process the default action is "terminate", so node exits right away, open sockets or not. But for PID 1 the kernel ignores signals that only have the default action, so the re-raised signal is dropped and node just keeps running. In /proc/1/status you can see the handler in SigCgt before the SIGTERM and gone after it, while the process is still there.

That's also why tini makes a difference: node is then no longer PID 1, so the default action applies again and it exits in 0.1 s. Your idea works just as well and is the cleaner one: with a process.on('SIGTERM') handler in the app the signal is delivered even as PID 1, and the app can call close() on what matters before it exits.

I think I explained it poorly, because I'm not suggesting to kill anything earlier or to shorten the timeout. The 10 s grace period is fine and should stay.

My point is that the apps in the table never get the "please stop" in the first place. With the app as PID 1 and no handler installed, the kernel doesn't deliver the SIGTERM, so the app isn't using those 10 seconds to commit state: it just keeps running as if nothing happened and is then killed with SIGKILL. That is the forced kill you'd want to avoid, and for these apps it happens on every stop. With an init as PID 1 the signal does arrive and the app gets the same 10 seconds to finish, exactly the ctrl+c behaviour you describe.

I agree the gain is small, so I'm fine leaving it at this. I mainly wanted to make sure it's a known thing.

While auditing one of my own packages I noticed this line in journalctl -u docker, once for every update of that app:

Container failed to exit within 10s of signal 15 - using the force

My start.sh ended with exec gosu cloudron:cloudron python3 server.py, so the app itself was PID 1, and PID 1 only receives SIGTERM if it installs a handler for it. To be clear upfront: nothing broke. The update takes 10 seconds longer and requests that are in flight get cut off, and for an app with SQLite in WAL mode that is harmless. I'm only posting because it turned out not to be limited to my own packages.

I then looked at how long the stop takes for every app on my three servers (Cloudron 10.0.5), using the time between stopContainer and deleteContainer in the app task logs since 1 September:

App	PID 1	Stops that took 10 s or more
Surfer (4 installs)	node /app/code/server.js …	16 of 16
MiroTalk SFU (3 installs)	npm start	12 of 12
IP2Location	node /app/code/index.js	4 of 4
Release Bell	node /app/code/index.js	2 of 2

For comparison, the apps with apache2 or supervisord as PID 1 (WordPress, Nextcloud, FreeScout, EspoCRM) stop in under 4 seconds, and so does another app that runs plain node as PID 1 but apparently handles the signal itself. A throwaway container from the Surfer image shows the same thing in isolation: plain node as PID 1 takes 10.1 s for docker stop, with a process.on('SIGTERM') handler 0.1 s, and with docker run --init also 0.1 s.

In my own two packages I changed the last line of start.sh to exec /usr/bin/tini -- gosu cloudron:cloudron python3 server.py (tini is already in cloudron/base). The stop went from 10.2 s to 0.2 s and the line is gone from the docker log.

Is this something you'd rather handle per package, or is the hard kill simply fine for these apps? I can imagine it matters little for something like Surfer, I was mostly surprised to see it in the log.

Thanks! One data point from today's restore: the time was driven by the number of files, not their size. The ~100k small thumbnails took most of it, since every file costs a round trip over sshfs, so skipping unchanged small files would help just as much. A quick size + mtime check before hashing would probably keep the CPU side cheap too.

On Cloudron 10.0.5, the progress of a running backup doesn't move on the Backup Sites overview. The percentage and message stay where they were when the page was opened. After a page reload they jump to the current state. The backup itself runs fine, and the task API shows the progress advancing normally.

It looks like the overview loads the task once and doesn't poll afterwards, unlike the backup list and the app backup page. On the detail page of a single site it does poll, but there the polling seems to stop for good after a single failed request, so it can freeze as well.

Noticed it with a second (local filesystem) backup site next to the default one. Is this something that could go into a next release?