Backup task crashes when a ClickHouse app deletes a temp merge dir mid-snapshot

[LoudLemur]

Backup task crashes when a ClickHouse app deletes a temp merge dir mid-snapshot

(Cannot read properties of null (reading 'sort'))

Summary

A full backup task aborts entirely when it reaches a ClickHouse-backed app (in my case Langfuse). The rsync syncer walks the app's data tree, but ClickHouse deletes a background-merge temp directory (data/clickhouse/store/*/tmp_merge_*) between enumeration and read. readTree gets null for that directory's contents and throws on .sort(), which propagates up and kills the whole task — not just the one app.

Impact

This is the important part: one racy app aborts the entire backup run. In the run below the task died at app 31 of 72, so all 41 remaining apps were silently left on the previous night's snapshot. The failure is timing-dependent (it only triggers if ClickHouse happens to be merging during the snapshot window), so backups appear intermittently broken with no config change on the user's side.

Environment

Cloudron version	9.2.0
OS	Ubuntu 24.04.2 LTS
Kernel	6.8.0-124-generic
Backup format	rsync, encrypted
Backup storage	Hetzner Storage Box over SSH (port 23)
Affected app	Langfuse (ClickHouse)

What happens

The same vanishing-temp-dir condition shows up twice in one snapshot. First it's caught harmlessly by the precondition du, which just warns and continues:

du: cannot access '/home/yellowtent/appsdata/5edfb175-…/data/clickhouse/store/f0a/f0a84db1-…/tmp_merge_202606_23202_23297_19': No such file or directory

Then it hits the rsync syncer, which does not tolerate it and crashes the task:

backupupload: upload completed. error: TypeError: Cannot read properties of null (reading 'sort')
    at readTree (file:///home/yellowtent/box/src/syncer.js:31:47)
    at traverse (file:///home/yellowtent/box/src/syncer.js:130:30)
    at traverse (file:///home/yellowtent/box/src/syncer.js:136:17)
    … (recursion)
    at Object.sync (file:///home/yellowtent/box/src/syncer.js:159:5)
    at sync (file:///home/yellowtent/box/src/backupformat/rsync.js:166:63)
    at Object.upload (file:///home/yellowtent/box/src/backupformat/rsync.js:336:18)
    at async Object.upload (file:///home/yellowtent/box/src/backuptask.js:101:37)

tasks: setCompleted - 4798: {"result":null,"error":{"message":"Cannot read properties of null (reading 'sort')","reason":"External Error"},"percent":100}

Root cause

readTree (box/src/syncer.js:31) reads a directory's entries and sorts them. When a subdirectory is removed between the initial find/enumeration and the per-directory read, the read returns null and null.sort() throws. The exception unwinds through traverse → sync → rsync.js → backuptask.js, so the whole task is marked failed rather than the single directory being skipped.

This is expected, normal ClickHouse behaviour, not app misbehaviour: ClickHouse continuously creates and renames/deletes tmp_merge_* (and tmp_insert_*, tmp_fetch_*) directories under store/<uuid>/ as background merges complete. Any tool that enumerates the store and then reads it will occasionally find a directory gone. ClickHouse upstream has hit and patched the equivalent race in their own tooling, e.g. https://github.com/ClickHouse/ClickHouse/pull/44874 — so accommodating it on the reader side is the standard approach.

Suggested fix

Make readTree resilient to a directory disappearing mid-traversal instead of propagating null:

• Guard against a null/undefined entry list before .sort().
• On ENOENT (or a null read) for a directory that vanished after enumeration, treat it as empty / skip it and continue the walk, rather than aborting.

This matches how du already behaves in the same task (warn and continue) and would make every ClickHouse-based package back up reliably.

Why this matters beyond Langfuse

Any package built on ClickHouse is affected the same way — Plausible, PostHog, SigNoz, Langfuse, etc. As more analytics apps adopt ClickHouse, more users will see intermittent, hard-to-diagnose full-backup aborts where the visible symptom (a stale backup on an unrelated app) is far from the actual cause.

EDIT:

Workarounds in the meantime

• Exclude the ClickHouse app from automatic backups so it can't abort the whole run, and back it up separately.
• Note: stopping the app and then running cloudron backup create --app <fqdn> does not work, because Cloudron does not back up a stopped app. A quiesced backup therefore has to be a filesystem-level snapshot taken outside the platform. Otherwise, because the failure is a timing collision, simply re-running the scheduled backup usually clears it.

[LoudLemur]

Following up on our above report, confirming this still reproduces on Cloudron 9.2.0, with the exact spot in the source in case it helps.

We hit it on a real app backup just now:

App backup error: Backup failed: Cannot read properties of null (reading 'sort')

It reproduced on the second backup attempt under active ClickHouse merge load, so the race window isn't narrow.

9.2.0 does add a guard, but it sits after the .sort(). In box/src/syncer.js, readTree is:

const names = safe.fs.readdirSync(dirPath).sort();   // line 31
if (!names) return [];                                // line 32

@cloudron/safetydance's readdirSync returns null on a vanished directory (the ENOENT when a ClickHouse tmp_merge_* part is renamed/removed mid-snapshot). So line 31 evaluates null.sort() and throws before the line-32 if (!names) guard can run — the guard is dead code for this crash, and the whole-server backup run still aborts.

The fix is to null-check before .sort():

const names = safe.fs.readdirSync(dirPath);
if (!names) return [];
names.sort();

(or (safe.fs.readdirSync(dirPath) || []).sort()). The same file's readCache (around lines 22–23) already does it in this order — assign, null-check, then use — so this just makes readTree consistent with the existing pattern in the file.

Happy to test a patch against a live ClickHouse-bundling app under merge load.

[girish]

@loudlemur thanks for the report. Fixed in https://git.cloudron.io/platform/box/-/commit/33f3ca39996ad5b3af2b5dd2320d1552dd8952c0

[LoudLemur]

@girish thank you, that was a quick turnaround and much appreciated.

Adding one more data point for anyone who finds this before the fix reaches a release. We hit the same crash again on 2026-07-06, still on 9.2.0 (a build from before commit 33f3ca3), so it behaves exactly as expected until the patch ships:

tasks: setCompleted - 4900: {"result":null,"error":{"message":"Cannot read properties of null (reading 'sort')","reason":"External Error"},"percent":100}

Same app (Langfuse), same place: the task died at app 31 of 73, so the roughly 40 apps after it were left on the previous snapshot again. The du precondition logged the vanished directories first, this time a delete_tmp_tmp_merge_* variant alongside the usual tmp_merge_* ones, and then readTree threw during the upload.

One question: which release will carry commit 33f3ca3, so we know when to expect it? We have a reliable reproducer here under merge load and are glad to confirm the fix once it lands.

In the meantime we have excluded the ClickHouse apps from the automatic schedule so a single mid-merge app cannot abort the whole run, and we re-run on the rare miss, which clears it as noted above.

[girish]

@LoudLemur will be part of Cloudron 10 .

[LoudLemur]

@girish thank you, and thanks for confirming the fix ships in Cloudron 10.

Adding one data point for anyone who finds this while still on 9.x: we hit the same crash again on 2026-07-06, on 9.2.0 (a build from before commit 33f3ca3), so it recurs exactly as expected until the v10 upgrade:

tasks: setCompleted - 4900: {"result":null,"error":{"message":"Cannot read properties of null (reading 'sort')","reason":"External Error"},"percent":100}

Same app (Langfuse), same place: the task died at app 31 of 73, so the roughly 40 apps after it were left on the previous snapshot again. The du precondition logged the vanished directories first, this time a delete_tmp_tmp_merge_* variant alongside the usual tmp_merge_* ones, and then readTree threw during the upload.

Until we are on 10 we have excluded the ClickHouse apps from the automatic schedule so a single mid-merge app cannot abort the whole run, and we re-run on the rare miss, which clears it. Glad to confirm the fix here once we upgrade, we have a reliable reproducer under merge load.