(Cannot read properties of null (reading 'sort'))

Summary

A full backup task aborts entirely when it reaches a ClickHouse-backed app (in my case Langfuse). The rsync syncer walks the app's data tree, but ClickHouse deletes a background-merge temp directory (data/clickhouse/store/*/tmp_merge_*) between enumeration and read. readTree gets null for that directory's contents and throws on .sort(), which propagates up and kills the whole task — not just the one app.

Impact

This is the important part: one racy app aborts the entire backup run. In the run below the task died at app 31 of 72, so all 41 remaining apps were silently left on the previous night's snapshot. The failure is timing-dependent (it only triggers if ClickHouse happens to be merging during the snapshot window), so backups appear intermittently broken with no config change on the user's side.

Environment

What happens

The same vanishing-temp-dir condition shows up twice in one snapshot. First it's caught harmlessly by the precondition du, which just warns and continues:

du: cannot access '/home/yellowtent/appsdata/5edfb175-…/data/clickhouse/store/f0a/f0a84db1-…/tmp_merge_202606_23202_23297_19': No such file or directory

Then it hits the rsync syncer, which does not tolerate it and crashes the task:

backupupload: upload completed. error: TypeError: Cannot read properties of null (reading 'sort')
    at readTree (file:///home/yellowtent/box/src/syncer.js:31:47)
    at traverse (file:///home/yellowtent/box/src/syncer.js:130:30)
    at traverse (file:///home/yellowtent/box/src/syncer.js:136:17)
    … (recursion)
    at Object.sync (file:///home/yellowtent/box/src/syncer.js:159:5)
    at sync (file:///home/yellowtent/box/src/backupformat/rsync.js:166:63)
    at Object.upload (file:///home/yellowtent/box/src/backupformat/rsync.js:336:18)
    at async Object.upload (file:///home/yellowtent/box/src/backuptask.js:101:37)

tasks: setCompleted - 4798: {"result":null,"error":{"message":"Cannot read properties of null (reading 'sort')","reason":"External Error"},"percent":100}

Root cause

readTree (box/src/syncer.js:31) reads a directory's entries and sorts them. When a subdirectory is removed between the initial find/enumeration and the per-directory read, the read returns null and null.sort() throws. The exception unwinds through traverse → sync → rsync.js → backuptask.js, so the whole task is marked failed rather than the single directory being skipped.

This is expected, normal ClickHouse behaviour, not app misbehaviour: ClickHouse continuously creates and renames/deletes tmp_merge_* (and tmp_insert_*, tmp_fetch_*) directories under store/<uuid>/ as background merges complete. Any tool that enumerates the store and then reads it will occasionally find a directory gone. ClickHouse upstream has hit and patched the equivalent race in their own tooling, e.g. https://github.com/ClickHouse/ClickHouse/pull/44874 — so accommodating it on the reader side is the standard approach.

Suggested fix

Make readTree resilient to a directory disappearing mid-traversal instead of propagating null:

• Guard against a null/undefined entry list before .sort().
• On ENOENT (or a null read) for a directory that vanished after enumeration, treat it as empty / skip it and continue the walk, rather than aborting.

This matches how du already behaves in the same task (warn and continue) and would make every ClickHouse-based package back up reliably.

Why this matters beyond Langfuse

Any package built on ClickHouse is affected the same way — Plausible, PostHog, SigNoz, Langfuse, etc. As more analytics apps adopt ClickHouse, more users will see intermittent, hard-to-diagnose full-backup aborts where the visible symptom (a stale backup on an unrelated app) is far from the actual cause.

EDIT:

Workarounds in the meantime

• Exclude the ClickHouse app from automatic backups so it can't abort the whole run, and back it up separately.
• Note: stopping the app and then running cloudron backup create --app <fqdn> does not work, because Cloudron does not back up a stopped app. A quiesced backup therefore has to be a filesystem-level snapshot taken outside the platform. Otherwise, because the failure is a timing collision, simply re-running the scheduled backup usually clears it.