I spent several days trying to understand why backups kept failing on my Cloudron instance. After investigating the issue with Claude’s help, I identified and fixed two infrastructure issues on my own (detailed below for context), but Claude also highlighted a weakness on the Cloudron side: the lack of retries for transient network/DNS errors during S3 backup operations. I’m sharing this report written by Claude here for review.
Title: Backup fails entirely on a single transient DNS/network error during S3 copy/list operations — no retry
Version: Cloudron 9.2.0 (Ubuntu 22.04.5 LTS Linux 5.15.0-186-generic)
Context : self-hosted server, backups to Scaleway Object Storage (S3-compatible), both tar and rsync backup formats affected
Summary
Over the past week I've had repeated backup failures, all sharing the same final symptom:
Error: getaddrinfo EAI_AGAIN cloudron-rsync.s3.fr-par.scw.cloud
This occurred at different stages of the backup pipeline across different incidents: during initial tar packing/upload, during copyInternal (S3-to-S3 copy for snapshot rotation), during checkPreconditions (mount point check), and during listDir (listing objects for rsync-based copy). The common thread in every case: a single DNS resolution hiccup during a long-running backup (which can involve tens of thousands of individual S3 operations) causes the entire backup task to crash and be marked as failed, with no retry attempted at the DNS/network level or at the individual-operation level.
Root causes I found and fixed on my own infrastructure (not Cloudron bugs, but worth noting for context)
A local mail service (Haraka, bundled with Cloudron) was stuck in an infinite retry loop performing reverse DNS (PTR) lookups on what turned out to be an IP in the reserved 240.0.0.0/4 range — never a real client IP, likely a bug in Haraka's own DNS-resolution/rDNS-check code generating a bad address internally. This flooded systemd-resolved with several lookups per second, 24/7, unrelated to backup timing.
One of my two DHCP-provided DNS resolvers silently failed to respond at all over DNS-over-TCP (confirmed with dig +tcp @<ip>, consistent timeout), while working fine over UDP. When systemd-resolved fell back to TCP (e.g. due to the Haraka-induced UDP noise, or any other reason), it would sometimes pick this broken resolver and hang.
Both issues are now fixed (mail service restart + config fix for #1, resolver removed from netplan config for #2), and backups now get much further before any failure — but a single transient DNS hiccup can still occasionally happen (as it can on any network), and it still takes down the whole backup task when it does.
What would help
Wrap listDir / copyDir / copyInternal (and any other network-dependent step in the backup pipeline) with a retry mechanism (e.g. 3 attempts with exponential backoff) for transient DNS/network errors like EAI_AGAIN, ETIMEDOUT, ECONNRESET. A single flaky lookup during a backup touching 80k+ files should not fail a multi-hour backup task that has otherwise fully succeeded.
If a retry budget is exhausted, consider resuming/retrying at the file level rather than aborting the entire task — some backups got quite far (tens of thousands of files copied) before the single failure.
More generally: the current behavior converts what should be a transient, self-healing network condition into a hard failure with reason: 'External Error' / 'Internal Error', requiring full backup task re-runs from scratch.
