Cloudron Forum

I spent several days trying to understand why backups kept failing on my Cloudron instance. After investigating the issue with Claude’s help, I identified and fixed two infrastructure issues on my own (detailed below for context), but Claude also highlighted a weakness on the Cloudron side: the lack of retries for transient network/DNS errors during S3 backup operations. I’m sharing this report written by Claude here for review. Title: Backup fails entirely on a single transient DNS/network error during S3 copy/list operations — no retry Version: Cloudron 9.2.0 (Ubuntu 22.04.5 LTS Linux 5.15.0-186-generic) Context : self-hosted server, backups to Scaleway Object Storage (S3-compatible), both tar and rsync backup formats affected Summary Over the past week I've had repeated backup failures, all sharing the same final symptom: Error: getaddrinfo EAI_AGAIN cloudron-rsync.s3.fr-par.scw.cloud This occurred at different stages of the backup pipeline across different incidents: during initial tar packing/upload, during copyInternal (S3-to-S3 copy for snapshot rotation), during checkPreconditions (mount point check), and during listDir (listing objects for rsync-based copy). The common thread in every case: a single DNS resolution hiccup during a long-running backup (which can involve tens of thousands of individual S3 operations) causes the entire backup task to crash and be marked as failed, with no retry attempted at the DNS/network level or at the individual-operation level. Root causes I found and fixed on my own infrastructure (not Cloudron bugs, but worth noting for context) A local mail service (Haraka, bundled with Cloudron) was stuck in an infinite retry loop performing reverse DNS (PTR) lookups on what turned out to be an IP in the reserved 240.0.0.0/4 range — never a real client IP, likely a bug in Haraka's own DNS-resolution/rDNS-check code generating a bad address internally. This flooded systemd-resolved with several lookups per second, 24/7, unrelated to backup timing. One of my two DHCP-provided DNS resolvers silently failed to respond at all over DNS-over-TCP (confirmed with dig +tcp @<ip>, consistent timeout), while working fine over UDP. When systemd-resolved fell back to TCP (e.g. due to the Haraka-induced UDP noise, or any other reason), it would sometimes pick this broken resolver and hang. Both issues are now fixed (mail service restart + config fix for #1, resolver removed from netplan config for #2), and backups now get much further before any failure — but a single transient DNS hiccup can still occasionally happen (as it can on any network), and it still takes down the whole backup task when it does. What would help Wrap listDir / copyDir / copyInternal (and any other network-dependent step in the backup pipeline) with a retry mechanism (e.g. 3 attempts with exponential backoff) for transient DNS/network errors like EAI_AGAIN, ETIMEDOUT, ECONNRESET. A single flaky lookup during a backup touching 80k+ files should not fail a multi-hour backup task that has otherwise fully succeeded. If a retry budget is exhausted, consider resuming/retrying at the file level rather than aborting the entire task — some backups got quite far (tens of thousands of files copied) before the single failure. More generally: the current behavior converts what should be a transient, self-healing network condition into a hard failure with reason: 'External Error' / 'Internal Error', requiring full backup task re-runs from scratch.

In the end I used S3 compatible option to set this up successfully. IDK why I could not get the Scaleway option to work. Thanks all for your help & input

Thank you @james Increasing the part size has fixed the issue.

Update: cloudron-support --troubleshoot returns: [WARN] netplan configuration is empty. this might be OK depending on your networking setup [FAIL] DNS is not resolving Further, /etc/netplan/ is empty. How do I generate the correct .yaml file to include there? Final Update: I was able to crate a netplan .yaml file by following these instructions.

Most of the time the scheduled backups now succeed, but not always. I'll keep an eye on the backups and make manual backups if the scheduled failed. Let's see if the changes of scaleway improve the situation. Thanks again for the suggestions and help.

I would not do this with backups for all the reasons @girish has mentioned (ie you'd have to restore the objects to standard before you'd be able to use them and this could take a while, not what you want when you need to quickly restore a backup), but what you could do if you really wanted to is to move older backups to Glacier automatically after a certain amount of time using automated lifecycle rules [image: 1676065407656-img_20230210_214239.jpg]

So is this then fixed by now with getting unbound back alive?

@scooke I tried "cleaning" but didn't make a difference. Will keep an eye on it.

it's solved, it was a problem with resolv.conf. The DNS servers of my scaleway instance are not responding anymore ...

@fortytwo we haven't done anything on your setup, since we don't even have access to your system. I suspect from the logs that there might have been some initial docker hickup on your server, which was resolved by the reboot.

@girish no, they're pretty useless. Their web UI S3 console is such crap it can't handle the chatty API requests and keeps timing out. Also I may be wrong that multiple directories are because of failures and restarts. It just looks like multiple changed apps per day get a new dir. So I am attempting other workarounds. Like creating a new bucket and just nuking the old one. rsync isn't great for object store backups as it makes a ton of small files. tgz isn't great as it's a lot of repeated information. We need something hybrid that is the best of both. Something like backing up to a local Minio much more quickly then doing an object to object store transfer offsite, which is much more efficient. This may also offer an opportunity to dedupe and further optimize.

@tx-hermit Ah, here's the explanation - https://serverfault.com/questions/1024770/ubuntu-20-04-time-sync-problems-and-possibly-incorrect-status-information . Uninstalling/disabling ntp service does the trick! I will make a fix for next release.

I think we won't change the minimum requirement tested for in the cloudron-setup script for now. It seem quite the edge case to have that little storage available, however it is good that we know there is a possible workaround for those cases.

@girish They are unusual I would say, yes. It's a file that is being updated on a daily basis so it has a file history of 30 different versions in NextCloud. They are .db.crypt12 files, which is a database backup of WhatsApp about 120MB in size.

@timmeh let us know if @NCKNE's workaround works or not.

This comment was very informative - https://bugs.launchpad.net/ubuntu/+source/ansible/+bug/1833013/comments/6 . I didn't realize you have to pass so many options to make it not ask questions.