About backups to objects storage and DNS requests

[mpevhgnuragistes]

Hello,

Using backup to some objects storage I've encountered multiple issues with errors like:

FS file The operation was aborted
or…
Error: getaddrinfo EAI_AGAIN

When a backup is processed, I see every second a DNS query about the storage destination.

Regarding the DNS server used, I suspect that those queries may trigger some kind of delayed answer. And if this happens, the backup task fails miserably even if there is no changes about the answer to the DNS query.

I understand that those DNS records have to be on the bleeding edge of availability but I prefer to have a working backup instead of an error, especially when the answer still the same, which is more often the case.

If I want to keep the DNS server I used to use, Is there something to do about that ?

[james]

Hello @mpevhgnuragistes and welcome to the Cloudron Forum

@mpevhgnuragistes said:

some objects storage

What is this object storage you are using?

@mpevhgnuragistes said:

Regarding the DNS server used

To analyse this issue the DNS used is important. Can you please share this information?

[mpevhgnuragistes]

Aloa,

• contabo-objectstorage with the eu2.contabostorage.com destination.
• FDN's servers 80.67.169.12, 2001:910:800::12, 80.67.169.40, 2001:910:800::40

I tried tar and rsync for the format.

[james]

Hello @mpevhgnuragistes

I had to look up FDN and found FortiGuard Distribution Network, is this correct?

[joseph]

Can you check cloudron-support --troubleshoot ?

Does this happens only with Contabo? Have you tried any other storage provider by any chance?

[mpevhgnuragistes]

@james said:

I had to look up FDN and found FortiGuard Distribution Network , is this correct?

Nope… $ whois 80.67.169.12 tells you that it's French Data Network and they are public resolvers like those you can find on the list of dns servers provided by ddgo.

[mpevhgnuragistes]

@joseph off course, cloudron-support sees nothing here because the issue is not « general ». It's really when a backup is on it's way and for some obscure reason it stops with one or the other errors provided in the original post of this thread.

I tried different kind of backup destinations, CIFS and sshfs to Hetzner Storage Boxes (not Hetzner Storage Objects), both with rsync or tar, and I had backup errors to from time to time, but I did not really dig on the logs about the errors.

Then because the cloudron vm I work with is hosted on Contabo, I tried Contabo Object Storage but I kept having backup issues regularly.

So I looked in the logs and found those errors but they had no real sens to me because why does it works sometimes and sometimes not ?

So I had the felling that the problem was elsewhere (meaning not really with the backup destination kind of storage) but maybe the network because it's the common ground for all the things I tried.

Because it's a small server with not so much network traffic. But with the backups on course, there is a little bit more and I think that using an external / public DNS resolver from inside Contabo's network may reach some kind of treshold from Contabo (where the server is) or FDN (resolvers for the server) firewall or whatever.

And when there is a delay to the DNS request « what is the ip for eu2.contabostorage.com » or maybe in the past the hetzner storage box the backup just stop with those errors, even if the ip adresse of the backup destination did not change.

May be it's too close to i suspect network/dns delays to trigger those errors instead of deeper diagnosis within the backup stack.

Whatever, the vm uses the contabo's resolvers to access the contabo objets storage and for the moment it seems to work, but it's only since friday.

I just prefer to use other dns resolvers then the ones provided by the hosting provider or the internet service provider.

Voila.

[james]

Hello @mpevhgnuragistes

This all hints toward some obscure DNS issue with the set DNS server.
I'd assume when using a common DNS Server like quad9, Cloudflare, Google etc. this issue would not appear.

[mpevhgnuragistes]

@james said:

I'd assume when using a common DNS Server like quad9, Cloudflare, Google etc. this issue would not appear.

The real issue here is the backup stack which miserably end backup tasks when there is no real errors, but maybe a badly handled timeout (what is a timeout today, 1ms, 10ms, 100ms, 1s) of some sort.

Using common private and big tech dns-to-rule-them-all to solve some kind of issue is not my way of thinking, sorry.

But whatever, thanks for your time.

[james]

Hello @mpevhgnuragistes

I am not sure if the timeout is actual real reason.
We'd have to debug this in depth, because it could be that the DNS server is rate limiting and denying the request and then a timeout error is raised.
Which then would be misleading since a timeout is displayed but in reality it is the DNS server causing the issue by denying the request.

But like I wrote, this is speculation and would need in depth debugging with such a DNS server.
It is good that you have raised this issue so we can be on the lookout if this issue arises again so we can have a deeper look at what is going on.

[joseph]

@mpevhgnuragistes said:

Using common private and big tech dns-to-rule-them-all to solve some kind of issue is not my way of thinking, sorry.

you could also use the local unbound dns which is running in 127.0.0.150 on the server. just to see if that solves anything.