New apps can't deploy - Will retry: queryNs ESERVFAIL

[True]

Dear Support,

I've moved to a new networking setup, and installing new items from appstore seems to be not working anymore.
Logs state something like

box:dns/waitfordns Attempt 1 failed. Will retry: queryNs ESERVFAIL mydomain.com
box:dns/waitfordns Attempt 2 failed. Will retry: queryNs ESERVFAIL mydomain.com

• The app has been created via cloudflare DNS API
• I see the new A record created in the registrar.
• The DNS is pointing to the same IP as the dashboard (no cloudflare proxy).
• The ports are correctly forwarded and other installed apps are working.
• Only the new application install seems to be affected.

Can you advise?

[nebulon]

Are you able to query the nameservers via SSH from that Cloudron?

host -t NS domain.com

[True]

Yes, so SSH from cloudron instance gives me

host -t NS mydomain.com
mydomain.com name server kristin.ns.cloudflare.com.
mydomain.com name server titan.ns.cloudflare.com.

Upon doing dig on my subdomain after they are live in the registrar for a day.

dig +trace subdomain.mydomain.com

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> +trace subdomain.mydomain.com
;; global options: +cmd
.                       86399   IN      NS      a.root-servers.net.
.                       86399   IN      NS      b.root-servers.net.
.                       86399   IN      NS      c.root-servers.net.
.                       86399   IN      NS      d.root-servers.net.
.                       86399   IN      NS      e.root-servers.net.
.                       86399   IN      NS      f.root-servers.net.
.                       86399   IN      NS      g.root-servers.net.
.                       86399   IN      NS      h.root-servers.net.
.                       86399   IN      NS      i.root-servers.net.
.                       86399   IN      NS      j.root-servers.net.
.                       86399   IN      NS      k.root-servers.net.
.                       86399   IN      NS      l.root-servers.net.
.                       86399   IN      NS      m.root-servers.net.
;; Received 239 bytes from 127.0.0.53#53(127.0.0.53) in 51 ms

;; UDP setup with 2001:7fe::53#53(2001:7fe::53) for subdomain.mydomain.com failed: network unreachable.
;; no servers could be reached

;; UDP setup with 2001:7fe::53#53(2001:7fe::53) for subdomain.mydomain.com failed: network unreachable.
;; no servers could be reached

;; UDP setup with 2001:7fe::53#53(2001:7fe::53) for subdomain.mydomain.com failed: network unreachable.
;; Received 45 bytes from 192.112.36.4#53(g.root-servers.net) in 27 ms

or

nslookup -type=NS subdomain.mydomain.com
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
*** Can't find subdomain.mydomain.com: No answer

Authoritative answers can be found from:
mydomain.com
        origin = kristin.ns.cloudflare.com

Further updates:

• After restarting the service a few times it managed to propagate. This is unusual...
• However even after proper propagation when I make even a simple adjustment for the service like enabling a port for the container over the GUI, the container restarts and "Will retry: queryNs ESERVFAIL" comes back again, and I have to restart the container a few more times until the comes back online properly. This is also very unusal.

Additional problems caused by the DNS propagation issue:

• The built-in mailbox DNS entries also got broken:

MX record 
- Hostname: @
- Record type: MX
- Expected value: 10 mail.mydomain.com.
- Current value: [not set]
DKIM record 
- Hostname: cloudron._domainkey
- Record type: TXT
- Expected value: v=DKIM1; t=s; p=[dkim_key_goes here]
- Current value: [not set]
SPF record 
- Hostname: @
- Record type: TXT
- Expected value: v=spf1 a:mail.mydomain.com ~all
- Current value: [not set]
DMARC record 
- Hostname: _dmarc
- Record type: TXT
- Expected value: v=DMARC1; p=reject; pct=100
- Current value: [not set]
PTR record 
- The PTR record is set by your VPS provider and not by your DNS provider. 
- Hostname: 45.56.67.78 {not my real IP}
- Record type: PTR
- Expected value: mail.mydomain.com
- Current value: [not set]

It was not an issue before. I've checked and in the registrar they are there still. But somehow they don't propagate.

[joseph]

@True can you also try host -t NS mydomain.com 127.0.0.150 ? If that doesn't work can you restart unbound from Services -> unbound and try?

[True]

RESOLVED:
It's likely that cloudflare was having some internal issue, but upon checking https://www.cloudflarestatus.com/ I was left more confused is there was any incident on their end in my region. It's buried under scheduled maintanaces and other stuff.

I still not sure if this was the correct resolution, but it started working after executing these steps.

sudo apt-get install -y systemd
sudo nano /etc/systemd/resolved.conf

Added the DNS of Cloudflare and google

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1
#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111
#cloudflare-d># Google:     8.8.8.8
#dns.google 8.8.4.4
#dns.google 2001:4860:4860::8888
#dns.google 2001:4860:4860::8>
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112
#dns.quad9.net 2620:fe::fe
#dns.quad9.net 2620:fe::>
DNS=1.1.1.1 8.8.8.8

sudo systemd-resolve --flush-caches
# didn't work
systemd-resolve --status 
# wasn't even running. So maybe the config change was irrelevant

Reboot and started working... and I am not sure how this is possible...

Anyway that for the help. This issue was plaguing me for a few days.