queryNs ESERVFAIL for new apps

[OrezBodajoy]

I am not able to deploy new apps, I am on Ubuntu 22.

host -t NS mydomain.com works fine.
host -t NS mydomain.com 127.0.0.150 does not work (timeout or not reachable IIRC).

The unbound service says it's active and running, restarting it does not fix the issue.
unbound-control says control is not enabled, and the service is stopped.

I also tried removing the forward everything unbound config and restarting.

Do you know a way to test if unbound is working correctly?

[james]

Hello @OrezBodajoy
Please run the following command and provide output of cloudron-support --troubleshoot

[OrezBodajoy]

I got this

Vendor: QEMU Product: Standard PC (Q35 + ICH9, 2009)
Linux: 5.15.0-153-generic
Ubuntu: jammy 22.04
Processor: AMD Ryzen 5 PRO 2400G with Radeon Vega Graphics x 4
RAM: 11205884KB
Disk: /dev/mapper/ubuntu--vg-ubuntu--lv   23G
[OK]	node version is correct
[OK]	IPv6 is enabled in kernel. No public IPv6 address
[OK]	docker is running
[OK]	docker version is correct
[OK]	MySQL is running
[OK]	nginx is running
[OK]	dashboard cert is valid
[OK]	dashboard is reachable via loopback
[OK]	box v8.3.2 is running
[OK]	netplan is good
[OK]	DNS is resolving via systemd-resolved
[OK]	Dashboard is reachable via domain name
[WARN]	Domain myredacteddomain.com expiry check skipped because whois does not have this information
[FAIL]	Unbound is not resolving, maybe try forwarding all DNS requests. You can do this by running 'cloudron-support --unbound-use-external-dns' option
;; communications error to 127.0.0.150#53: timed out
;; communications error to 127.0.0.150#53: timed out
;; no servers could be reached

and when I try the command suggested to forward everything, I get

/usr/bin/cloudron-support: line 803: unbound_use_external_dns: command not found

[james]

Hello @OrezBodajoy
It seems your unbound service is not working correctly.
Can you please check the output of the following command and post the result here:

systemctl status unbound.service

If the unbound.service is not running and can't be started or restarted with systemctl restart unbound.service please check if any other program is using the port 53.
To find out if a program is using port 53 you can run:

lsof -i :53

Please also post the output of the command above.

[OrezBodajoy]

systemctl status unbound.service says active and running, and lsof -i :53 has no output at all

I will post the full output of the first command

● unbound.service - Unbound DNS Resolver
     Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-09-24 10:56:03 UTC; 6h ago
    Process: 755 ExecStartPre=/usr/sbin/unbound-anchor -a /var/lib/unbound/root.key (code=exited, status=0/SUCCESS)
   Main PID: 6775 (unbound)
      Tasks: 1 (limit: 12995)
     Memory: 9.6M
        CPU: 365ms
     CGroup: /system.slice/unbound.service
             └─6775 /usr/sbin/unbound -d

Notice: journal has been rotated since unit was started, output may be incomplete.

Here is the output of netstat -ntlp | grep 53

tcp        0      0 172.18.0.1:53           0.0.0.0:*               LISTEN      6775/unbound        
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      715/systemd-resolve 
tcp        0      0 10.0.2.15:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 10.0.2.15:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 10.0.2.15:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 10.0.2.15:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 127.0.0.1:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 127.0.0.1:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 127.0.0.1:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 127.0.0.1:5349          0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 127.0.0.150:53          0.0.0.0:*               LISTEN      6775/unbound        
tcp        0      0 172.18.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.18.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.18.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.18.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.17.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.17.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.17.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp        0      0 172.17.0.1:5349         0.0.0.0:*               LISTEN      1444/turnserver     
tcp6       0      0 fec0::5054:ff:fe12:5349 :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fec0::5054:ff:fe12:5349 :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fec0::5054:ff:fe12:5349 :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fec0::5054:ff:fe12:5349 :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fd00:c107:d509::1:5349  :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fd00:c107:d509::1:5349  :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fd00:c107:d509::1:5349  :::*                    LISTEN      1444/turnserver     
tcp6       0      0 fd00:c107:d509::1:5349  :::*                    LISTEN      1444/turnserver     
tcp6       0      0 ::1:5349                :::*                    LISTEN      1444/turnserver     
tcp6       0      0 ::1:5349                :::*                    LISTEN      1444/turnserver     
tcp6       0      0 ::1:5349                :::*                    LISTEN      1444/turnserver     
tcp6       0      0 ::1:5349                :::*                    LISTEN      1444/turnserver     

[james]

Hello @OrezBodajoy
That lsof -i :53 returns nothing is suspicious.

Can you please share the output of the following files:

cat /etc/unbound/unbound.conf

and

cat /etc/unbound/unbound.conf.d/cloudron-network.conf

[joseph]

@OrezBodajoy Looks like unbound is running but the DNS requests don't work. Are you in an internal network? You can try forwarding all the DNS queries from unbound to google/cloudflare. Run

cloudron-support --unbound-use-external-dns

this is suggested by the tool output above.

It's doing what is documented in https://docs.cloudron.io/networking/#unbound

[OrezBodajoy]

Sorry for the delay, I can now check this issue.

Please note that the issue seems to be slightly different, I have moved the server's disk image and it seems that now unbound is working but not systemd-resolve.

Here are some command results.

redacted@cloudron:~$ ping 9.9.9.9
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=255 time=16.4 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=255 time=15.6 ms
64 bytes from 9.9.9.9: icmp_seq=3 ttl=255 time=22.8 ms
64 bytes from 9.9.9.9: icmp_seq=4 ttl=255 time=14.2 ms
64 bytes from 9.9.9.9: icmp_seq=5 ttl=255 time=15.3 ms
^C
--- 9.9.9.9 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 14.202/16.835/22.778/3.052 ms

redacted@cloudron:~$ host -t NS myredacteddomain.com
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
;; no servers could be reached
redacted@cloudron:~$ host -t NS myredacteddomain.com 127.0.0.150
;; communications error to 127.0.0.150#53: timed out
Using domain server:
Name: 127.0.0.150
Address: 127.0.0.150#53
Aliases: 

myredacteddomain.com  name server 2-can.njalla.in.
myredacteddomain.com  name server 1-you.njalla.no.
myredacteddomain.com name server 3-get.njalla.fo.
redacted@cloudron:~$ systemctl status unbound.service
WARNING: terminal is not fully functional
Press RETURN to continue 
● unbound.service - Unbound DNS Resolver
     Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2025-10-13 05:03:53 UTC; 10h ago
    Process: 752 ExecStartPre=/usr/sbin/unbound-anchor -a /var/lib/unbound/root.key (code=exited, status=0/SUCCESS)
   Main PID: 1002 (unbound)
      Tasks: 1 (limit: 17831)
     Memory: 11.1M
        CPU: 599ms
     CGroup: /system.slice/unbound.service
             └─1002 /usr/sbin/unbound -d

Oct 13 15:43:10 cloudron unbound[1002]: [1002:0] info: generate keytag query _ta-4f66-9728. NULL IN
Notice: journal has been rotated since unit was started, output may be incomplete.
redacted@cloudron:~$ lsof -i :53

redacted@cloudron:~$ lsof -i :53
redacted@cloudron:~$ cat /etc/unbound/unbound.conf
# Unbound configuration file for Debian.
#
# See the unbound.conf(5) man page.
#
# See /usr/share/doc/unbound/examples/unbound.conf for a commented
# reference config file.
#
# The following line includes additional configuration files from the
# /etc/unbound/unbound.conf.d directory.
include-toplevel: "/etc/unbound/unbound.conf.d/*.conf"
redacted@cloudron:~$ ll /etc/unbound//unbound.conf.d/
total 20
drwxr-xr-x 2 root root 4096 Sep 19 18:28 ./
drwxr-xr-x 3 root root 4096 Jul 23 06:24 ../
-rw-r--r-- 1 root root  692 Sep 15 17:33 cloudron-network.conf
-rw-r--r-- 1 root root  142 Sep 10 19:26 forward-everything.conf.moved
-rw-r--r-- 1 root root  190 Sep  7  2022 root-auto-trust-anchor-file.conf
redacted@cloudron:~$ cat /etc/unbound/unbound.conf.d/cloudron-network.conf
# Unbound is used primarily for RBL queries (host 2.0.0.127.zen.spamhaus.org)
# We cannot use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!)

server:
        port: 53
        interface: 127.0.0.150
        interface: 172.18.0.1
        ip-freebind: yes
        do-ip6: yes
        access-control: 127.0.0.1 allow
        access-control: 172.18.0.1/16 allow
        cache-max-negative-ttl: 30
        cache-max-ttl: 300
        # enable below for logging to journalctl -u unbound
        # verbosity: 5
        # LOG-QUERIES: yes

# https://github.com/NLnetLabs/unbound/issues/806
remote-control:
    control-enable: no

~$  systemctl status systemd-resolvectl
WARNING: terminal is not fully functional
Press RETURN to continue 
Unit systemd-resolvectl.service could not be found.
:~$  systemctl status systemd-resolved  
WARNING: terminal is not fully functional
Press RETURN to continue 
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2025-10-13 16:20:15 UTC; 4min 55s ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 302750 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 17831)
     Memory: 8.8M
        CPU: 1.013s
     CGroup: /system.slice/systemd-resolved.service
             └─302750 /lib/systemd/systemd-resolved

Oct 13 16:23:29 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3.
Oct 13 16:23:34 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3.
Oct 13 16:23:45 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3.
Oct 13 16:23:50 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3.
Oct 13 16:24:10 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3.
Oct 13 16:24:15 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3.
Oct 13 16:24:25 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3.
Oct 13 16:24:30 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3.
Oct 13 16:24:50 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3.
Oct 13 16:24:55 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3.

~$ resolvectl status         
WARNING: terminal is not fully functional
Press RETURN to continue 
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp0s9)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.0.2.3
       DNS Servers: 10.0.2.3

Note, on this page in the documentation I see the command
systemctl status systemd-resolved
It should be systemctl status systemd-resolvectl right?

[joseph]

@OrezBodajoy are you able to query Quad9 DNS directly? Ping is using ICMP and is different from DNS (UDP). Can you try host www.cloudron.io 9.9.9.9 ?

[OrezBodajoy]

Thanks for your input @joseph, I have found the issue!
It turns out that I am running cloudron in a qemu vm, and qemu has a bug in the default config where it will not resolve dns correctly if the first name server configured on the host fails.
In that case, it will not try the second nameserver configured on the host. DNS resolution will work on the host, but not on the guest.
Here is the link: https://unix.stackexchange.com/a/614603

[necrevistonnezr]

@OrezBodajoy said in queryNs ESERVFAIL for new apps:

It turns out that I am running cloudron in a qemu vm

Just a suggestion: Maybe you should have lead with that, makes virtual debugging much easier

[joseph]

@OrezBodajoy wow, great find