Severe DNS problems (still on 7.7.2)

cstork

I don't know how it got so far but it seems my DNS lookup fails already for quite some time. I had problems before but then it seemed that restarting unbound after it went down solved the problems.

$ nslookup example.com
;; Got SERVFAIL reply from 127.0.0.53
Server:		127.0.0.53
Address:	127.0.0.53#53

** server can't find example.com: SERVFAIL

I'm running Cloudron 7.7.2 on Ubuntu 20.04.2.

$ sudo cloudron-support --troubleshoot
[sudo] password for cstork:
[OK]	node version is correct
[OK]	docker is running
[OK]	MySQL is running
[OK]	nginx is running
[OK]	box is running
	unbound is down. updating root anchor to see if it fixes it
[OK]	unbound is running
[OK]	Dashboard is reachable via domain name
getaddrinfo(whois.denic.de): Temporary failure in name resolution
[WARN]	Domain eichenlaub.de expiry check skipped because whois does not have this information

What's the best way to further analyse this?

joseph

@cstork I think this happened because of a failed upgrade to Cloudron 8. Please do this:

apt remove resolvconf
systemctl start systemd-resolved
In /etc/resolv.conf, you should have see nameserver 127.0.0.53 (which is systemd-resolved)
At this point, nslookup should work. If it does not , it means that systemd-resolved does not have nameservers. Check the output of resolvectl to see what nameservers it has. You can edit /etc/systemd/resolved.conf and put the DNS=1.1.1.1 for example and systemctl restart systemd-resolved.
Once the DNS works, upgrade to Cloudron 8 immediately .

cstork

I don't think there was an attempt to upgrade to 8 yet. I think the DNS problems were there already before.

Anyway, here's what I got:

root@my:~#  apt remove resolvconf
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'resolvconf' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 117 not upgraded.

root@my:~#  ps aux | grep resolv
systemd+     762  0.0  0.1  24000  5604 ?        Ss   Sep25   0:08 /lib/systemd/systemd-resolved

root@my:~#  cat /etc/resolv.conf
# ...
nameserver 127.0.0.53
options edns0 trust-ad

root@my:~#  resolvectl status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 46 (veth7845bc1)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

... lots more veth....

Link 4 (br-9ecb37f36dc9)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (docker0)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (ens32)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

root@my:~# echo DNS=1.1.1.1 >> /etc/resolv.conf
root@my:~# systemctl restart systemd-resolved
root@my:~# ping google.com
ping: google.com: Temporary failure in name resolution

cstork

Oops, wait! By now the DNS entry in /etc/resolv.conf has disappeared again. Give me some more time to figure out what to do....

cstork

Ok, so this fixed it... for now:

sudo resolvectl dns ens32 1.1.1.1 8.8.8.8 && sudo systemctl restart systemd-resolved

But I'm still baffled as to how it could get into this state. Any idea?

cstork

However, this does not survive a reboot:

# after reboot
$ resolvectl dns ens32
Link 2 (ens32):

How do I permanently change the nameserver (and why was it "unset" without me doing anything)?

joseph

@cstork said in Severe DNS problems (still on 7.7.2):

root@my:~# echo DNS=1.1.1.1 >> /etc/resolv.conf

Oh no, this is not correct...

/etc/resolv.conf should look like https://paste.cloudron.io/jikayupofe.apache
Then in /etc/systemd/resolved.conf (which is not the same as above file!) , edit DNS=1.1.1.1 . Usually, there is already a commented out line in this file. After making that changed systemctl restart systemd-resolved

cstork

Hmm, my /etc/resolv.conf lacks the last line:

cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad

Do I need to fix that?

Re /etc/systemd/resolved.conf: indeed I confused the two files and now – with the edit – DNS lookup works even after reboot! Thank you!

However, I still had problems with the Cloudron OpenID login to my apps. I ran:

[sudo] password for cstork:
[OK]	node version is correct
[OK]	docker is running
[OK]	MySQL is running
[OK]	nginx is running
[OK]	box is running
	unbound is down. updating root anchor to see if it fixes it
[OK]	unbound is running
[OK]	Dashboard is reachable via domain name
[WARN]	Domain xx expiry check skipped because whois does not have this information

and that fixed the login issues apparently.

Now my only question is for a word of advice: First update to Cloudron 8 or to Ubuntu 22.04/24.04 ?

joseph

@cstork said in Severe DNS problems (still on 7.7.2):

Do I need to fix that?

The search is not needed. It's used when you try to use names without fully qualifying them. For example, ping blah can magically become blah.mydomain.com or something else depending on how you configure search. This is not used by Cloudron in any case...

Now my only question is for a word of advice: First update to Cloudron 8 or to Ubuntu 22.04/24.04 ?

Update to Cloudron 8 and then you can upgrade ubuntu after. You cannot upgrade to Ubuntu 24 until upgrading to Cloudron 8...

cstork

Great! Thank you!

BTW, it seems in retrospect that the start of the DNS problems coincided with me beginning to use the OpenVPN app. Take that for whatever it's worth.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Severe DNS problems (still on 7.7.2)