Auto-Update to 8.0.3 - systemd-resolved empty response

[TLeM4]

Hi,

An automatic upgrade happened this night on our server and all DNS queries result in empty response from systemd-resolved.

Here a part of the box.log where we saw the first error:

2024-08-10T01:33:35.165Z box:apphealthmonitor app health: 24 running / 2 stopped / 0 unresponsive
2024-08-10T01:33:36.851Z box:box Received SIGTERM. Shutting down.
2024-08-10T01:33:36.858Z box:platform uninitializing platform
2024-08-10T01:33:36.859Z box:shell startTask: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 12905 /home/yellowtent/platformdata/logs/tasks/12905.log 15 1024 errored BoxError: startTask exited with code null signal SIGTERM
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:110:19)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess._handle.onexit (node:internal/child_process:291:12) {
  reason: 'Shell Error',
  details: {},
  code: null,
  signal: 'SIGTERM'
}
2024-08-10T01:33:36.934Z box:tasks startTask: 12905 completed with code null
2024-08-10T01:33:36.937Z box:tasks setCompleted - 12905: {"error":{"message":"Task 12905 crashed with code null","code":"crashed"}}
2024-08-10T01:33:36.937Z box:tasks update 12905: {"percent":100,"error":{"message":"Task 12905 crashed with code null","code":"crashed"}}
2024-08-10T01:33:36.938Z box:platform onDeactivated: stopping post activation services
2024-08-10T01:33:36.939Z box:tasks stopAllTasks: stopping all tasks
2024-08-10T01:33:36.940Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2024-08-10T01:33:36.952Z box:locker Released : box_update
2024-08-10T01:33:36.952Z box:updater Update failed with error. { message: 'Task 12905 crashed with code null', code: 'crashed' }
2024-08-10T01:33:36.953Z box:tasks startTask: 12905 done. error: { message: 'Task 12905 crashed with code null', code: 'crashed' }
2024-08-10T01:35:10.561Z box:server ==========================================
2024-08-10T01:35:10.562Z box:server            Cloudron 8.0.3  
2024-08-10T01:35:10.562Z box:server ==========================================
2024-08-10T01:35:10.562Z box:platform initialize: start platform
2024-08-10T01:35:10.902Z box:tasks stopAllTasks: stopping all tasks
2024-08-10T01:35:10.902Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
sudo: unable to resolve host my: Temporary failure in name resolution
All tasks stopped
2024-08-10T01:35:11.141Z box:reverseproxy writeDashboardConfig: writing dashboard config for a38.fr
2024-08-10T01:35:11.158Z box:shell isOscpEnabled execArgs: openssl ["x509","-in","/home/yellowtent/platformdata/nginx/cert/_.a38.fr.cert","-noout","-ocsp_uri"]
2024-08-10T01:35:11.179Z box:shell reload /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
sudo: unable to resolve host my: Temporary failure in name resolution
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.a38.fr.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/a38.fr.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.a38.fr.cert"
nginx: [warn] "ssl_stapling" ignored, host not found in OCSP responder "e5.o.lencr.org" in the certificate "/home/yellowtent/platformdata/nginx/cert/_.a38.fr.cert"

dig response:

$ dig api.cloudron.io

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> api.cloudron.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 12668
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;api.cloudron.io.		IN	A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sat Aug 10 15:56:07 UTC 2024
;; MSG SIZE  rcvd: 44

All apps were in error state and i successfully started them all with write api.cloudron.io in host file and retry configure task in cloudron panel.

Unfortunately, this is only temporary fix as dns issue is still there and app store for example is still unavailable (so i commented out this fix).

[girish]

@TLeM4 I guess systemd-resolved is not being configured with any nameserver. Not sure about your setup, but it works like this:

• IP address and DNS is configured via this netplan config file. Since, it seems you don't use DHCP in netplan, the DNS has to come from somewhere... This can be hardcoded in netplan like below (important: merge the section below to your existing netplan). Instead of 8.8.8.8, you should ideally use your network local DNS. A reboot should then make systemd-resolved pick up the DNS below.

    ethernets:
        eth0:
            nameservers:
                addresses:
                - 8.8.8.8
                - 1.1.1.1

[girish]

@TLeM4 can you tell us a bit about your server setup (internal or public VPS) ? The DNS resolution is now based on systemd-resolved. You can run resolvectl to see the DNS servers that the server is configured with.

[TLeM4]

@girish this is a public VPS (this one)
This is the command result:

resolvectl |cat
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (br-3d41c8b84b0d)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (veth80c01b5)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 8 (vethe16e1d4)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 10 (vethed7b139)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 12 (vethe67b486)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 14 (vethd4ef98c)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 16 (vethe6749b8)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 20 (veth9b5beb5)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 22 (vethe3dfaa9)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 24 (veth109052f)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 26 (veth093bad2)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 28 (vethe6fe0f0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 30 (veth4c250c4)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 32 (veth9612c7c)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 34 (veth32f80d0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 38 (veth362eae4)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 42 (veth4c8efe9)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 44 (veth75c6814)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 46 (vethf18555a)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 48 (vetha0fbec1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 52 (vethd427026)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 54 (veth2b70d82)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 56 (vetha7b4ccb)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 58 (veth1894ed4)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 60 (vetheadf5c5)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 62 (veth69ed68b)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 64 (veth6b6f42a)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 66 (vethc2e2a32)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 70 (vetha39cf1d)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 72 (veth3493a4d)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 74 (veth79e2f75)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 76 (vethc6fa10b)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 98 (veth40adda1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 104 (vethb251de5)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 128 (veth2a70946)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 130 (vethc338380)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 132 (veth34771c4)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 134 (vethd1f944b)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

[girish]

@TLeM4 said in Auto-Update to 8.0.3 - systemd-resolved empty response:

Link 2 (eth0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

This interface should have DNS but for some reason it doesn't . Can you check /etc/netplan/50-cloud-init.yaml ? We actually have code specific to netup for 8.0 migration. Maybe something failed here.... Specifically:

• /etc/netplan/50-cloud-init.yaml must have a ethernets.eth0.nameservers.addresses section . This should have netcup's DNS. Does it?
• Check /etc/resolv.conf . This should contain nameserver 127.0.0.53
• To double check this, systemctl status systemd-resolved should show active and running .

Most likely something went wrong with netplan re-configuration

[TLeM4]

Thanks. This is the commands results:

$ cat /etc/netplan/50-cloud-init.yaml
network:
    version: 2
    renderer: networkd
    ethernets:
        eth0:
            addresses:
            - 45.83.105.92/22
            - 2a03:4000:46:463:880b:33ff:fe29:8d7c/64
            gateway4: 45.83.104.1
            gateway6: fe80::1
            match:
                macaddress: 8a:0b:33:29:8d:7c

$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search .

$ systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2024-08-10 09:56:51 UTC; 3 days ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
   Main PID: 14835 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 19050)
     Memory: 4.9M
        CPU: 1min 33.372s
     CGroup: /system.slice/systemd-resolved.service
             └─14835 /lib/systemd/systemd-resolved

Notice: journal has been rotated since unit was started, output may be incomplete.

[girish]

@TLeM4 I guess systemd-resolved is not being configured with any nameserver. Not sure about your setup, but it works like this:

• IP address and DNS is configured via this netplan config file. Since, it seems you don't use DHCP in netplan, the DNS has to come from somewhere... This can be hardcoded in netplan like below (important: merge the section below to your existing netplan). Instead of 8.8.8.8, you should ideally use your network local DNS. A reboot should then make systemd-resolved pick up the DNS below.

    ethernets:
        eth0:
            nameservers:
                addresses:
                - 8.8.8.8
                - 1.1.1.1

[TLeM4]

Thanks, it's working !