Update to 8.0.3 / DNS challenges

[jadudm]

Hi all,

This thread is intended to document my reading of other support tickets, and what I've done in an attempt to bring my cloudron back.

BLUF: I succeeded, but I don't like how I did it. Ultimately, I had to edit resolv.conf. This ticket can likely be closed.

Initial problem

My experience so far is similar to this thread: https://forum.cloudron.io/topic/12294/autoupdate-cloudron-aborts/9

I pushed the button for the update to 8.0.3. Unfortunately, I did this shortly before leaving the house for two days. I thought "everything will be fine."

Return from trip

I came back to find apps and services down. (Unbound was down, for example.) I did a reboot to see what would happen. ("Turn it off and turn it back on again.") The services came back, but the apps did not.

Looking at one of the apps, I see this error:

An error occurred during the configure operation: Network Error: Network error downloading icon : getaddrinfo EAI_AGAIN api.cloudron.io

Explore the forum

I tried some possibly obvious fixes from the forum.

Update unbound settings

Following this page:

https://docs.cloudron.io/networking/#dns

I added a file to /etc/unbound/unbound.conf.d called override.conf, and it contains the following:

server:
    val-permissive-mode: yes
forward-zone:
    name: "."
    forward-addr: 10.0.0.1

My cloudron lives behind an OpnSense instance which serves as my router, and it is at 10.0.0.1. If you are following this, your DNS server is probably not 10.0.0.1. So, if you're also trying to fix something similar... you cannot just "plug and play" the values I use, but you might be able to follow the process.

After that change and an unbound restart, I could do:

host www.cloudron.io 127.0.0.150

and it reported back

Using domain server:
Name: 127.0.0.150
Address: 127.0.0.150#53
Aliases: 

www.cloudron.io has address 165.227.67.76
www.cloudron.io has IPv6 address 2604:a880:800:10::b66:f001

This is good. I then tried to configure a failing app, and it still would not talk to api.cloudron.io. So, the problem did not go away.

Update netplan

I still had a DNS problem. Based on this thread:

https://forum.cloudron.io/topic/12266/auto-update-to-8-0-3-systemd-resolved-empty-response/6

I considered the possibility that the issue might be deeper down. I went into my netplan, and modified my configuration. It was set for straight DHCP. My netplan now reads as:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s31f6:
      dhcp4: true
      dhcp6: false
      dhcp4-overrides:
        use-dns: false
      nameservers:
        addresses:
          - 10.0.0.1
          - 1.1.1.1
          - 8.8.8.8

This should, I think, set my router as the first DNS option, with Cloudflare and Google as fallbacks.

I did a netplan apply and then I rebooted the machine (shutdown -r now), and everything came back. I also enabled DNS query forwarding in OpnSense at this point, just in case. I don't know that it mattered.

host www.cloudron.io 127.0.0.150

works.

curl https://releases.cloudron.io/versions.json

does not work. So, curl is not picking up the DNS; it seems to be looking to the default :53, which I suspect is not unbound, but instead... something else. dig has the same problem.

 dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> google.com
;; global options: +cmd
;; no servers could be reached

Edit /etc/resolv.conf

I don't like the solution I ended up with.

Based on this thread:

https://forum.cloudron.io/topic/12286/dns-failure/8

I decided that my empty resolv.conf was a problem. (I don't know why it was empty, and I don't like editing a file that claims it will be overwritten.)

I entered

nameserver 10.0.0.1

(again... for me is a good value.)

Reboot

After that change, I was able to manually configure/restart two apps. From there, I decided the fastest way to bring everything back (and confirm things might be good in general) would be to do a reboot from the admin panel.

This is not fix all of the apps. I had to retry the configure task on each, and Restart App for each.

Conclusion

At this point, I now have all my apps back. Cloudron claims v8.0.3 (Ubuntu 22.04.1 LTS), which is the most recent version.

Platform Version: v8.0.3 (Ubuntu 22.04.1 LTS)
Vendor: Dell Inc.
Product: OptiPlex 7040
CPU: 4 Core "Intel(R) Core(TM) i5-6500T CPU @ 2.50GHz"
Memory: 33.51 GB RAM & 8.59 GB Swap
Uptime: 9 minutes

I think this ticket can be closed, but given that a number of updates to 8.0.3 seem to have had DNS issues, I am going to submit this here. It may also be that I fixed the problem incorrectly.