All Apps Error Status After Update to 8.0.3

[nebulon]

Right it is unclear here what the Cloudron update has to do really. All I can think of is the change from not using unbound anymore for local DNS resolving but now relying on whatever the environment the server runs in provides. So all in all all whatever ubuntu would setup without Cloudron should be the same as what your Cloudron should be configured with.

Maybe the previously used unbound just covered up some other networking setup issues, which are now revealed with stock ubuntu network setup.

[playsia]

Really don't think it's DNS, I have a lot of other systems (15+) that are working perfectly fine in this environment.

Trying to get to the bottom of this regression and solve it, having my cloudron stuck on an old version is not a long term solution. What's the command line equivalent of Cloudron downloading the icon?

[nebulon]

Yes something is off with the networking on that server then it seems. To replicate what the Cloudron code does here, the curl command I posted earlier is the one:

curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon

with the mentioned flavor of adding --ipv6 to ensure curl does not automatically fallback to ipv4 if ipv6 is not working as expected. So if curl works with ipv6 and ipv4 I can't see how Cloudron wouldn't be able to make requests.

[playsia]

Without adding --ipv6 it works fine once it falls back to ipv4.

Adding --ipv6:
curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon --ipv6

• Trying 2604:a880:800:10::b66:f001:443...
• Immediate connect fail for 2604:a880:800:10::b66:f001: Network is unreachable
• Closing connection 0
  curl: (7) Couldn't connect to server

[nebulon]

That seems to be the root cause, somehow ipv6 is not working correctly in your setup, causeing all tools/code which chooses ipv6 by default if available to then fail. Curl essentially has extra handling for this to fallback to ipv4 but that is specific to curl only.

[playsia]

ok so next thing to try...

Install Ubuntu Server 22.04 LTS
Remove ipv6 via grub BEFORE installing cloudron
Install and restore cloudron to working 7.7.2
Upgrade to 8.0.3... and hope things fall back to ipv4 like they should

Is there any other steps I could take to try and get this working?

[joseph]

@playsia yes, disabling ipv6 at the kernel level is a good start, I think. Cloudron will automatically figure that ipv6 is disabled as well.

[playsia]

Not solved, haven't had a chance to try it yet.

[playsia]

First attempt failed.

What I did in the terminal

<snip>:~$ wget https://cloudron.io/cloudron-setup
--2024-08-20 10:33:02-- https://cloudron.io/cloudron-setup
Resolving cloudron.io (cloudron.io)... 165.227.67.76, 2604:a880:800:10::b66:f001
Connecting to cloudron.io (cloudron.io)|165.227.67.76|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12646 (12K) [application/octet-stream]
Saving to: ‘cloudron-setup’

cloudron-setup 100%[===================>] 12.35K --.-KB/s in 0s

2024-08-20 10:33:03 (426 MB/s) - ‘cloudron-setup’ saved [12646/12646]

<snip>:~$ chmod +x cloudron-setup
<snip>:~$ sudo ./cloudron-setup --version 7.7.2
[sudo] password for <snip>:

##############################################
Cloudron Setup (7.7.2)
##############################################

Follow setup logs in a second terminal with:
$ tail -f /var/log/cloudron-setup.log

Join us at https://forum.cloudron.io for any questions.

=> Updating apt and installing script dependencies
=> Validating setup token
=> Checking version
=> Downloading Cloudron version 7.7.2 ...
=> Installing base dependencies (this takes some time) ...Init script failed. Se e /var/log/cloudron-setup.log for details

/var/log/cloudron-setup.log

Setting up mysql-server-8.0 (8.0.39-0ubuntu0.22.04.1) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
mysqld will log errors to /var/log/mysql/error.log
mysqld is running as pid 5694
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
Setting up unbound (1.13.1-1ubuntu5.5) ...
Created symlink /etc/systemd/system/multi-user.target.wants/unbound.service → /lib/systemd/system/unbound.service.
Created symlink /etc/systemd/system/unbound.service.wants/unbound-resolvconf.service → /lib/systemd/system/unbound-resolvconf.service.
Setting up cifs-utils (2:6.14-1ubuntu0.1) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so to provide /etc/cifs-utils/idmap-plugin (idmap-plugin) in auto mode
Setting up gcc (4:11.2.0-1ubuntu1) ...
Setting up libgd3:amd64 (2.3.0-2ubuntu2) ...
Setting up libstdc++-11-dev:amd64 (11.4.0-1ubuntu1~22.04) ...
Setting up g++-11 (11.4.0-1ubuntu1~22.04) ...
Setting up libnginx-mod-http-image-filter (1.18.0-6ubuntu14.4) ...
Setting up g++ (4:11.2.0-1ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.9ubuntu3) ...
Setting up nginx-core (1.18.0-6ubuntu14.4) ...
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "start" failed.
^[[0;1;31m×^[[0m nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (^[]8;;file://nuccloudronv2/lib/systemd/system/nginx.service^G/lib/systemd/system/nginx.service^[]8;;^G; enabled; vendor preset: enabled)
Active: ^[[0;1;31mfailed^[[0m (Result: exit-code) since Tue 2024-08-20 10:34:28 UTC; 23ms ago
Docs: ^[]8;;man:nginx(8)^Gman:nginx(8)^[]8;;^G
Process: 6179 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; ^[[0;1;31m(code=exited, status=1/FAILURE)^[[0m
CPU: 5ms

Aug 20 10:34:28 nuccloudronv2 systemd[1]: Starting A high performance web server and a reverse proxy server...
Aug 20 10:34:28 nuccloudronv2 nginx[6179]: nginx: [emerg] socket() [::]:80 failed (97: Unknown error)
Aug 20 10:34:28 nuccloudronv2 nginx[6179]: nginx: configuration file /etc/nginx/nginx.conf test failed
Aug 20 10:34:28 nuccloudronv2 systemd[1]: ^[[0;1;39m^[[0;1;31m^[[0;1;39mnginx.service: Control process exited, code=exited, status=1/FAILURE^[[0m
Aug 20 10:34:28 nuccloudronv2 systemd[1]: ^[[0;1;38;5;185m^[[0;1;39m^[[0;1;38;5;185mnginx.service: Failed with result 'exit-code'.^[[0m
Aug 20 10:34:28 nuccloudronv2 systemd[1]: ^[[0;1;31m^[[0;1;39m^[[0;1;31mFailed to start A high performance web server and a reverse proxy server.^[[0m
dpkg: error processing package nginx-core (--configure):
installed nginx-core package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nginx-full:
nginx-full depends on nginx-core (>= 1.18.0-6ubuntu14.4); however:
Package nginx-core is not configured yet.
nginx-full depends on nginx-core (<< 1.18.0-6ubuntu14.4.1~); however:
Package nginx-core is not configured yet.

dpkg: error processing package nginx-full (--configure):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
No apport report written because the error message indicates its a followup error from a previous failure.
Processing triggers for ufw (0.36.1-4ubuntu0.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for resolvconf (1.84ubuntu1) ...
Errors were encountered while processing:
nginx-core
nginx-full
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

[joseph]

@playsia Looks like ubuntu's nginx does not even start with ipv6 disabled. I can reproduce this in our local setups as well. Can you instead start with ipv6 enabled in kernel but have no ipv6 assigned?

[playsia]

I can try that next yes.

[playsia]

Worked for installing 7.7.2. Upgrading to 8.0.3 resulting in the same old downloading icon error.

[joseph]

@playsia ok, so back to DNS resolution error ? Can you check what is in your /etc/resolv.conf ?

[playsia]

GNU nano 6.2 /etc/resolv.conf

<snip comment prelude>

nameserver 127.0.0.53
options edns0 trust-ad
search .

[joseph]

@playsia ok, that looks correct.

Now, systemctl status systemd-resolved should say running. Finally, resolvectl should show some nameservers. I suspect resolvectl is not showing any nameservers. In that case, edit /etc/systemd/resolved.conf and edit the DNS line like so:

DNS=8.8.8.8 1.1.1.1

Put your own network's nameservers above, if you have them. Otherwise, 8.8.8.8 is google and 1.1.1.1 is cloudflare dns. systemctl restart systemd-resolved . After this dns will work. resolvectl will also show the DNS servers you added.

[playsia]

systemctl status systemd-resolved: yes it's running

Results of resolvectl:
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp1s0)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.5
DNS Servers: 192.168.0.5 192.168.0.6

Link 3 (wlp2s0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (br-60558136c824)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 7 (veth14a60e8)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 9 (veth781f0fd)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 11 (veth6430967)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 13 (veth1baaf71)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 15 (vethf12b789)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 17 (veth41c82ea)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 19 (veth23598f8)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 21 (veth279bdec)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 23 (veth6e84b1b)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 25 (veth305c936)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

[playsia]

resolvectl after changing /etc/systemd/resolved.conf. Only change is Global.

Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Current DNS Server: 192.168.0.5
DNS Servers: 192.168.0.5 192.168.0.6

Log:
box:tasks startTask: 7791 done. error: { stack: 'BoxError: Network error downloading icon : \n' + ' at /home/yellowtent/box/src/appstore.js:493:33\n' + ' at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' + ' at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n' + ' at async Object.downloadIcon (/home/yellowtent/box/src/appstore.js:487:12)\n' + ' at async downloadIcon (/home/yellowtent/box/src/apptask.js:208:26)\n' + ' at async configure (/home/yellowtent/box/src/apptask.js:564:5)', name: 'BoxError', reason: 'Network Error', details: {}, message: 'Network error downloading icon : ' }

[joseph]

So your DNS servers are 192.168.0.5 and 192.168.0.6 ? I am not able to work out why it still cannot resolve. If you can email support@ , we can take a look.

[micmc]

@joseph said in All Apps Error Status After Update to 8.0.3:

So your DNS servers are 192.168.0.5 and 192.168.0.6 ? I am not able to work out why it still cannot resolve. If you can email support@ , we can take a look.

These are local network IP addresses, very likely the local DNSs from his router.

[playsia]

They are my two piholes. I will reinstall with DNS=8.8.8.8 1.1.1.1 but I doubt this is the issue.