All Apps Error Status After Update to 8.0.3

[playsia]

my HTPC and ROG Ally both have ipv6 addresses and can ping local ipv6. As I said though, my ISP does not support/offer ipv6 so any connection to the internet can only be ipv4.

[nebulon]

I am not familar with what your ISP uses, but if it has no actual ipv6 connectivity, why is your router then setting up ipv6 for your local network? The German ISPs which only offer ipv6 over ipv4 make outbound ipv6 connections work, which is what seems to not work in your case. But as said Cloudron does not do any special network configuration as such, so whatever ubuntu sets up is used, so not sure how to proceed here.

[playsia]

My pihole is setting up (dhcp) ipv6, which I switched on trying to get ipv6 to work. I can turn it back off.

[necrevistonnezr]

Having a pi-hole between Cloudron and the internet is, generally speaking, not a good idea (nor required) from my personal experience. You want to block end devices, particular those you cannot really control (like TVs, mobiles, etc.) but not a server that you can control.
In particular, since you can run Adguard Home on Cloudron and have it serve the rest of the network as an effective adblocker.

[playsia]

Cloudron on ipv4 is static ip and only uses pihole for DNS. I can set the dns server to be something else but that totally isn't the problem here.

Ubuntu Server 22.04 LTS on 7.7.2, without ipv6 = fine
Upgrade to 8.0.3 in same environment = apps don't launch with initially reported error

[nebulon]

Right it is unclear here what the Cloudron update has to do really. All I can think of is the change from not using unbound anymore for local DNS resolving but now relying on whatever the environment the server runs in provides. So all in all all whatever ubuntu would setup without Cloudron should be the same as what your Cloudron should be configured with.

Maybe the previously used unbound just covered up some other networking setup issues, which are now revealed with stock ubuntu network setup.

[playsia]

Really don't think it's DNS, I have a lot of other systems (15+) that are working perfectly fine in this environment.

Trying to get to the bottom of this regression and solve it, having my cloudron stuck on an old version is not a long term solution. What's the command line equivalent of Cloudron downloading the icon?

[nebulon]

Yes something is off with the networking on that server then it seems. To replicate what the Cloudron code does here, the curl command I posted earlier is the one:

curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon

with the mentioned flavor of adding --ipv6 to ensure curl does not automatically fallback to ipv4 if ipv6 is not working as expected. So if curl works with ipv6 and ipv4 I can't see how Cloudron wouldn't be able to make requests.

[playsia]

Without adding --ipv6 it works fine once it falls back to ipv4.

Adding --ipv6:
curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon --ipv6

• Trying 2604:a880:800:10::b66:f001:443...
• Immediate connect fail for 2604:a880:800:10::b66:f001: Network is unreachable
• Closing connection 0
  curl: (7) Couldn't connect to server

[nebulon]

That seems to be the root cause, somehow ipv6 is not working correctly in your setup, causeing all tools/code which chooses ipv6 by default if available to then fail. Curl essentially has extra handling for this to fallback to ipv4 but that is specific to curl only.

[playsia]

ok so next thing to try...

Install Ubuntu Server 22.04 LTS
Remove ipv6 via grub BEFORE installing cloudron
Install and restore cloudron to working 7.7.2
Upgrade to 8.0.3... and hope things fall back to ipv4 like they should

Is there any other steps I could take to try and get this working?

[joseph]

@playsia yes, disabling ipv6 at the kernel level is a good start, I think. Cloudron will automatically figure that ipv6 is disabled as well.

[playsia]

Not solved, haven't had a chance to try it yet.

[playsia]

First attempt failed.

What I did in the terminal

<snip>:~$ wget https://cloudron.io/cloudron-setup
--2024-08-20 10:33:02-- https://cloudron.io/cloudron-setup
Resolving cloudron.io (cloudron.io)... 165.227.67.76, 2604:a880:800:10::b66:f001
Connecting to cloudron.io (cloudron.io)|165.227.67.76|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12646 (12K) [application/octet-stream]
Saving to: ‘cloudron-setup’

cloudron-setup 100%[===================>] 12.35K --.-KB/s in 0s

2024-08-20 10:33:03 (426 MB/s) - ‘cloudron-setup’ saved [12646/12646]

<snip>:~$ chmod +x cloudron-setup
<snip>:~$ sudo ./cloudron-setup --version 7.7.2
[sudo] password for <snip>:

##############################################
Cloudron Setup (7.7.2)
##############################################

Follow setup logs in a second terminal with:
$ tail -f /var/log/cloudron-setup.log

Join us at https://forum.cloudron.io for any questions.

=> Updating apt and installing script dependencies
=> Validating setup token
=> Checking version
=> Downloading Cloudron version 7.7.2 ...
=> Installing base dependencies (this takes some time) ...Init script failed. Se e /var/log/cloudron-setup.log for details

/var/log/cloudron-setup.log

Setting up mysql-server-8.0 (8.0.39-0ubuntu0.22.04.1) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
mysqld will log errors to /var/log/mysql/error.log
mysqld is running as pid 5694
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
Setting up unbound (1.13.1-1ubuntu5.5) ...
Created symlink /etc/systemd/system/multi-user.target.wants/unbound.service → /lib/systemd/system/unbound.service.
Created symlink /etc/systemd/system/unbound.service.wants/unbound-resolvconf.service → /lib/systemd/system/unbound-resolvconf.service.
Setting up cifs-utils (2:6.14-1ubuntu0.1) ...
update-alternatives: using /usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so to provide /etc/cifs-utils/idmap-plugin (idmap-plugin) in auto mode
Setting up gcc (4:11.2.0-1ubuntu1) ...
Setting up libgd3:amd64 (2.3.0-2ubuntu2) ...
Setting up libstdc++-11-dev:amd64 (11.4.0-1ubuntu1~22.04) ...
Setting up g++-11 (11.4.0-1ubuntu1~22.04) ...
Setting up libnginx-mod-http-image-filter (1.18.0-6ubuntu14.4) ...
Setting up g++ (4:11.2.0-1ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.9ubuntu3) ...
Setting up nginx-core (1.18.0-6ubuntu14.4) ...
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
invoke-rc.d: initscript nginx, action "start" failed.
^[[0;1;31m×^[[0m nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (^[]8;;file://nuccloudronv2/lib/systemd/system/nginx.service^G/lib/systemd/system/nginx.service^[]8;;^G; enabled; vendor preset: enabled)
Active: ^[[0;1;31mfailed^[[0m (Result: exit-code) since Tue 2024-08-20 10:34:28 UTC; 23ms ago
Docs: ^[]8;;man:nginx(8)^Gman:nginx(8)^[]8;;^G
Process: 6179 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; ^[[0;1;31m(code=exited, status=1/FAILURE)^[[0m
CPU: 5ms

Aug 20 10:34:28 nuccloudronv2 systemd[1]: Starting A high performance web server and a reverse proxy server...
Aug 20 10:34:28 nuccloudronv2 nginx[6179]: nginx: [emerg] socket() [::]:80 failed (97: Unknown error)
Aug 20 10:34:28 nuccloudronv2 nginx[6179]: nginx: configuration file /etc/nginx/nginx.conf test failed
Aug 20 10:34:28 nuccloudronv2 systemd[1]: ^[[0;1;39m^[[0;1;31m^[[0;1;39mnginx.service: Control process exited, code=exited, status=1/FAILURE^[[0m
Aug 20 10:34:28 nuccloudronv2 systemd[1]: ^[[0;1;38;5;185m^[[0;1;39m^[[0;1;38;5;185mnginx.service: Failed with result 'exit-code'.^[[0m
Aug 20 10:34:28 nuccloudronv2 systemd[1]: ^[[0;1;31m^[[0;1;39m^[[0;1;31mFailed to start A high performance web server and a reverse proxy server.^[[0m
dpkg: error processing package nginx-core (--configure):
installed nginx-core package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nginx-full:
nginx-full depends on nginx-core (>= 1.18.0-6ubuntu14.4); however:
Package nginx-core is not configured yet.
nginx-full depends on nginx-core (<< 1.18.0-6ubuntu14.4.1~); however:
Package nginx-core is not configured yet.

dpkg: error processing package nginx-full (--configure):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
No apport report written because the error message indicates its a followup error from a previous failure.
Processing triggers for ufw (0.36.1-4ubuntu0.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for resolvconf (1.84ubuntu1) ...
Errors were encountered while processing:
nginx-core
nginx-full
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

[joseph]

@playsia Looks like ubuntu's nginx does not even start with ipv6 disabled. I can reproduce this in our local setups as well. Can you instead start with ipv6 enabled in kernel but have no ipv6 assigned?

[playsia]

I can try that next yes.

[playsia]

Worked for installing 7.7.2. Upgrading to 8.0.3 resulting in the same old downloading icon error.

[joseph]

@playsia ok, so back to DNS resolution error ? Can you check what is in your /etc/resolv.conf ?

[playsia]

GNU nano 6.2 /etc/resolv.conf

<snip comment prelude>

nameserver 127.0.0.53
options edns0 trust-ad
search .

[joseph]

@playsia ok, that looks correct.

Now, systemctl status systemd-resolved should say running. Finally, resolvectl should show some nameservers. I suspect resolvectl is not showing any nameservers. In that case, edit /etc/systemd/resolved.conf and edit the DNS line like so:

DNS=8.8.8.8 1.1.1.1

Put your own network's nameservers above, if you have them. Otherwise, 8.8.8.8 is google and 1.1.1.1 is cloudflare dns. systemctl restart systemd-resolved . After this dns will work. resolvectl will also show the DNS servers you added.