All Apps Error Status After Update to 8.0.3

[nebulon]

Glad the restore did work for you. Which VPS provider are you using, as network setup is not really part of Cloudron, but usually done by the server provider cloud-init scripts.

[playsia]

Not solved unfortunately. It updated to 8.0.3 this morning and everything is broken again in the same way.

Cloudron is run on a dedicated NUC in my house (selfhost). Running Ubuntu Server 22.04.4 LTS. Configuration is very simple, just going through the prompts and then immediately install cloudron once I get SSH, but I noticed that it didn't pull an ipv6 address when going through the initial config.

Either my DHCPv6 is broken or it's the problem I found here:
https://ubuntuforums.org/showthread.php?t=2487465
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2025382

I will try the sudo dhclient -6 <interface> fix when I get home, I'm out of time for today and need to rush off to work.

[playsia]

As a note, because the 8.0.3 backups weren't proper the restore I did was to the 7.7.2.

Another thing I could do is use a static ipv6 address? I'd need to figure out what that would be as I am unfamiliar with ipv6.

[playsia]

ok so investigation has discovered... ipv6 not enabled by default on my ISP which is most likely the issue. I have changed ISP since I last had an ipv6 issue with Cloudron.

On the phone to get it resolved atm.

[playsia]

Unfortunately ipv6 is not available for my internet service. This is not an isolated thing, after some quick searching there are plenty of ISPs still around that don't do IPv6.

[nebulon]

In such a case you probably have to disable ipv6 for the Cloudron and then things should work?

[playsia]

@nebulon
I tried the GRUB method here: https://itsfoss.com/disable-ipv6-ubuntu-linux/#2-disable-ipv6-using-grub

Still same issue. Is there a better or recommended way to disable ipv6?

As you've probably noted, 7.7.2 worked with no problems at all, it's only after it updated to 8.0.3

[nebulon]

For ipv6 as such, there is inbound and outbound connectivity. Inbound may not be routed correctly to your server directly depending on your ISP and thus the AAAA records should not be setup, to avoid clients trying this. For outbound, if your other systems like your laptop works fine, your Cloudron should also. Would be good to understand what the difference is.

[playsia]

my HTPC and ROG Ally both have ipv6 addresses and can ping local ipv6. As I said though, my ISP does not support/offer ipv6 so any connection to the internet can only be ipv4.

[nebulon]

I am not familar with what your ISP uses, but if it has no actual ipv6 connectivity, why is your router then setting up ipv6 for your local network? The German ISPs which only offer ipv6 over ipv4 make outbound ipv6 connections work, which is what seems to not work in your case. But as said Cloudron does not do any special network configuration as such, so whatever ubuntu sets up is used, so not sure how to proceed here.

[playsia]

My pihole is setting up (dhcp) ipv6, which I switched on trying to get ipv6 to work. I can turn it back off.

[necrevistonnezr]

Having a pi-hole between Cloudron and the internet is, generally speaking, not a good idea (nor required) from my personal experience. You want to block end devices, particular those you cannot really control (like TVs, mobiles, etc.) but not a server that you can control.
In particular, since you can run Adguard Home on Cloudron and have it serve the rest of the network as an effective adblocker.

[playsia]

Cloudron on ipv4 is static ip and only uses pihole for DNS. I can set the dns server to be something else but that totally isn't the problem here.

Ubuntu Server 22.04 LTS on 7.7.2, without ipv6 = fine
Upgrade to 8.0.3 in same environment = apps don't launch with initially reported error

[nebulon]

Right it is unclear here what the Cloudron update has to do really. All I can think of is the change from not using unbound anymore for local DNS resolving but now relying on whatever the environment the server runs in provides. So all in all all whatever ubuntu would setup without Cloudron should be the same as what your Cloudron should be configured with.

Maybe the previously used unbound just covered up some other networking setup issues, which are now revealed with stock ubuntu network setup.

[playsia]

Really don't think it's DNS, I have a lot of other systems (15+) that are working perfectly fine in this environment.

Trying to get to the bottom of this regression and solve it, having my cloudron stuck on an old version is not a long term solution. What's the command line equivalent of Cloudron downloading the icon?

[nebulon]

Yes something is off with the networking on that server then it seems. To replicate what the Cloudron code does here, the curl command I posted earlier is the one:

curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon

with the mentioned flavor of adding --ipv6 to ensure curl does not automatically fallback to ipv4 if ipv6 is not working as expected. So if curl works with ipv6 and ipv4 I can't see how Cloudron wouldn't be able to make requests.

[playsia]

Without adding --ipv6 it works fine once it falls back to ipv4.

Adding --ipv6:
curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon --ipv6

• Trying 2604:a880:800:10::b66:f001:443...
• Immediate connect fail for 2604:a880:800:10::b66:f001: Network is unreachable
• Closing connection 0
  curl: (7) Couldn't connect to server

[nebulon]

That seems to be the root cause, somehow ipv6 is not working correctly in your setup, causeing all tools/code which chooses ipv6 by default if available to then fail. Curl essentially has extra handling for this to fallback to ipv4 but that is specific to curl only.

[playsia]

ok so next thing to try...

Install Ubuntu Server 22.04 LTS
Remove ipv6 via grub BEFORE installing cloudron
Install and restore cloudron to working 7.7.2
Upgrade to 8.0.3... and hope things fall back to ipv4 like they should

Is there any other steps I could take to try and get this working?

[joseph]

@playsia yes, disabling ipv6 at the kernel level is a good start, I think. Cloudron will automatically figure that ipv6 is disabled as well.