All Apps Error Status After Update to 8.0.3

[playsia]

curl -v https://api.cloudron.io/api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon

• Trying 165.227.67.76:443...
• Trying 2604:a880:800:10::b66:f001:443...
• Immediate connect fail for 2604:a880:800:10::b66:f001: Network is unreachable
• Connected to api.cloudron.io (165.227.67.76) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS header, Certificate Status (22):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS header, Finished (20):
• TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS header, Certificate Status (22):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS header, Finished (20):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• ALPN, server accepted to use http/1.1
• Server certificate:
• subject: CN=api.cloudron.io
• start date: Jul 1 01:05:20 2024 GMT
• expire date: Sep 29 01:05:19 2024 GMT
• subjectAltName: host "api.cloudron.io" matched cert's "api.cloudron.io"
• issuer: C=US; O=Let's Encrypt; CN=R11
• SSL certificate verify ok.
• TLSv1.2 (OUT), TLS header, Supplemental data (23):

GET /api/v1/apps/com.openwebui.cloudronapp/versions/2.3.10/icon HTTP/1.1
Host: api.cloudron.io
User-Agent: curl/7.81.0
Accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23):
• Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Server: nginx/1.18.0 (Ubuntu)
  < Date: Tue, 06 Aug 2024 09:03:00 GMT
  < Content-Type: image/png
  < Content-Length: 6161
  < Connection: keep-alive
  < X-Powered-By: Express
  < Cache-Control: public, max-age=604800, immutable
  < ETag: 283fc22575da37e1782df9a9be3a00c0
  <
  Warning: Binary output can mess up your terminal. Use "--output -" to tell
  Warning: curl to output it to your terminal anyway, or consider "--output
  Warning: <FILE>" to save to a file.
• Failure writing output to destination
• Closing connection 0
• TLSv1.2 (OUT), TLS header, Unknown (21):
• TLSv1.2 (OUT), TLS alert, close notify (256):

sudo cloudron-support --troubleshoot

Vendor: Intel(R) Client Systems Product: NUC11PAHi5
Linux: 5.15.0-117-generic
Ubuntu: jammy 22.04
Processor: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz x 8
RAM: 32501008KB
Disk: /dev/sda2 1.7T
[OK] node version is correct
[OK] docker is running
[OK] MySQL is running
[OK] nginx is running
[OK] dashboard cert is valid
[OK] dashboard is reachable via loopback
[OK] box v8.0.3 is running
[OK] netplan is good
[OK] DNS is resolving via systemd-resolved
[OK] Dashboard is reachable via domain name
Domain snip expiry check skipped because whois is not installed. Run 'apt install whois' to check
[OK] unbound is running

[nebulon]

Looks like from the initial attempt of curl using ipv6 and failing, that ipv6 is not working correctly on your system and it falls back to ipv4. This likely causes various issues around the system. Can you double check your ipv6 setup?

Basically if you add --ipv6 it should always fail until ipv6 connections are routed correctly.

[playsia]

ipv6 is no longer working. Note that it was working before because I remember having to enable it because of an update issue in the past.

I'm going to attempt a backup and re-install.

Just went to backup and my backups are now 23mb instead of ~46GB. The apps are still taking up space in system info.

[nebulon]

It should be possible to fixup ipv6 in the current system. A wholesome restore will most likely not be required.

[playsia]

Should be possible yes. I tried a few things but got nowhere and I have limited time to fiddle with things atm. Restoration has everything working again.

Hoping the update doesn't mess up again...

[nebulon]

Glad the restore did work for you. Which VPS provider are you using, as network setup is not really part of Cloudron, but usually done by the server provider cloud-init scripts.

[playsia]

Not solved unfortunately. It updated to 8.0.3 this morning and everything is broken again in the same way.

Cloudron is run on a dedicated NUC in my house (selfhost). Running Ubuntu Server 22.04.4 LTS. Configuration is very simple, just going through the prompts and then immediately install cloudron once I get SSH, but I noticed that it didn't pull an ipv6 address when going through the initial config.

Either my DHCPv6 is broken or it's the problem I found here:
https://ubuntuforums.org/showthread.php?t=2487465
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2025382

I will try the sudo dhclient -6 <interface> fix when I get home, I'm out of time for today and need to rush off to work.

[playsia]

As a note, because the 8.0.3 backups weren't proper the restore I did was to the 7.7.2.

Another thing I could do is use a static ipv6 address? I'd need to figure out what that would be as I am unfamiliar with ipv6.

[playsia]

ok so investigation has discovered... ipv6 not enabled by default on my ISP which is most likely the issue. I have changed ISP since I last had an ipv6 issue with Cloudron.

On the phone to get it resolved atm.

[playsia]

Unfortunately ipv6 is not available for my internet service. This is not an isolated thing, after some quick searching there are plenty of ISPs still around that don't do IPv6.

[nebulon]

In such a case you probably have to disable ipv6 for the Cloudron and then things should work?

[playsia]

@nebulon
I tried the GRUB method here: https://itsfoss.com/disable-ipv6-ubuntu-linux/#2-disable-ipv6-using-grub

Still same issue. Is there a better or recommended way to disable ipv6?

As you've probably noted, 7.7.2 worked with no problems at all, it's only after it updated to 8.0.3

[nebulon]

For ipv6 as such, there is inbound and outbound connectivity. Inbound may not be routed correctly to your server directly depending on your ISP and thus the AAAA records should not be setup, to avoid clients trying this. For outbound, if your other systems like your laptop works fine, your Cloudron should also. Would be good to understand what the difference is.

[playsia]

my HTPC and ROG Ally both have ipv6 addresses and can ping local ipv6. As I said though, my ISP does not support/offer ipv6 so any connection to the internet can only be ipv4.

[nebulon]

I am not familar with what your ISP uses, but if it has no actual ipv6 connectivity, why is your router then setting up ipv6 for your local network? The German ISPs which only offer ipv6 over ipv4 make outbound ipv6 connections work, which is what seems to not work in your case. But as said Cloudron does not do any special network configuration as such, so whatever ubuntu sets up is used, so not sure how to proceed here.

[playsia]

My pihole is setting up (dhcp) ipv6, which I switched on trying to get ipv6 to work. I can turn it back off.

[necrevistonnezr]

Having a pi-hole between Cloudron and the internet is, generally speaking, not a good idea (nor required) from my personal experience. You want to block end devices, particular those you cannot really control (like TVs, mobiles, etc.) but not a server that you can control.
In particular, since you can run Adguard Home on Cloudron and have it serve the rest of the network as an effective adblocker.

[playsia]

Cloudron on ipv4 is static ip and only uses pihole for DNS. I can set the dns server to be something else but that totally isn't the problem here.

Ubuntu Server 22.04 LTS on 7.7.2, without ipv6 = fine
Upgrade to 8.0.3 in same environment = apps don't launch with initially reported error

[nebulon]

Right it is unclear here what the Cloudron update has to do really. All I can think of is the change from not using unbound anymore for local DNS resolving but now relying on whatever the environment the server runs in provides. So all in all all whatever ubuntu would setup without Cloudron should be the same as what your Cloudron should be configured with.

Maybe the previously used unbound just covered up some other networking setup issues, which are now revealed with stock ubuntu network setup.

[playsia]

Really don't think it's DNS, I have a lot of other systems (15+) that are working perfectly fine in this environment.

Trying to get to the bottom of this regression and solve it, having my cloudron stuck on an old version is not a long term solution. What's the command line equivalent of Cloudron downloading the icon?