IPv6 Cloudron Install and Update Issue

[playsia]

I was writing up the problem I had, which you can see below the bold line, and then I had one more idea to try. I am still writing this post because google search results was.. an adventure I don't wish upon other people.

The problem seemed to be related to ipv6 and DNS, I run my DNS on a PiHole with unbound. I temporarily set some upstream servers for IPv6 and then started the process again, install Ubuntu Server Jammy into cloudron setup.

Not only did doing this mean the cloudron setup worked, but now my original cloudron can update again.

So now, I should configure unbound on my PiHole to do IPv6, haven't looked into this yet.

Maybe the setup script should check for ipv6 connectivity and have an ipv4 fallback?

Original Post
-----------------------

My old VM was having trouble updating (same issue, I believe it's IPv6 related) and I am looking to migrate to new hardware anyway. Current cloudron is on 7.3.6 and wont upgrade because of the same issue. It's otherwise working though. I was going to just follow this.

I installed a fresh Ubuntu Server Jammy 22.04. Gave it a static IPv4, then went ahead with the install with --version 7.3.6

After not much happened for a while I had a look at cloudron-setup.log and found this:

rebuilt dependencies successfully
2023-05-17T13:20:30 ==> installer: downloading new addon images
2023-05-17T13:20:30 ==> installer: Pulling docker images: cloudron/base:3.2.0@sha256:ba1d566164a67c266782545ea9809dc611c4152e27686fd14060332dd88263ea cl>
Error response from daemon: Get "https://registry.ipv6.docker.com/v2/": dial tcp [2600:1f18:2148:bc02:cfd8:db68:ea1f:277c]:443: connect: network is unreachab>
2023-05-17T13:20:33 ==> installer: Could not pull cloudron/base:3.2.0@sha256:ba1d566164a67c266782545ea9809dc611c4152e27686fd14060332dd88263ea
Error response from daemon: Get "https://registry.ipv6.docker.com/v2/": dial tcp [2600:1f18:2148:bc01:a3b0:6734:c617:7c5c]:443: connect: network is unreachab>
2023-05-17T13:20:38 ==> installer: Could not pull cloudron/base:3.2.0@sha256:ba1d566164a67c266782545ea9809dc611c4152e27686fd14060332dd88263ea
<repeat forever>

Is there a way to force it into ipv4 mode?

[girish]

@playsia said in IPv6 Cloudron Install and Update Issue:

The problem seemed to be related to ipv6 and DNS, I run my DNS on a PiHole with unbound. I temporarily set some upstream servers for IPv6 and then started the process again, install Ubuntu Server Jammy into cloudron setup.

I have been trying to debug failures in Cloudron installations (and updates) for a while now. I narrowed down the issue to IPv6 and DNS as well.

To give some history: We put in some IPv6/IPv4 detection logic in 7.x . The idea was to have Cloudron work on pure IPv6 servers (i.e no IPv4). There exists some pure IPv6 server providers and also you get charged for IPv4 these days. In case you are wondering why we need to detect anything at all and not leave it to the OS, it's because many basic things (which we take for granted) don't actually support IPv6 even in 2023. npm does not support IPv6 out of the box without some magic flags (fixed only in node 16), docker hub registry needs a different endpoint for ipv6 and ipv6 (this got fixed 2 months ago) etc. In 7.5 (next release), we have already removed the detection logic since the services now finally support IPv6.

Unfortunately, there still seems to be some problem with unbound, our internal DNS server. If a server has IPv6 support but has IPv6 connectivity issues, it just behaves randomly. We don't understand why but https://github.com/NLnetLabs/unbound/issues/422 is the same issue. It got closed because upstream devs don't think it's a problem (but it is).

In your unbound config, in /etc/unbound/unbound.conf , you can add do-ip6: no and this resolves the problem. We don't want to, however, fix this in some other way because disabling ip6 means we have to forego our dream of IPv6 servers But let's see, if we can't find a fix, this is what we will do for 7.5.

[girish]

@playsia said in IPv6 Cloudron Install and Update Issue:

Is there a way to force it into ipv4 mode?

We have been trying to avoid putting this flag and somehow make it just work. So far, the solution eludes us...

Thanks for your report btw! It will help out others facing this issue.

[playsia]

@girish said in IPv6 Cloudron Install and Update Issue:

I have been trying to debug failures in Cloudron installations (and updates) for a while now. I narrowed down the issue to IPv6 and DNS as well.

In case it helps:
I have always just used IPv4 because as you alluded to, lots of stuff doesn't work on IPv6 and my life was easier
In this environment: My cloudron first failed to update on 19th April, going from 7.3.6 to 7.4.1. "Cloudron update failed 04/19/2023 Failed to update Cloudron: Task 4248 timed out"