Unusable application

[nebulon]

So it should work without https:// there so the socket hang up is the issue somehow. Can you telnet via ssh to the registry from that server?

[klawitterb]

Just tried, that just gives me a telnet: Unable to connect to remote host: Connection refused

[nebulon]

If docker.apehost.de is the actual server, then I am able to contact that app from here at least. Does it correctly resolve on the system where it fails?

[klawitterb]

From the server:
ping -n docker.apehost.de -> works
telnet idontexist.apehost.de -> Name or service not known
telnet docker.apehost.de -> connection refused
docker pull docker.apehost.de/path-to-exisiting-image -> Error response from daemon: Get "https://docker.apehost.de/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

From my local machine:
docker pull docker.apehost.de/path-to-exisiting-image -> works just fine

[nebulon]

So then telnet docker.apehost.de 443 also times out I assume. Is hairpin correctly working for you? run cloudron-support --troubleshoot to see this

[klawitterb]

Indeed seems like hairpin doesn't work, from troubleshoot:
[FAIL] Could not load dashboard domain.
Hairpin NAT is not working. Please check if your router supports it

I just tried the good old have you tried turning it off and on again, interestingly troubleshoot now errors with:
[FAIL] Could not load dashboard website with loopback check

[klawitterb]

So decided to upgrade my server to Ubuntu 24 and its working now again.

[nebulon]

Strange how that would have affected it, but well glad it works

[Lanhild]

@nebulon I did the upgrade from Ubuntu 22 to 24 today. I confirm that it fixed the issue for me as well.

[Lanhild]

@nebulon

I have another Cloudron server for which I did the upgrade too. Though, it didn't fix it for this one.

No outstanding errors in cloudron-support's output.

I'm pretty much at a dead-end here

[robi]

@Lanhild after the server upgrade, did the app upgrade yet?

[Lanhild]

@robi Yes

Cloudron Container Registry 2.6.0 (registry 3.1.0)

[james]

Hello @lanhild
Can you send me a direct message with some credentials (username + app password for the docker registry) and the URL for your docker registry app?
I would like to attempt a push from my side to validate if it is not some obscure local issue.

[Lanhild]

@james Done.

[james]

@lanhild and I did a debugging session.
Somehow this issue is connected to https://forum.cloudron.io/topic/15408/openid-configuration-url-timeout

[james]

Hello @lanhild
Can you give me the output of:

uname -nar

from your system that is failing?
We found systems with kernel version <110 are not effected.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1130336

[Lanhild]

@james

6.8.0-110-generic

My server's = 110. The other server, without the issue, has the same kernel.

[james]

Hello @Lanhild
On the other server that "does not have the issue".
Can you run:

# first run - should work
curl 127.0.0.1
# second run - should time out
curl 127.0.0.1

two times.
If the first time works and the second time fails, this system also has this issue.
The issue only is apparent when connecting to itself.

[Lanhild]

@james Indeed, the second curl times out.

But does the original issue from my topic really come from this? I dare have a doubt about it.

As mentioned in a previous reply, the original issue for that other server was solved with the Ubuntu 24 upgrade.

On the other hand, the problem is very much still present on the Cloudron.
Feel free to ask for more debugging cmds.

[james]

Hello @lanhild

@Lanhild said:

But does the original issue from my topic really come from this? I dare have a doubt about it.

It does.
Since Cloudron asks the docker registry app (localhost or more precise, the docker proxy) it times out.
Same with the cloudron cli issue you send me timing out.
All related to that network issue.
We already have an idea how to fix this, without downgrading the kernel.

If you'd like to try, you could apply this patch manually https://git.cloudron.io/platform/box/-/commit/bffc97ab2fedb06100bc507abd0f8b42d343ea6e and then systemctl restart cloudron-firewall