Something broke remote checks.. local OK

[robi]

@girish the DB size is at 62MB keeping data for 180 days. Doesn't look like that's the issue as it works for private IPs but not public IPs.

On a new instance, I even added 1.1.1.1, and it also fails by timing out. So it's not DNS related.

Something about docker/container but not host. (base image?)
So odd.

[girish]

@robi Does disabling telegram notification make things any better ? (the stack trace you posted is the telegram notification crashing)

[robi]

@girish no, because a new instance which doesn't have notifications setup has the same issue.

the crashing seems to be from being unable to reach Telegram.

so this is something with outbound container networking..

[girish]

@robi strange.. might want to report this upstream then. Not sure why something with no notifications is erroring in telegram code.

[robi]

@girish that's the wrong conclusion.

original app with TG notifications has the crash

new test app w/o TG notifications has only the monitors failing

The CL UI also seems slower than usual.
send you an email with logs

[robi]

Things I checked:

Network configuration: The Docker container may not be configured to use the host's network, which would prevent it from accessing public IPs.

Firewall rules: A firewall on the host machine may be blocking incoming connections from the Docker container.

DNS resolution: The Docker container may not be able to resolve public DNS names to IP addresses.

Network address translation (NAT): The host machine may be configured to use NAT, which would prevent the Docker container from accessing public IPs.

Inadequate permissions: The user running the container may not have the necessary permissions to access the host's network.

Network isolation: The network namespace of the container can be isolated from the host machine thus it has only access to the local IPs.

[robi]

Things are getting worse as other container apps are now failing, not able to reach their DB, or failing upgrades (DB migration), etc.

Not sure what's going on, but container networking has something to do with it.

[robi]

Email is no longer accessible, all clients cannot connect.

Checking services, email service is green, but logs are empty.. restarting the mail service fails with a long red error.

Cloudron Error
Command failed: docker run --restart=always -d --name="mail" --net cloudron --net-alias mail --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:2514 --log-opt syslog-format=rfc5424 --log-opt tag=mail -m 429916160 --memory-swap 536870912 --dns 172.18.0.1 --dns-search=. -e CLOUDRON_MAIL_TOKEN="xxxxxx" -e CLOUDRON_RELAY_TOKEN="xxxxxx" -e LOGLEVEL=info -v "/home/yellowtent/boxdata/mail:/app/data" -v "/home/yellowtent/platformdata/addons/mail:/etc/mail:ro" -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995 --label isCloudronManaged=true --read-only -v /run -v /tmp cloudron/mail:3.7.4@sha256:8ddbf13ee3fd479e18923c7bf1370d9d8aa5f12a94cbbda5afac8b5a4af72a28 docker: Error response from daemon: driver failed programming external connectivity on endpoint mail (ddb03fa18c2bf483ec4782d27b2e31a9f774bd5e835b1a15c830d2d38ee82b50): (iptables failed: ip6tables --wait -t filter -A DOCKER ! -i br-e5579f54c902 -o br-e5579f54c902 -p tcp -d fd00:c107:d509::17 --dport 4190 -j ACCEPT: ip6tables: No chain/target/match by that name. (exit status 1)).

I thought to restart docker, but if it fails to restart, all sites will be down ;-/

The plot thickens.

[jdaviescoates]

@robi said in Something broke remote checks.. local OK:

The plot thickens.

SSDNodes having a wobble?

[girish]

@robi I rebooted the server and it fixed up the iptable issue atleast. mail container seems to be back now and I am also able wget/curl etc from the uptime kuma container.

I am not sure what messed up with you iptables though. Did you upgrade recently? I do see that there are some non-cloudron docker container in the server (like watchtower), not sure what effect they have.

[robi]

@girish Did not think to look at iptables, ugh.

Yes, stuff works at least! Thank you.

There were some package upgrades. But don't remember which.

Ooh, watchtower from 2 years ago? Looks like I forgot about playing with that. Nope, no effect, but it would be nice to see those in the Cloudron UI.

Happy Sunday

[girish]

@robi ah, indeed. Maybe you installed something a long time ago! It's running for 2 years lol

[robi]

@girish done https://forum.cloudron.io/topic/8487/enhance-app-proxy-app-with-path-and-port-support