Prometheus can't reach node exporter (Blocked traffic)

[abset]

Hi everyone,

I'm facing an issue where Prometheus is unable to scrape Node Exporter running on the host at http://172.18.0.1:9100/metrics. The request times out with the following error:

Error scraping target: Get "http://172.18.0.1:9100/metrics": context deadline exceeded

From the syslog, I see the following dropped packet:

2025-02-11T17:20:13.907565+00:00 my kernel: Packet dropped: IN=br-xxx OUT= MAC=xxx SRC=172.18.18.148 DST=172.18.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=54789 DF PROTO=TCP SPT=44028 DPT=9100 WINDOW=64240 RES=0x00 SYN URGP=0

This confirms that the firewall is dropping packets from the Prometheus container (172.18.18.148) when trying to reach Node Exporter on 172.18.0.1:9100.

I’ve tried to manually add an iptables rule without success:

iptables -I DOCKER-USER -s 172.18.0.0/16 -d 172.18.0.1 -p tcp --dport 9100 -j ACCEPT

Testing connectivity from the Prometheus container:

docker exec -it $(docker ps | grep prometheus | awk '{print $1}') sh
curl -v http://172.18.0.1:9100/metrics

Result: The request hangs, confirming the firewall is blocking access.

This issue started after upgrading to Cloudron 8.2.3.

How can I persistently allow Prometheus to access 172.18.0.1:9100 within Cloudron’s firewall?

Any help would be greatly appreciated! Thanks in advance.

[joseph]

Trying to understand the setup here. Prometheus itself is an app on Cloudron and the run node exporter is a docker container directly on Cloudron?

We don't support running external containers this way, but if you are OK with that... Are you running node_exporter with --net host ? You can try --net cloudron instead. The cloudron internal network is where all the apps run. Apps should be able to talk to each other without firewall rules.

[abset]

Thanks for your reply!

To clarify, Prometheus is running as a Cloudron app, and Node Exporter is running directly on the host (not in a container). Prometheus needs to scrape metrics from http://172.18.0.1:9100/metrics, but Cloudron's firewall is blocking this traffic.

Since Node Exporter is on the host, I can't use --net cloudron. Is there a recommended way to allow Cloudron apps to access host services?

[joseph]

The apps cannot access localhost (they run on a custom network) and they can only access specific ports on the bridge (172.18.0.1).

Generally, we don't support these kinds of custom setups. But hey, it's your server If you are feeling brave:

iptables -t filter -A CLOUDRON -p tcp -s 172.18.0.0/16 -d 172.18.0.1 --dport 9100 -j ACCEPT

That allows containers to access port 9100 on the bridge. You have to put this in some start up file to make it persist reboots. Maybe a service that runs after cloudron-firewall.service.

[abset]

Thanks for your reply.

We are monitoring some Cloudron metrics with Prometheus (e.g. disk space).

What's implied from what you say makes sense from the security point of view.

We have to think about how to monitor Cloudron using different methods.