Use floating IP address only for outbound Cloudron Docker container communication

[d19dotca]

@girish It’s the former, that’s correct.

Basically, Gmail was refusing emails from my server due to a mismatch of FCrDNS (forward-confirmed reverse DNS). This happened after I had migrated to a new dedicated server at OVH with multiple IP addresses as I had started to use a floating IP address for IPv4 l. So the server now had two IPv4 addresses plus the IPv6 address. In Cloudron, I had set it to specifically only use the floating IPv4 address and disable IPv6, but that’s when I learned that doesn’t really do anything beyond what to set the DNS records with.

After a little investigation from seeing the Gmail delivery failures, I realized emails were being sent to Gmail using random IP addresses from my server rather than the IP address that I had manually set on the Cloudron network page to use. This meant that the IP address used in the DNS record for the mail server didn’t always match the outgoing IP from my server which is why Gmail was rejecting the message.

I think the disconnect for me was not realizing that the network tab didn’t control the IP addresses used for any outbound communication (mainly the mail server though).

So I think it’s critical that we have a way to determine the outbound IP to use for the mail container at the very least.

I worked around this by creating a service that runs on boot and basically adds to the firewall rules to for Docker to use only my preferred IP address for outbound traffic. But this seems temporary and almost “wrong” somehow even though it does the trick. I worry that I can’t rely on it long term and would think this logic needs to be added into Cloudron where perhaps on the Network page we get the chance to configure the outbound IP address for traffic, or even if it’s just unique to the mail container because that’s probably the only container where this is an issue.

I hope that helps clarify.

[jdaviescoates]

@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:

I had started to use a floating IP address for IPv4

I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?

[d19dotca]

@jdaviescoates said in Use floating IP address only for outbound Cloudron Docker container communication:

@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:

I had started to use a floating IP address for IPv4

I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?

That’s basically what I’m trying to do. But instead of using the IP address that comes with the servers I am trying to use the IP address that is floating. The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service. While I control most of the domains DNS records for my clients, there are a few outside of my control and it’s a pain to ask them to manually update the DNS records because sometimes it takes them days to complete it which makes data migrations or failovers difficult to do for them.

Basically, I’m paying a buck a month for an additional IP for peace of mind to use when disaster happens, and unfortunately today Cloudron (or perhaps more specifically Docker) is using all available IP addresses to send traffic rather than the only one I set on Cloudron to use.

Hopefully that helps clarify that, I’m rushing a bit so let me know if I confused anything. Haha.

[jdaviescoates]

@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:

The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service.

Ah, I see. I was indeed wondering why

[girish]

@d19dotca did the original routing trick you did sort out the issue?

Since everything is dockerized, mail container doesn't see the external IPs . So, we cannot select the IP address in haraka config. We have to use routing magic.

So far, we don't actually have real "networking" code in Cloudron since we focus primarily on app deployment . The IPv6 selector in fact is only configuring the AAAA records and nothing else. No route handling or disabling interfaces. Just saying that, since this is a complex new topic to implement in Cloudron code.

[d19dotca]

@girish Yes, I no longer got the Gmail issues when I created my own system service to implement the iptables rule. So it seemed to do the trick.

The IP tables rule to add is really just this: iptables -t nat -I POSTROUTING -s 172.18.0.0/16 -o enp3s0f0 -j SNAT --to-source {FLOATING_IP}

Where FLOATING_IP is really just replaced with whatever the recognized IP address is that’s used in the DNS records for the MX record. I supposed it could be further improved to only be applicable to the mail container rather than all Docker traffic. And of course the interface would have to be dynamic too.

I guess an alternative is for me to create additional MX records with the other IP addresses but then it’s manually done and prone to mistakes/issues.

In my opinion, I think we really need an option to select the outbound IP interface using Cloudron for the mail component, in order to avoid the Gmail issues (and any other provider who will use FCrDNS for verifying or rejecting emails in the future). I recognize this may not be a common concern as most people probably only have the one IP address of each type and so the DNS records if setup automatically by Cloudron will use both the IPv4 and IPv6 address, but for those of us who use a floating/failover IP addresses that we want to be the one true IP address being used, this becomes an issue without that workaround in place.

[d19dotca]

Just a quick update... I found a slight alternative to this that I wanted to share. Please let me know though if you see any issues with this at all that I may be overlooking. So far everything seems good though.

First step: I created my own bash script named d19-cloudron-snat.sh. What this script does is take the Cloudron Docker subnet and creates the iptables rules to use which go to the failover/additional public IP that has the MX record and reverse DNS set for it:

sudo tee /usr/local/bin/d19-cloudron-snat.sh > /dev/null <<'EOF'
#!/bin/bash
set -e

CLOUDRON_SUBNET="172.18.0.0/16"
INTERFACE="$(ip -o -4 route show to default | awk '{print $5}')"
SNAT_IP="149.56.126.160"

while iptables -t nat -C POSTROUTING -s "$CLOUDRON_SUBNET" -o "$INTERFACE" -j SNAT --to-source "$SNAT_IP" 2>/dev/null; do
  iptables -t nat -D POSTROUTING -s "$CLOUDRON_SUBNET" -o "$INTERFACE" -j SNAT --to-source "$SNAT_IP"
done
iptables -t nat -I POSTROUTING 1 -s "$CLOUDRON_SUBNET" -o "$INTERFACE" -j SNAT --to-source "$SNAT_IP"
netfilter-persistent save
EOF

sudo chmod +x /usr/local/bin/d19-cloudron-snat.sh

Then I have a service that is a oneshot service which simply invokes the script above when the Docker service is running:

sudo tee /etc/systemd/system/d19-cloudron-snat.service > /dev/null <<EOF
[Unit]
Description=SNAT all Cloudron Docker traffic to 149.56.126.160
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/d19-cloudron-snat.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target
EOF

This results in the following:

1. Out of the two IPv4 addresses on my dedicated server, it sends all outbound traffic from the Cloudron Docker network over the failover/additional IPv4 address that I want to use (and matches the IPv4 address I have set in the Cloudron Network page) instead of the default assigned IPv4 address.

2. I also had run the following commands to disable IPv6 for outbound mail ports only:

sudo ip6tables -I DOCKER-USER 1 -p tcp --dport 25 -j REJECT
sudo ip6tables -I DOCKER-USER 2 -p tcp --dport 465 -j REJECT
sudo ip6tables -I DOCKER-USER 3 -p tcp --dport 587 -j REJECT

Originally by the way, I had the above working but for just the mail container only, however the more I thought about it, it seemed like it'd be easier for troubleshooting if I had directed all outbound traffic from the Docker Cloudron network to the floating IP address instead of just the mail container. That way things are at least consistent. Now if I run something like curl -4 https://ifconfig.me in any of the Docker containers for Cloudron, it will always return the floating IP as intended.

All of this seems to work without issue, but would love any insight you have into this too for me in case I'm accidentally causing any issues.

[Kubernetes]

@d19dotca Thanks for sharing your solution — it’s a solid and pragmatic setup! Just two key points to consider:

1. Docker Subnet Might Change in Future

Your script currently hardcodes 172.18.0.0/16 as the Docker subnet. While this is the default for Cloudron, it’s technically subject to change — either through manual config changes or future Cloudron updates. To make your script more robust, you could dynamically retrieve the subnet like this:

CLOUDRON_SUBNET=$(docker network inspect cloudron --format '{{ (index .IPAM.Config 0).Subnet }}')

This ensures your SNAT rule always uses the correct range.

2. Host-Level Traffic Still Uses Default IP

Your setup correctly SNATs all container traffic, but be aware that outbound connections from the Cloudron host itself (e.g. system updates, wget, DNS lookups) will still use the primary server IP unless you apply a similar SNAT rule for host traffic. If consistency across all outbound traffic matters (e.g. for whitelisting or compliance), you might want to extend the approach accordingly.

[d19dotca]

@Kubernetes Absolutely agreed, and good points! I modified it slightly for the purposes of this forum post to just show the real values to follow easier, but yes I did have it set dynamically in my cloud-init script to automatically calculate the interface and the Docker subnet for Cloudron which is nice.

My hope is that the Cloudron team can consider using something like this (dynamically) where there's perhaps a feature checkbox added to the Network page (if needed) where we can choose to have all outbound traffic from the Cloudron network use the particular IP address set in Cloudron, as that will help a lot for outbound mail (especially to Gmail I believe due to their more aggressive checks and rejections).

For now though, the solution above is what has helped me and should work for any others who run into a similar situation.

Good note on the host-level traffic too. I think since the server is fully dedicated to Cloudron I left the rest of it and only cared about the Cloudron network traffic, but it is a good reminder as I suppose doing it system-wide may be helpful to keep everything consistent across the entire system rather than just Cloudron's Docker network.

[girish]

Thanks for sharing! I think if this works. But maybe the systemd service needs to be adjusted to run after cloudron-firewall.service (which also after docker.service) ?

[d19dotca]

@girish said in Use floating IP address only for outbound Cloudron Docker container communication:

Thanks for sharing! I think if this works. But maybe the systemd service needs to be adjusted to run after cloudron-firewall.service (which also after docker.service) ?

Oh that’s interesting, I can definitely make that change then just to be safe. I haven’t noticed any issues so far but I’ll adjust it regardless to test that out.

Hoping something like this can be added to Cloudron more formally down the road so we don’t have to manually set these scripts up.