Use floating IP address only for outbound Cloudron Docker container communication

[d19dotca]

Hello,

I am wanting to make it so that all Cloudron / Docker containers are using a certain outbound IP address as the source instead of a different IPv4 address.

Let's say my IPv4 interface has 1.2.3.4 by default, and I have a floating IP of 6.7.8.9. Right now, it seems like the dedicated server routes traffic through all available IPs at seemingly random. I want to force it to use 6.7.8.9 instead from the Cloudron Docker containers.

Currently, I've been able to do this by creating a system service at /etc/systemd/system/docker-snat.service (for example) to add in a firewall rule. The service looks like this:

[Unit]
Description=Insert SNAT rule for Docker container traffic
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/iptables -t nat -I POSTROUTING -s 172.18.0.0/16 -o enp3s0f0 -j SNAT --to-source {FLOATING_IP}
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The context for this request is to do with mail and avoiding issues with FCrDNS (forward-confirmed reverse DNS). Specifically, Gmail started blocking delivery from some emails due to the IP address resolving to my mail server not matching the IP address used to send the message. Cloudron is using the floating IP address when it sets the DNS records, but since that doesn't appear to be the only IP being used for outbound mail, some mail are getting rejected.

Is there a better way to achieve this? I added a floating IP recently to the server and that's what triggered this issue as any mail being sent from the original IPs failed because the DNS records now only contained the floating IP instead for the mail server.

[joseph]

This seems OK to me . But can this not be done by normal routing tables? (I guess the answer to your question is a general linux question and not related to Cloudron specifically)

[joseph]

FWIW, mistral also says we have to create a postrouting SNAT rule as you posted

[d19dotca]

This post is deleted!

[d19dotca]

@joseph Yeah, I tried a few different ways including modifying the Netplan 51-cloud-init.yaml file which is used for adding the additional/floating IP address (this works okay for adding the IP itself), but no matter what I tried (with the help of AI too), nothing allowed Docker to use the floating IP for all outbound communication until it suggested the service to add that rule persistently which seemed to work okay thankfully. It just feels like this is "wrong" somehow though even though it works. I want to believe there's a much easier way to do this.

I'm also surprised nobody else has had this issue, although I suppose most people aren't using floating IPs, and those that do either don't run into the issue or already knew how to work around it which I didn't until today after hours of troubleshooting, haha. Or maybe it's something that Gmail is recently applying in their MTA checks. I'm not sure how I haven't run into this myself before.

[robi]

Why not have the mailer only use a specified MX IP?

Let the system spray the rest.

[d19dotca]

@robi I think the question still becomes… how does one apply that change to the mail container only to send over one IP? The change I made works fine and applies to all containers which is okay overall. But it just doesn’t seem ‘right’, I keep thinking I’m overlooking something.

[d19dotca]

Just wanted to follow up on this. Paging @girish , in case this is something you think we can try to include for the next Cloudron version. I think you’re busy working on 9.0 so it’d be good if we can perhaps include this mail / networking improvement. I found the following in case this helps for Haraka where we can supposedly set the specific IP address to use: https://haraka.github.io/core/Outbound#outbound-ip-address

It seems like we can set a hook to determine the IP address to use for outbound mail.

IMO, it should be automatically set to the IP address detected by Cloudron and perhaps more specifically the IP address used on the MX DNS record so that we can be better with the Gmail delivery in particular.

[girish]

@d19dotca is this for the case where you have multiple IP addresses or you want to disable IPv6 ? I think it's the former since that is the original comment in the thread.

So far, all this is just left to the end user to configure on their system. The OS has to be configured to route all traffic via whichever IP you want. But I understand this is complicated.. To be fair, it's this way even without Cloudron. Just having a specific docker container route via a specific IP is not easy.

[d19dotca]

@girish It’s the former, that’s correct.

Basically, Gmail was refusing emails from my server due to a mismatch of FCrDNS (forward-confirmed reverse DNS). This happened after I had migrated to a new dedicated server at OVH with multiple IP addresses as I had started to use a floating IP address for IPv4 l. So the server now had two IPv4 addresses plus the IPv6 address. In Cloudron, I had set it to specifically only use the floating IPv4 address and disable IPv6, but that’s when I learned that doesn’t really do anything beyond what to set the DNS records with.

After a little investigation from seeing the Gmail delivery failures, I realized emails were being sent to Gmail using random IP addresses from my server rather than the IP address that I had manually set on the Cloudron network page to use. This meant that the IP address used in the DNS record for the mail server didn’t always match the outgoing IP from my server which is why Gmail was rejecting the message.

I think the disconnect for me was not realizing that the network tab didn’t control the IP addresses used for any outbound communication (mainly the mail server though).

So I think it’s critical that we have a way to determine the outbound IP to use for the mail container at the very least.

I worked around this by creating a service that runs on boot and basically adds to the firewall rules to for Docker to use only my preferred IP address for outbound traffic. But this seems temporary and almost “wrong” somehow even though it does the trick. I worry that I can’t rely on it long term and would think this logic needs to be added into Cloudron where perhaps on the Network page we get the chance to configure the outbound IP address for traffic, or even if it’s just unique to the mail container because that’s probably the only container where this is an issue.

I hope that helps clarify.

[jdaviescoates]

@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:

I had started to use a floating IP address for IPv4

I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?

[d19dotca]

@jdaviescoates said in Use floating IP address only for outbound Cloudron Docker container communication:

@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:

I had started to use a floating IP address for IPv4

I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?

That’s basically what I’m trying to do. But instead of using the IP address that comes with the servers I am trying to use the IP address that is floating. The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service. While I control most of the domains DNS records for my clients, there are a few outside of my control and it’s a pain to ask them to manually update the DNS records because sometimes it takes them days to complete it which makes data migrations or failovers difficult to do for them.

Basically, I’m paying a buck a month for an additional IP for peace of mind to use when disaster happens, and unfortunately today Cloudron (or perhaps more specifically Docker) is using all available IP addresses to send traffic rather than the only one I set on Cloudron to use.

Hopefully that helps clarify that, I’m rushing a bit so let me know if I confused anything. Haha.

[jdaviescoates]

@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:

The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service.

Ah, I see. I was indeed wondering why

[girish]

@d19dotca did the original routing trick you did sort out the issue?

Since everything is dockerized, mail container doesn't see the external IPs . So, we cannot select the IP address in haraka config. We have to use routing magic.

So far, we don't actually have real "networking" code in Cloudron since we focus primarily on app deployment . The IPv6 selector in fact is only configuring the AAAA records and nothing else. No route handling or disabling interfaces. Just saying that, since this is a complex new topic to implement in Cloudron code.

[d19dotca]

@girish Yes, I no longer got the Gmail issues when I created my own system service to implement the iptables rule. So it seemed to do the trick.

The IP tables rule to add is really just this: iptables -t nat -I POSTROUTING -s 172.18.0.0/16 -o enp3s0f0 -j SNAT --to-source {FLOATING_IP}

Where FLOATING_IP is really just replaced with whatever the recognized IP address is that’s used in the DNS records for the MX record. I supposed it could be further improved to only be applicable to the mail container rather than all Docker traffic. And of course the interface would have to be dynamic too.

I guess an alternative is for me to create additional MX records with the other IP addresses but then it’s manually done and prone to mistakes/issues.

In my opinion, I think we really need an option to select the outbound IP interface using Cloudron for the mail component, in order to avoid the Gmail issues (and any other provider who will use FCrDNS for verifying or rejecting emails in the future). I recognize this may not be a common concern as most people probably only have the one IP address of each type and so the DNS records if setup automatically by Cloudron will use both the IPv4 and IPv6 address, but for those of us who use a floating/failover IP addresses that we want to be the one true IP address being used, this becomes an issue without that workaround in place.