Use floating IP address only for outbound Cloudron Docker container communication
-
@joseph Yeah, I tried a few different ways including modifying the Netplan 51-cloud-init.yaml file which is used for adding the additional/floating IP address (this works okay for adding the IP itself), but no matter what I tried (with the help of AI too), nothing allowed Docker to use the floating IP for all outbound communication until it suggested the service to add that rule persistently which seemed to work okay thankfully. It just feels like this is "wrong" somehow though even though it works. I want to believe there's a much easier way to do this.
I'm also surprised nobody else has had this issue, although I suppose most people aren't using floating IPs, and those that do either don't run into the issue or already knew how to work around it which I didn't until today after hours of troubleshooting, haha. Or maybe it's something that Gmail is recently applying in their MTA checks. I'm not sure how I haven't run into this myself before.
-
@robi I think the question still becomesā¦ how does one apply that change to the mail container only to send over one IP? The change I made works fine and applies to all containers which is okay overall. But it just doesnāt seem ārightā, I keep thinking Iām overlooking something.
-
D d19dotca referenced this topic
-
Just wanted to follow up on this. Paging @girish , in case this is something you think we can try to include for the next Cloudron version. I think youāre busy working on 9.0 so itād be good if we can perhaps include this mail / networking improvement. I found the following in case this helps for Haraka where we can supposedly set the specific IP address to use: https://haraka.github.io/core/Outbound#outbound-ip-address
It seems like we can set a hook to determine the IP address to use for outbound mail.
IMO, it should be automatically set to the IP address detected by Cloudron and perhaps more specifically the IP address used on the MX DNS record so that we can be better with the Gmail delivery in particular.
-
@d19dotca is this for the case where you have multiple IP addresses or you want to disable IPv6 ? I think it's the former since that is the original comment in the thread.
So far, all this is just left to the end user to configure on their system. The OS has to be configured to route all traffic via whichever IP you want. But I understand this is complicated.. To be fair, it's this way even without Cloudron. Just having a specific docker container route via a specific IP is not easy.
-
@d19dotca is this for the case where you have multiple IP addresses or you want to disable IPv6 ? I think it's the former since that is the original comment in the thread.
So far, all this is just left to the end user to configure on their system. The OS has to be configured to route all traffic via whichever IP you want. But I understand this is complicated.. To be fair, it's this way even without Cloudron. Just having a specific docker container route via a specific IP is not easy.
@girish Itās the former, thatās correct.
Basically, Gmail was refusing emails from my server due to a mismatch of FCrDNS (forward-confirmed reverse DNS). This happened after I had migrated to a new dedicated server at OVH with multiple IP addresses as I had started to use a floating IP address for IPv4 l. So the server now had two IPv4 addresses plus the IPv6 address. In Cloudron, I had set it to specifically only use the floating IPv4 address and disable IPv6, but thatās when I learned that doesnāt really do anything beyond what to set the DNS records with.
After a little investigation from seeing the Gmail delivery failures, I realized emails were being sent to Gmail using random IP addresses from my server rather than the IP address that I had manually set on the Cloudron network page to use. This meant that the IP address used in the DNS record for the mail server didnāt always match the outgoing IP from my server which is why Gmail was rejecting the message.
I think the disconnect for me was not realizing that the network tab didnāt control the IP addresses used for any outbound communication (mainly the mail server though).
So I think itās critical that we have a way to determine the outbound IP to use for the mail container at the very least.
I worked around this by creating a service that runs on boot and basically adds to the firewall rules to for Docker to use only my preferred IP address for outbound traffic. But this seems temporary and almost āwrongā somehow even though it does the trick. I worry that I canāt rely on it long term and would think this logic needs to be added into Cloudron where perhaps on the Network page we get the chance to configure the outbound IP address for traffic, or even if itās just unique to the mail container because thatās probably the only container where this is an issue.
I hope that helps clarify.
-
@girish Itās the former, thatās correct.
Basically, Gmail was refusing emails from my server due to a mismatch of FCrDNS (forward-confirmed reverse DNS). This happened after I had migrated to a new dedicated server at OVH with multiple IP addresses as I had started to use a floating IP address for IPv4 l. So the server now had two IPv4 addresses plus the IPv6 address. In Cloudron, I had set it to specifically only use the floating IPv4 address and disable IPv6, but thatās when I learned that doesnāt really do anything beyond what to set the DNS records with.
After a little investigation from seeing the Gmail delivery failures, I realized emails were being sent to Gmail using random IP addresses from my server rather than the IP address that I had manually set on the Cloudron network page to use. This meant that the IP address used in the DNS record for the mail server didnāt always match the outgoing IP from my server which is why Gmail was rejecting the message.
I think the disconnect for me was not realizing that the network tab didnāt control the IP addresses used for any outbound communication (mainly the mail server though).
So I think itās critical that we have a way to determine the outbound IP to use for the mail container at the very least.
I worked around this by creating a service that runs on boot and basically adds to the firewall rules to for Docker to use only my preferred IP address for outbound traffic. But this seems temporary and almost āwrongā somehow even though it does the trick. I worry that I canāt rely on it long term and would think this logic needs to be added into Cloudron where perhaps on the Network page we get the chance to configure the outbound IP address for traffic, or even if itās just unique to the mail container because thatās probably the only container where this is an issue.
I hope that helps clarify.
@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:
I had started to use a floating IP address for IPv4
I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?
-
@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:
I had started to use a floating IP address for IPv4
I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?
@jdaviescoates said in Use floating IP address only for outbound Cloudron Docker container communication:
@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:
I had started to use a floating IP address for IPv4
I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?
Thatās basically what Iām trying to do. But instead of using the IP address that comes with the servers I am trying to use the IP address that is floating. The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service. While I control most of the domains DNS records for my clients, there are a few outside of my control and itās a pain to ask them to manually update the DNS records because sometimes it takes them days to complete it which makes data migrations or failovers difficult to do for them.
Basically, Iām paying a buck a month for an additional IP for peace of mind to use when disaster happens, and unfortunately today Cloudron (or perhaps more specifically Docker) is using all available IP addresses to send traffic rather than the only one I set on Cloudron to use.
Hopefully that helps clarify that, Iām rushing a bit so let me know if I confused anything. Haha.
-
@jdaviescoates said in Use floating IP address only for outbound Cloudron Docker container communication:
@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:
I had started to use a floating IP address for IPv4
I'm not really following this too closely, but isn't that the root of the problem? I may be missing something obvious, but one not just used a fixed IP? Wouldn't that solve your problem?
Thatās basically what Iām trying to do. But instead of using the IP address that comes with the servers I am trying to use the IP address that is floating. The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service. While I control most of the domains DNS records for my clients, there are a few outside of my control and itās a pain to ask them to manually update the DNS records because sometimes it takes them days to complete it which makes data migrations or failovers difficult to do for them.
Basically, Iām paying a buck a month for an additional IP for peace of mind to use when disaster happens, and unfortunately today Cloudron (or perhaps more specifically Docker) is using all available IP addresses to send traffic rather than the only one I set on Cloudron to use.
Hopefully that helps clarify that, Iām rushing a bit so let me know if I confused anything. Haha.
@d19dotca said in Use floating IP address only for outbound Cloudron Docker container communication:
The reason for that is it means if I change servers (whether for upgrades or failover) then there is no real interruption in service.
Ah, I see. I was indeed wondering why
-
@d19dotca did the original routing trick you did sort out the issue?
Since everything is dockerized, mail container doesn't see the external IPs . So, we cannot select the IP address in haraka config. We have to use routing magic.
So far, we don't actually have real "networking" code in Cloudron since we focus primarily on app deployment . The IPv6 selector in fact is only configuring the AAAA records and nothing else. No route handling or disabling interfaces. Just saying that, since this is a complex new topic to implement in Cloudron code.
-
@d19dotca did the original routing trick you did sort out the issue?
Since everything is dockerized, mail container doesn't see the external IPs . So, we cannot select the IP address in haraka config. We have to use routing magic.
So far, we don't actually have real "networking" code in Cloudron since we focus primarily on app deployment . The IPv6 selector in fact is only configuring the AAAA records and nothing else. No route handling or disabling interfaces. Just saying that, since this is a complex new topic to implement in Cloudron code.
@girish Yes, I no longer got the Gmail issues when I created my own system service to implement the iptables rule. So it seemed to do the trick.
The IP tables rule to add is really just this:
iptables -t nat -I POSTROUTING -s 172.18.0.0/16 -o enp3s0f0 -j SNAT --to-source {FLOATING_IP}Where FLOATING_IP is really just replaced with whatever the recognized IP address is thatās used in the DNS records for the MX record. I supposed it could be further improved to only be applicable to the mail container rather than all Docker traffic. And of course the interface would have to be dynamic too.
I guess an alternative is for me to create additional MX records with the other IP addresses but then itās manually done and prone to mistakes/issues.
In my opinion, I think we really need an option to select the outbound IP interface using Cloudron for the mail component, in order to avoid the Gmail issues (and any other provider who will use FCrDNS for verifying or rejecting emails in the future). I recognize this may not be a common concern as most people probably only have the one IP address of each type and so the DNS records if setup automatically by Cloudron will use both the IPv4 and IPv6 address, but for those of us who use a floating/failover IP addresses that we want to be the one true IP address being used, this becomes an issue without that workaround in place.
