SMTP using wrong IP address on interface with multiple addresses

[d19dotca]

I recently setup a new dedicated server and noticed that Cloudron is using the default assigned IP address for sending email instead of the static IP address I assigned to both the NIC and the Cloudron configuration under Networking.

I haven't had this issue before, normally adding a failover IP is easy (adjust a few things in the cloud-init YAML) and then assign that IP to Cloudron's Networking tab as a static IP.

But currently it's going over the wrong IP address. Any recommendations? It's an unusual issue for me. Restarted the server a few times, no difference made by the way.

Any help would be appreciated.

For context, let's use the following example IP addresses:

• IP address 10.10.10.10 was assigned to my server by default from the provider
• IP address 20.20.20.20 is the one I want to use as it's a failover IP which I added to the server. The server is definitely listening on it as all DNS traffic point to this IP address and it works successfully. If I powed off the last server that had this IP, the apps continue to work so I know for a fact it's hitting the new server.

When I send outgoing email, I get the odd bounce (mostly from Microsoft) that lists the 10.10.10.10 address as the IP address which is blocked. How can I get my emails to send over 20.20.20.20 instead?

[Kubernetes]

@d19dotca Not sure, but I think the problem is, that from SMTP Server perspective your server does come with 10.10.10.10

The 20.20.20.20 is set for incoming traffic to your server, but your server is outgoing with 10.10.10.10

[d19dotca]

This post is deleted!

[d19dotca]

@Kubernetes - Right, but I'm wondering why that is. In the past, the servers I've had with OVH I've done the same things by adding in the additional IP address and setting that additional IP as a static IP for Cloudron. This has generally bypassed any bounce issues associated with the auto-assigned IP address. But for some reason this time it seems to always favour the auto-assigned IP instead of the static one I've added in cloud-init and Cloudron. It's very odd.

[Kubernetes]

@d19dotca Is your default gateway on the server the interface with 20.20.20.20 oder with 10.10.10.10?

[d19dotca]

@Kubernetes - Good question! I don’t recall ever setting a different gateway for a failover IP, so that hasn’t been changed. It’d be using the default gateway which would be set by OVH. I’ll explore that area further though.

[d19dotca]

Okay I got a little bit further, as I can get my IP address changed using something like this in cloud-init:

network:
    version: 2
    ethernets:
        enp5s0f0:
            accept-ra: false
            addresses: [{floatingIP}]
            dhcp4: true
            match:
                macaddress: {redacted}
            nameservers:
                addresses: [{redacted}]
            set-name: enp5s0f0
            routes:
                - to: 0.0.0.0/0
                  via: {gatewayIP}  # Gateway IP address
                  from: {floatingIP}
                  metric: 100
                  on-link: true

This survives a restart and when I run a command like curl -sS ifconfig.me, I will now see my floating IP address instead of the main IP address assigned by DHCP.

However, while this works at the system level, it seems like emails from Cloudron still go over the main IP instead of my floating IP, and I'm at a loss as to why.

Can confirm this by running this command: docker run --rm --network bridge curlimages/curl -sS ifconfig.me

[robi]

Would this be from the interface/IP the mail server binds to?

[d19dotca]

@robi there’s only one interface that I can configure, and multiple IPs are attached to the same interface. So I don’t think it’s so much an interface issue as a selection of which IP to use on the interface.

[d19dotca]

@girish - is there a way to define which IP address is used as the source when sending outbound traffic (i.e. mail)? I'm really struggling with this. My gut tells me this is coming from an OVH DHCP issue, but I've at least gotten the host to send all traffic outbound with a source of my floating IP, however it's now the Cloudron Docker containers that don't seem to be adhering to that, which is where I'm sure OVH Support will point back to Cloudron at that point. So any help on this would be appreciated.

[robi]

@d19dotca In older kernels, additional IPs on say eth0 had additional designations, like eth0:1 or another virtual interface label.

Not sure what it is now..

Does Haraka conf have options for interface/IP priority?

[d19dotca]

Hmm, that rings a bell but I don't think I've seen that for a few years now. The interface (doing a ip a command) shows this:

2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether {macAddress} brd ff:ff:ff:ff:ff:ff
    inet {floatingIP} scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet {autoAssignedIP} metric 100 brd {gatewayIP} scope global dynamic enp5s0f0
       valid_lft 72471sec preferred_lft 72471sec
    inet6 {autoAssignedIPv6} scope link 
       valid_lft forever preferred_lft forever

I assumed too since the floating IP is listed first that it'd be the preferred one, but doesn't seem to be. In the past, I'm pretty sure that's what happened, as long as I added the floating IP higher in the list than the auto-assigned IP, Cloudron's Docker containers would automatically pick up the floating IP.

[Kubernetes]

To change the outgoing traffic to use the floatingIP, you will need to set up a network route or modify the routing table.

Here are the general steps to change the outgoing traffic to use the floatingIP:

Identify the current default gateway IP address by running the command:
ip route show default

Delete the existing default gateway by running the command:
sudo ip route del default (be careful with this as you will lose the connection to your server via ssh and won't be able to set the new default route!)

Add a new default gateway that uses the floatingIP by running the command:
sudo ip route add default via {floatingIP}

Verify the changes by running the command:
ip route show default

After following these steps, the outgoing traffic should now use the floatingIP as the source IP address.

[robi]

The metric param also has a weight function, lower being higher priority.

[girish]

@d19dotca said in SMTP using wrong IP address on interface with multiple addresses:

I recently setup a new dedicated server and noticed that Cloudron is using the default assigned IP address for sending email instead of the static IP address I assigned to both the NIC and the Cloudron configuration under Networking.

The Networking configuration is primarily affecting what IP Cloudron uses to configure the DNS.

It's not routing traffic via this IP etc, all that is handled by the OS/Kernel. In essence, Cloudron relies on Docker and Docker relies on IP tables and routing to route traffic. So, I guess you have to look into IP tables, not sure?

Also, is the Floating IP like DigitalOcean Floating IP? DO floating IP cannot be used as address for outbound traffic , it is only used for incoming traffic. This is because the IP is not assigned to any interface, if I recall correctly.

[d19dotca]

@Kubernetes That sounds like something similar to what ChatGTP told me too so that's good to know. It's strange, because I have what I assumed to be the correct rules in place, as I can run a command curl ifconfig.me and it will now show me the correct IP, but this is for the host, it still uses the unwanted IP if I am running that from a container. So it seems like I need to do something specific to Docker.

@robi said in SMTP using wrong IP address on interface with multiple addresses:

The metric param also has a weight function, lower being higher priority.

Ah yes, I added different weights to mine and still unfortunately see no difference in behaviour. It works fine for host traffic, just not Docker traffic.

@girish said in SMTP using wrong IP address on interface with multiple addresses:

Also, is the Floating IP like DigitalOcean Floating IP? DO floating IP cannot be used as address for outbound traffic , it is only used for incoming traffic. This is because the IP is not assigned to any interface, if I recall correctly.

I don't think that's accurate anymore, as I actually stumbled across this (in part trying to see if I could learn anything from it to apply to OVH, lol): https://docs.digitalocean.com/products/networking/reserved-ips/how-to/outbound-traffic/ - unless "Reserved IP" is different than "Floating IP" for DigitalOcean? Also I found this URL which I was using (it got me in the right direction as my host traffic now goes properly out the floating IP, but just not my Docker traffic yet) which is showing how to use the OVH Floating IP as an outbound source IP: https://medium.com/@PHaroZ_/use-ovh-floating-ip-for-outgoing-traffic-including-from-containers-cd03edaecc3

I'm somehow stuck at trying to get Docker to follow the same rules as my host is following.

[robi]

@d19dotca Agree, you have the outer layer done, now it's up to the inner layer of your Docker networking setup to get it right or follow the host settings/weights.

[girish]

@d19dotca heh, I didn't even know Floating IP got renamed to Reserved IP (in DigitalOcean). Good info!

[d19dotca]

Well I'm kind of ashamed to say it but I gave up, I spent far too many hours on this one (felt like ~25 between the last three days) just on this one single issue in order to avoid an IP block for my clients as Microsoft was blocking the IP of the host machine, and I still couldn't get it to work. I eventually opted instead to create a new Public Cloud instance at OVH instead, migrated all the data to it, using the floating IP, and things seem to be working fine now (only accessible with one public IP address so Docker had no other choice, lol). Sorry there's no confirmed solution to this one.