Email sending broken after updating to 8.2.x (due to IPv6 issues)

[avatar1024]

@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces. sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 for example . You have to put net.ipv6.conf.ens18.disable_ipv6=1 in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.

Hi @girish,

IPv6 keeps coming back on on my servers network interface (it's not activated on Cloudron). Every so often curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip keeps returning the IPv6 address and I get the Gmail errors again. I have done both: sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 and put net.ipv6.conf.ens18.disable_ipv6=1 in /etc/sysctl.conf (replacing ens18 by eth0 which does disable IPv6 for that interface...but then after a while it comes back on).

Any idea?

[joseph]

@avatar1024 I would check if ens18 is actually your interface ? ip addr

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

@avatar1024 I would check if ens18 is actually your interface ? ip addr

sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 returns
sysctl: cannot stat /proc/sys/net/ipv6/conf/ens18/disable_ipv6: No such file or directory

whereas

sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 returns
net.ipv6.conf.eth0.disable_ipv6 = 1

and subsequently

curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip returns as expected
curl: (7) Failed to connect to ipv6.api.cloudron.io port 443 after 1 ms: Couldn't connect to server, instead of the IPv6 address which shows IPv6 is deactivated.

So I'm pretty sure eth0 is the correct one (also, when active, the IPv6 address is displayed for that interface via ifconfig).

And it all works fine temporarily, it just doesn't stays that way, even after adding net.ipv6.conf.eth0.disable_ipv6=1 at the bottom of the /etc/sysctl.conf file.

[avatar1024]

Right so, after adding to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1

on reboot, cat /proc/sys/net/ipv6/conf/all/disable_ipv6 return 1 so one would think it's all good, but it isn't. curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip still retunrs the IPv6 address and I can still see it in ifconfig for the network interface. Yet manually running either sysctl -p or sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 post boot works just fine to disable IPv6. So there is something at boot time that prevents disabling IPv6.

I therefore went the bulletproof way and disabled ipv6 as a boot argument in grub (GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"). But then I got of errors at boot time when services start, especially for nginx so that;s not working either.

Any idea?

[joseph]

@avatar1024 usually, if that happens it's because something else is enabling ipv6 on the interface. Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?

I was not aware I did use netplan (I don;t even really know what it is), but yes there is this config file and it seems to contain static IPs, see:

network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "xx:xx:xx:xx:xx:xx"
      addresses:
      - "152.xx.xx.43/22"
      - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"
      nameservers:
        addresses:
        - 46.38.225.230
        - 46.38.252.230
        - 2a03:4000:0:1::e1e6
      routes:
      - to: "default"
        via: "152.xx.xx.1"
      - on-link: true
        to: "default"
        via: "fe80::1"

If you tell me how i should modify the file then I'll try it out.

Thanks a lot for your help.

[joseph]

@avatar1024 said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

  - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"

yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.

THANK YOU! That worked.

Seems like netplan is here by default since Ubuntu 18: https://pscl4rke.wordpress.com/2019/10/01/disabling-ipv6-on-ubuntu-18-04-the-netplan-version/

[joseph]

@avatar1024 I found this https://github.com/canonical/netplan/blob/main/doc/netplan-tutorial.md#editing-netplan-yaml-files-to-disable-ipv6 . Can't figure where this page is "published" . The yaml ref is at https://netplan.readthedocs.io/en/latest/netplan-yaml/

[potemkin_ai]

Just for the reference: netplan try is usually a better way, as it might save from end up getting locked out from the server. I also usually run a background cron task, that removes those temporary files from /etc/netplan, when messing with the network.

[privsec]

Ok, its not working again (or maybe it never did.

These are my screens

From my understadning, it is set up correctly.

[jdaviescoates]

@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

its not working again

What exactly isn't working?

Have you set up the ipv6 reverse dns/ PTR /rDNS record too?

[avatar1024]

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

What exactly isn't working?

I imagine they are getting bounce to Gmail.

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Have you set up the ipv6 reverse dns/ PTR /rDNS record too?

Looks like they have looking at screenshot 2....but possibly badly. I think it should say my.businessdoctorut.com

Though for me, after setting it all up properly I kept getting issues and random bounces with the same error message. Only completely disabling IPv6 from the network interface works (and even that turned out to be a struggle )

[privsec]

@avatar1024 My mail server location is

I do keep getting bounced emails from Gmail.

[potemkin_ai]

Just in case - I've just found that IPv6 might be enabled at the NetPlan level, despite all of the sysctl and other configs.

To disable - add to the appropriate yaml file in /etc/netplan - link-local: [ ipv4 ] - on the same level as dhcp instruction. Run netplan try and netplan apply if you are lucky.

That seems to disable IPv6 on Linux.

<rant> Linux start looking more and more like Windows </rant>

[avatar1024]

@potemkin_ai said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I've just found that IPv6 might be enabled at the NetPlan level

Yep this has been discussed above in this thread...from post #52 onward. The solution provided by @joseph worked on my end. I had originally tried something along the line you mentioned (adding link-local: [ ipv4 ] to the yaml file but that didn't work...although I might have done it wrong).

[avatar1024]

@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I do keep getting bounced emails from Gmail.

In that case I would just disable IPv6 altogether from your server network interface. That's the only only thing that worked for me.

[privsec]

OK, sorry I am off and on. I do not do this as a full time job. I need this resolved but im lost as to what to do.

At this point, I just want to future proof myself from IPV6. How do I permanently disable and remove this functionality on my VPS from Netcup.

[girish]

@Gengar I signed up but it is stuck in "Order Pending"

[Gengar]

@privsec Are you also having the issue of PTR6 value turning to "null" ?

@privsec why would you turn off IPv6 completly ? You can do it but why ? I understood that emails servers from Google and Microsoft are now using full IPv6 resolution and if they receive an email coming from an IPv4 mail server they may or will consider it as spam more easily...

But regarding your question @privsec , how to disable IPv6 completly, I think I would do it on the OS level because I've seen that IPv6 is not enabled inside the docker containers.

First I would start to disable it temporarily using this command as I did in my tests before :
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

It will disable ipv6 until next reboot.

And if it works like that for you, then you can reboot your server and try to disable it permanently by creating a new file : sudo nano /etc/sysctl.d/10-disable-ipv6.conf

And you write this in this file :

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

to save your file and quit nano
CTRL+0 - ENTER - CTRL+X

And then apply this new config file by doing :
sudo sysctl --system

And IPv6 should be disabled completly at the OS level. Reboot your server.