Email sending broken after updating to 8.2.x (due to IPv6 issues)

[jdaviescoates]

@humptydumpty I've got a dedicated server with Hetzner and having /64 at the end didn't work for me:

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I had tried this:

Screenshot from 2025-01-14 12-45-06.png

Which hadn't worked, but looking at https://docs.hetzner.com/cloud/servers/cloud-server-rdns/ gave me the clue that I should remove the /64 bit and now it seems to have worked!

[humptydumpty]

@jdaviescoates Does it fail the reverse lookup or Hetzner doesn't save the entry?

[jdaviescoates]

@humptydumpty said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Hetzner doesn't save the entry?

This. Hetzner just says it's invalid.

[avatar1024]

@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces. sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 for example . You have to put net.ipv6.conf.ens18.disable_ipv6=1 in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.

Hi @girish,

IPv6 keeps coming back on on my servers network interface (it's not activated on Cloudron). Every so often curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip keeps returning the IPv6 address and I get the Gmail errors again. I have done both: sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 and put net.ipv6.conf.ens18.disable_ipv6=1 in /etc/sysctl.conf (replacing ens18 by eth0 which does disable IPv6 for that interface...but then after a while it comes back on).

Any idea?

[joseph]

@avatar1024 I would check if ens18 is actually your interface ? ip addr

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

@avatar1024 I would check if ens18 is actually your interface ? ip addr

sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 returns
sysctl: cannot stat /proc/sys/net/ipv6/conf/ens18/disable_ipv6: No such file or directory

whereas

sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 returns
net.ipv6.conf.eth0.disable_ipv6 = 1

and subsequently

curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip returns as expected
curl: (7) Failed to connect to ipv6.api.cloudron.io port 443 after 1 ms: Couldn't connect to server, instead of the IPv6 address which shows IPv6 is deactivated.

So I'm pretty sure eth0 is the correct one (also, when active, the IPv6 address is displayed for that interface via ifconfig).

And it all works fine temporarily, it just doesn't stays that way, even after adding net.ipv6.conf.eth0.disable_ipv6=1 at the bottom of the /etc/sysctl.conf file.

[avatar1024]

Right so, after adding to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1

on reboot, cat /proc/sys/net/ipv6/conf/all/disable_ipv6 return 1 so one would think it's all good, but it isn't. curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip still retunrs the IPv6 address and I can still see it in ifconfig for the network interface. Yet manually running either sysctl -p or sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 post boot works just fine to disable IPv6. So there is something at boot time that prevents disabling IPv6.

I therefore went the bulletproof way and disabled ipv6 as a boot argument in grub (GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"). But then I got of errors at boot time when services start, especially for nginx so that;s not working either.

Any idea?

[joseph]

@avatar1024 usually, if that happens it's because something else is enabling ipv6 on the interface. Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?

I was not aware I did use netplan (I don;t even really know what it is), but yes there is this config file and it seems to contain static IPs, see:

network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "xx:xx:xx:xx:xx:xx"
      addresses:
      - "152.xx.xx.43/22"
      - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"
      nameservers:
        addresses:
        - 46.38.225.230
        - 46.38.252.230
        - 2a03:4000:0:1::e1e6
      routes:
      - to: "default"
        via: "152.xx.xx.1"
      - on-link: true
        to: "default"
        via: "fe80::1"

If you tell me how i should modify the file then I'll try it out.

Thanks a lot for your help.

[joseph]

@avatar1024 said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

  - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"

yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.

THANK YOU! That worked.

Seems like netplan is here by default since Ubuntu 18: https://pscl4rke.wordpress.com/2019/10/01/disabling-ipv6-on-ubuntu-18-04-the-netplan-version/

[joseph]

@avatar1024 I found this https://github.com/canonical/netplan/blob/main/doc/netplan-tutorial.md#editing-netplan-yaml-files-to-disable-ipv6 . Can't figure where this page is "published" . The yaml ref is at https://netplan.readthedocs.io/en/latest/netplan-yaml/

[potemkin_ai]

Just for the reference: netplan try is usually a better way, as it might save from end up getting locked out from the server. I also usually run a background cron task, that removes those temporary files from /etc/netplan, when messing with the network.

[privsec]

Ok, its not working again (or maybe it never did.

These are my screens

From my understadning, it is set up correctly.

[jdaviescoates]

@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

its not working again

What exactly isn't working?

Have you set up the ipv6 reverse dns/ PTR /rDNS record too?

[avatar1024]

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

What exactly isn't working?

I imagine they are getting bounce to Gmail.

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Have you set up the ipv6 reverse dns/ PTR /rDNS record too?

Looks like they have looking at screenshot 2....but possibly badly. I think it should say my.businessdoctorut.com

Though for me, after setting it all up properly I kept getting issues and random bounces with the same error message. Only completely disabling IPv6 from the network interface works (and even that turned out to be a struggle )

[privsec]

@avatar1024 My mail server location is

I do keep getting bounced emails from Gmail.

[potemkin_ai]

Just in case - I've just found that IPv6 might be enabled at the NetPlan level, despite all of the sysctl and other configs.

To disable - add to the appropriate yaml file in /etc/netplan - link-local: [ ipv4 ] - on the same level as dhcp instruction. Run netplan try and netplan apply if you are lucky.

That seems to disable IPv6 on Linux.

<rant> Linux start looking more and more like Windows </rant>

[avatar1024]

@potemkin_ai said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I've just found that IPv6 might be enabled at the NetPlan level

Yep this has been discussed above in this thread...from post #52 onward. The solution provided by @joseph worked on my end. I had originally tried something along the line you mentioned (adding link-local: [ ipv4 ] to the yaml file but that didn't work...although I might have done it wrong).

[avatar1024]

@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I do keep getting bounced emails from Gmail.

In that case I would just disable IPv6 altogether from your server network interface. That's the only only thing that worked for me.