Email sending broken after updating to 8.2.x (due to IPv6 issues)
-
@humptydumpty I've got a dedicated server with Hetzner and having /64 at the end didn't work for me:
@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
I had tried this:
Screenshot from 2025-01-14 12-45-06.png
Which hadn't worked, but looking at https://docs.hetzner.com/cloud/servers/cloud-server-rdns/ gave me the clue that I should remove the /64 bit and now it seems to have worked!
@jdaviescoates Does it fail the reverse lookup or Hetzner doesn't save the entry?
-
@jdaviescoates Does it fail the reverse lookup or Hetzner doesn't save the entry?
@humptydumpty said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
Hetzner doesn't save the entry?
This. Hetzner just says it's invalid.
-
To summarize the situation:
-
starting 8.2, it seems the mail server has started to prefer using IPv6 for gmail. This wasn't a change in Cloudron consciously at least. I have looked into the Haraka changes and cannot find anything specific there either. I do see that gmail has IPv6 mail servers now, not sure if they were there before or not.
-
To fix the situation, you simply have to set IPv6 PTR record . Cloudron has not implemented a IPv6 PTR check in 8.2 but a check is implemented for next release. The PTR record is set in the VPS provider. Usually, IPv6 is allocated a block of addresses and not a single address like IPv4.
-
If you run
curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip
, this will give you the specific IPv6 address that Cloudron is using to connect to gmail. You have to set the PTR for this specific IPv6 address. -
If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces.
sysctl -w net.ipv6.conf.ens18.disable_ipv6=1
for example . You have to putnet.ipv6.conf.ens18.disable_ipv6=1
in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.
@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces. sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 for example . You have to put net.ipv6.conf.ens18.disable_ipv6=1 in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.
Hi @girish,
IPv6 keeps coming back on on my servers network interface (it's not activated on Cloudron). Every so often
curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip
keeps returning the IPv6 address and I get the Gmail errors again. I have done both:sysctl -w net.ipv6.conf.ens18.disable_ipv6=1
and putnet.ipv6.conf.ens18.disable_ipv6=1
in /etc/sysctl.conf (replacing ens18 by eth0 which does disable IPv6 for that interface...but then after a while it comes back on).Any idea?
-
-
@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
@avatar1024 I would check if ens18 is actually your interface ? ip addr
sysctl -w net.ipv6.conf.ens18.disable_ipv6=1
returns
sysctl: cannot stat /proc/sys/net/ipv6/conf/ens18/disable_ipv6: No such file or directory
whereas
sysctl -w net.ipv6.conf.eth0.disable_ipv6=1
returns
net.ipv6.conf.eth0.disable_ipv6 = 1
and subsequently
curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip
returns as expected
curl: (7) Failed to connect to ipv6.api.cloudron.io port 443 after 1 ms: Couldn't connect to server
, instead of the IPv6 address which shows IPv6 is deactivated.So I'm pretty sure eth0 is the correct one (also, when active, the IPv6 address is displayed for that interface via ifconfig).
And it all works fine temporarily, it just doesn't stays that way, even after adding
net.ipv6.conf.eth0.disable_ipv6=1
at the bottom of the /etc/sysctl.conf file. -
Right so, after adding to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 net.ipv6.conf.eth0.disable_ipv6 = 1
on reboot,
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
return1
so one would think it's all good, but it isn't.curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip
still retunrs the IPv6 address and I can still see it in ifconfig for the network interface. Yet manually running eithersysctl -p
orsysctl -w net.ipv6.conf.eth0.disable_ipv6=1
post boot works just fine to disable IPv6. So there is something at boot time that prevents disabling IPv6.I therefore went the bulletproof way and disabled ipv6 as a boot argument in grub (GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"). But then I got of errors at boot time when services start, especially for nginx so that;s not working either.
Any idea?
-
Right so, after adding to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 net.ipv6.conf.eth0.disable_ipv6 = 1
on reboot,
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
return1
so one would think it's all good, but it isn't.curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip
still retunrs the IPv6 address and I can still see it in ifconfig for the network interface. Yet manually running eithersysctl -p
orsysctl -w net.ipv6.conf.eth0.disable_ipv6=1
post boot works just fine to disable IPv6. So there is something at boot time that prevents disabling IPv6.I therefore went the bulletproof way and disabled ipv6 as a boot argument in grub (GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"). But then I got of errors at boot time when services start, especially for nginx so that;s not working either.
Any idea?
-
@avatar1024 usually, if that happens it's because something else is enabling ipv6 on the interface. Do you use netplan? If so, I could chek
/etc/netplan/50-cloud-init.yaml
. Does it have some statically assigned ipv6 block?@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?
I was not aware I did use netplan (I don;t even really know what it is), but yes there is this config file and it seems to contain static IPs, see:
network: version: 2 ethernets: eth0: match: macaddress: "xx:xx:xx:xx:xx:xx" addresses: - "152.xx.xx.43/22" - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64" nameservers: addresses: - 46.38.225.230 - 46.38.252.230 - 2a03:4000:0:1::e1e6 routes: - to: "default" via: "152.xx.xx.1" - on-link: true to: "default" via: "fe80::1"
If you tell me how i should modify the file then I'll try it out.
Thanks a lot for your help.
-
@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?
I was not aware I did use netplan (I don;t even really know what it is), but yes there is this config file and it seems to contain static IPs, see:
network: version: 2 ethernets: eth0: match: macaddress: "xx:xx:xx:xx:xx:xx" addresses: - "152.xx.xx.43/22" - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64" nameservers: addresses: - 46.38.225.230 - 46.38.252.230 - 2a03:4000:0:1::e1e6 routes: - to: "default" via: "152.xx.xx.1" - on-link: true to: "default" via: "fe80::1"
If you tell me how i should modify the file then I'll try it out.
Thanks a lot for your help.
@avatar1024 said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
- "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"
yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.
-
@avatar1024 said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
- "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"
yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.
@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
yup, this is the culprit. The IPv6 address is statically assigned here. You have to put a # in front of the line to comment it out and reboot the machine.
THANK YOU! That worked.
Seems like netplan is here by default since Ubuntu 18: https://pscl4rke.wordpress.com/2019/10/01/disabling-ipv6-on-ubuntu-18-04-the-netplan-version/
-
@avatar1024 I found this https://github.com/canonical/netplan/blob/main/doc/netplan-tutorial.md#editing-netplan-yaml-files-to-disable-ipv6 . Can't figure where this page is "published" . The yaml ref is at https://netplan.readthedocs.io/en/latest/netplan-yaml/
-
Just for the reference:
netplan try
is usually a better way, as it might save from end up getting locked out from the server. I also usually run a background cron task, that removes those temporary files from /etc/netplan, when messing with the network. -
Ok, its not working again (or maybe it never did.
These are my screens
From my understadning, it is set up correctly.@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
its not working again
What exactly isn't working?
Have you set up the ipv6 reverse dns/ PTR /rDNS record too?
-
@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
its not working again
What exactly isn't working?
Have you set up the ipv6 reverse dns/ PTR /rDNS record too?
@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
What exactly isn't working?
I imagine they are getting bounce to Gmail.
@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
Have you set up the ipv6 reverse dns/ PTR /rDNS record too?
Looks like they have looking at screenshot 2....but possibly badly. I think it should say my.businessdoctorut.com
Though for me, after setting it all up properly I kept getting issues and random bounces with the same error message. Only completely disabling IPv6 from the network interface works (and even that turned out to be a struggle
)
-
@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
What exactly isn't working?
I imagine they are getting bounce to Gmail.
@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
Have you set up the ipv6 reverse dns/ PTR /rDNS record too?
Looks like they have looking at screenshot 2....but possibly badly. I think it should say my.businessdoctorut.com
Though for me, after setting it all up properly I kept getting issues and random bounces with the same error message. Only completely disabling IPv6 from the network interface works (and even that turned out to be a struggle
)
@avatar1024 My mail server location is
I do keep getting bounced emails from Gmail.
-
Just in case - I've just found that IPv6 might be enabled at the NetPlan level, despite all of the sysctl and other configs.
To disable - add to the appropriate
yaml
file in/etc/netplan
-link-local: [ ipv4 ]
- on the same level asdhcp
instruction. Runnetplan try
andnetplan apply
if you are lucky.That seems to disable IPv6 on Linux.
<rant> Linux start looking more and more like Windows
</rant>
-
Just in case - I've just found that IPv6 might be enabled at the NetPlan level, despite all of the sysctl and other configs.
To disable - add to the appropriate
yaml
file in/etc/netplan
-link-local: [ ipv4 ]
- on the same level asdhcp
instruction. Runnetplan try
andnetplan apply
if you are lucky.That seems to disable IPv6 on Linux.
<rant> Linux start looking more and more like Windows
</rant>
@potemkin_ai said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
I've just found that IPv6 might be enabled at the NetPlan level
Yep this has been discussed above in this thread...from post #52 onward. The solution provided by @joseph worked on my end. I had originally tried something along the line you mentioned (adding
link-local: [ ipv4 ]
to the yaml file but that didn't work...although I might have done it wrong). -
@avatar1024 My mail server location is
I do keep getting bounced emails from Gmail.
@privsec said in Email sending broken after updating to 8.2.x (due to IPv6 issues):
I do keep getting bounced emails from Gmail.
In that case I would just disable IPv6 altogether from your server network interface. That's the only only thing that worked for me.
-
OK, sorry I am off and on. I do not do this as a full time job. I need this resolved but im lost as to what to do.
At this point, I just want to future proof myself from IPV6. How do I permanently disable and remove this functionality on my VPS from Netcup.