Email sending broken after updating to 8.2.x (due to IPv6 issues)

[AartJansen]

So with a self hosted install I would need to ask the ISP to set up the ipv6 PTR like they ip4 record ?

[avatar1024]

Ok so I've now noticed that the bounce are only when someone with a Gmail address email say user1@mydomain (mailbox hosted on Cloudron) which then forwards to user1@gmail (redirect through roundcube).

• anyuser@mydomain > user1@gmail works (
• anyuser@posteo > user1@mydomain > user1@gmail works
• anyuser@gmail > user1@mydomain > user1@gmail fails

So something seem to be up with redirect specifically when the sender is a Gmail user and the recipient of the redirect is Gmail user.

Error is the usual IPv6 Google BS:

"Upstream error: 421 4.7.23 [2a03:xxxx:xx:xxx:xxxx:xxxx:xxxx:51af]
The IP address sending this 4.7.23 message does not have a PTR record, or the corresponding forward DNS 4.7.23 entry does not match the sending IP. 
To protect our users from spam, 4.7.23 mail has been temporarily rate limited. To learn more about IP 4.7.23 address requirements for sending to Gmail, visit  4.7.23  https://support.google.com/a?p=sender-guidelines-ip  4.7.23 
To learn more about Gmail requirements for bulk senders, visit 4.7.23  https://support.google.com/a?p=sender-guidelines. a640c23a62f3a-ab7d06caa39si398649666b.93 - gsmtp",  "delay": 128

[jdaviescoates]

@avatar1024 said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Ok so I've now noticed that the bounce are only when someone with a Gmail address email say user1@mydomain (mailbox hosted on Cloudron) which then forwards to user1@gmail (redirect through roundcube).

anyuser@mydomain > user1@gmail works ☑ (
anyuser@posteo > user1@mydomain > user1@gmail works ☑
anyuser@gmail > user1@mydomain > user1@gmail fails ❎

So something seem to be up with redirect specifically when the sender is a Gmail user and the recipient of the redirect is Gmail user.

This sounds like the same or very similar problem as @fengchang is having too https://forum.cloudron.io/topic/13277/forward-email-with-ses-got-554-message-rejected-email-address-is-not-verified?_=1739395101471

[avatar1024]

@jdaviescoates Indeed that looks very similar...thanks I'll post there. In my case I'm using the Cloudron built SMTP server

I wonder if it is also linked to this: https://forum.cloudron.io/post/99711

[scooke]

@avatar1024 Is 421 4.7.23 your VPS IP?

[avatar1024]

@scooke nope 421 4.7.23 is the Google error code I believe, see https://support.google.com/a/answer/81126?visit_id=638750398959602051-541616874&p=sender-guidelines&rd=1.

The IP(v6) of the server is the one in the error message right after 421 4.7.23 in [] which I've hidden with xxx.

[avatar1024]

Given this only happens with forwards it seems it might have to do with the SRS rewrite. The weird thing is why does it only happens if the original sender is Gmail?

[joseph]

@AartJansen said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

So with a self hosted install I would need to ask the ISP to set up the ipv6 PTR like they ip4 record ?

yes

[avatar1024]

@avatar1024 Right so here I had made a mistake, I hadn't disabled IPv6 on the network interface on my server persistently and I reckon it got reactivate after a reboot. I've updated my post here for clarity: https://forum.cloudron.io/post/100505

[dsp76]

For Hetzner... it looks like the PTR setting for an IPv6 is not checking if its correctly set. You will need to add the last piece - in my case "::1" - to the PTR record - even if Hetzner also accepts "::" only. I guessed this will cover the whole range - but I was wrong.

[humptydumpty]

@dsp76 I have it as ::/64 which is my full IPv6 address as shown at dash > IP's > IPv6 and it's been working fine for me. In case it matters, I'm on the CCX plan (VDS).

Edit: asked chatgpt about this and it looks like I got it wrong too (possibly a security issue too). But, it mentioned that ::1 might be reserved by Hetzner so we'll need to manually assign it to something else.

That means your IPv6 address is `xxxx:xxx:xx:xxxx::1`, and it’s using the `/64` subnet.  

If others have had issues with `::1`, you might want to pick another address, like `::100` or `::abcd`. You can assign it manually with:  


ip -6 addr add xxxx:xxx:xx:xxxx::100/64 dev eth0
 

Then, set the PTR record for `xxxx:xxx:xx:xxxx::100` in Hetzner’s panel. If everything works, make it persistent by adding it to your network config (`/etc/network/interfaces` or equivalent, depending on your OS). 

Edit 2: so dug some more and setting the ptr to cover the entire /64 block is a bit riskier. With that said, it's OK in my case since I'm using Hetzner's vDedicated (CCX) plan and according to Mr.GPT, I'm the sole user on for that IP. Also, security wise, I'm in the clear if I have set SPF, DMARC, and DKIM records.

I did a Reverse lookup on Mxtoolbox for my IPv6 address and it's reporting back my mailserver domain .

Give it a shot and see if that works for you.

[jdaviescoates]

@humptydumpty I've got a dedicated server with Hetzner and having /64 at the end didn't work for me:

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I had tried this:

Screenshot from 2025-01-14 12-45-06.png

Which hadn't worked, but looking at https://docs.hetzner.com/cloud/servers/cloud-server-rdns/ gave me the clue that I should remove the /64 bit and now it seems to have worked!

[humptydumpty]

@jdaviescoates Does it fail the reverse lookup or Hetzner doesn't save the entry?

[jdaviescoates]

@humptydumpty said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Hetzner doesn't save the entry?

This. Hetzner just says it's invalid.

[avatar1024]

@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces. sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 for example . You have to put net.ipv6.conf.ens18.disable_ipv6=1 in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.

Hi @girish,

IPv6 keeps coming back on on my servers network interface (it's not activated on Cloudron). Every so often curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip keeps returning the IPv6 address and I get the Gmail errors again. I have done both: sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 and put net.ipv6.conf.ens18.disable_ipv6=1 in /etc/sysctl.conf (replacing ens18 by eth0 which does disable IPv6 for that interface...but then after a while it comes back on).

Any idea?

[joseph]

@avatar1024 I would check if ens18 is actually your interface ? ip addr

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

@avatar1024 I would check if ens18 is actually your interface ? ip addr

sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 returns
sysctl: cannot stat /proc/sys/net/ipv6/conf/ens18/disable_ipv6: No such file or directory

whereas

sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 returns
net.ipv6.conf.eth0.disable_ipv6 = 1

and subsequently

curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip returns as expected
curl: (7) Failed to connect to ipv6.api.cloudron.io port 443 after 1 ms: Couldn't connect to server, instead of the IPv6 address which shows IPv6 is deactivated.

So I'm pretty sure eth0 is the correct one (also, when active, the IPv6 address is displayed for that interface via ifconfig).

And it all works fine temporarily, it just doesn't stays that way, even after adding net.ipv6.conf.eth0.disable_ipv6=1 at the bottom of the /etc/sysctl.conf file.

[avatar1024]

Right so, after adding to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1

on reboot, cat /proc/sys/net/ipv6/conf/all/disable_ipv6 return 1 so one would think it's all good, but it isn't. curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip still retunrs the IPv6 address and I can still see it in ifconfig for the network interface. Yet manually running either sysctl -p or sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 post boot works just fine to disable IPv6. So there is something at boot time that prevents disabling IPv6.

I therefore went the bulletproof way and disabled ipv6 as a boot argument in grub (GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"). But then I got of errors at boot time when services start, especially for nginx so that;s not working either.

Any idea?

[joseph]

@avatar1024 usually, if that happens it's because something else is enabling ipv6 on the interface. Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?

[avatar1024]

@joseph said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

Do you use netplan? If so, I could chek /etc/netplan/50-cloud-init.yaml . Does it have some statically assigned ipv6 block?

I was not aware I did use netplan (I don;t even really know what it is), but yes there is this config file and it seems to contain static IPs, see:

network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: "xx:xx:xx:xx:xx:xx"
      addresses:
      - "152.xx.xx.43/22"
      - "2a03:4000:xx:xx:xxxx:xxxx:xxxx:6007/64"
      nameservers:
        addresses:
        - 46.38.225.230
        - 46.38.252.230
        - 2a03:4000:0:1::e1e6
      routes:
      - to: "default"
        via: "152.xx.xx.1"
      - on-link: true
        to: "default"
        via: "fe80::1"

If you tell me how i should modify the file then I'll try it out.

Thanks a lot for your help.