Email sending broken after updating to 8.2.x (due to IPv6 issues)

[jdaviescoates]

See e.g.

@jdaviescoates said in Cloudron 8.2 Released:

@jdaviescoates said in Cloudron 8.2 Released:

Warning to everyone on 8.2: it seems lots of things (including email delivery) break now break if you don't have lots of ipv6 stuff set-up already on your server, e.g. I just got this:

"message": "Upstream error: 450 4.7.25 Service unavailable, sending IPv6 address [2a01:4f9:6b:54cd::2] must have reverse DNS record (S820). [LO1PEPF000028CD.GBRP265.PROD.OUTLOOK.COM 2025-01-14T11:30:53.543Z 08DD32A234230673]"

I'm unclear why such crucial things were allowed to break with this update - I don't recall seeing any warnings on here about setting up ipv6 stuff before updating (and perhaps auto-updates shouldn't've have happened at all without such warnings being read).

It seems to me that the email status check stuff needs to be updated - given the warning above I evidently should not be getting a green light for my PTR

I read somewhere that one way to avoid issues with IPv6 stuff was to just have it completely disabled. But I found the settings in Network:

In my case it was already disabled, and so that obviously wasn't helping.

I found:

@girish said in IPv6 issue:

Also, If you use programmatic DNS, then you can just go to Domains -> Sync DNS (after enabling IPv6). It will setup the DNS entries automatically.

So I've done this:

1. in Network: under IPv6 section I chose Public IP
2. in Domains -> hit Sync DNS

Now I guess I just need to work out how to set-up IPv6 PTR/rDNS/reverse DNS on my Hetzner server - then perhaps I'll be able to send emails again!

[potemkin_ai]

I'm getting the following error when trying to send mail to Outlook:

Failure Reason: Error: Too many failures (Upstream error: 450 4.7.25 Service unavailable, sending IPv6 address [$MyIPv6Address] must have reverse DNS record (S820). [DU2PEPF00028CFF.eurprd03.prod.outlook.com 2025-01-14T10:52:08.920Z 08DD32C9AA1BC14F])

[potemkin_ai]

While Cloudron believes everything is Ok

[nebulon]

The check and thus the warning is fixed with https://git.cloudron.io/platform/box/-/commit/6fcfa6cac06cf2f0880669b22f154d2ae89be6de

As for other providers we also have a link to setup PTR on hetzner at https://docs.cloudron.io/email/#ptr-record

I merged the two topics as they are really about the same issue.

[jdaviescoates]

@nebulon said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

The check and thus the warning is fixed with https://git.cloudron.io/platform/box/-/commit/6fcfa6cac06cf2f0880669b22f154d2ae89be6de

As for other providers we also have a link to setup PTR on hetzner at https://docs.cloudron.io/email/#ptr-record

@nebulon I don't have a Hetzner Cloud server (I have dedicated) so I hadn't looked at

https://docs.hetzner.com/cloud/servers/cloud-server-rdns/ as linked to there, I had only seen:

https://docs.hetzner.com/robot/dedicated-server/ip/ip-addresses#reverse-dns

Which wasn't very informative.

I had tried this:

Which hadn't worked, but looking at https://docs.hetzner.com/cloud/servers/cloud-server-rdns/ gave me the clue that I should remove the /64 bit and now it seems to have worked! Phew! I just once that has propagated I'll hopefully be able to send emails again...

[avatar1024]

@jdaviescoates said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

I read somewhere that one way to avoid issues with IPv6 stuff was to just have it completely disabled.

It needs to be disabled on the server, not on Cloudron (although it also needs to be disabled on cloudron to avoid issues): https://forum.cloudron.io/post/99661

For me setting the correct IPv6 PTR was not enough to completely resolve this.

It's interesting to see that this issue is happening for you since updating to Cloudron 8.2 as I thought it was a change in Google's end. I have spotted several issues with email delivery recently (which I have submitted in separate threads in the forum) so some changes in emails in Cloudron 8.2 might have created a bunch of problems.

[girish]

To summarize the situation:

• starting 8.2, it seems the mail server has started to prefer using IPv6 for gmail. This wasn't a change in Cloudron consciously at least. I have looked into the Haraka changes and cannot find anything specific there either. I do see that gmail has IPv6 mail servers now, not sure if they were there before or not.

• To fix the situation, you simply have to set IPv6 PTR record . Cloudron has not implemented a IPv6 PTR check in 8.2 but a check is implemented for next release. The PTR record is set in the VPS provider. Usually, IPv6 is allocated a block of addresses and not a single address like IPv4.

• If you run curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip, this will give you the specific IPv6 address that Cloudron is using to connect to gmail. You have to set the PTR for this specific IPv6 address.

• If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces. sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 for example . You have to put net.ipv6.conf.ens18.disable_ipv6=1 in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.

[avatar1024]

@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

To fix the situation, you simply have to set IPv6 PTR record .

For me at least this hasn't worked well. I have three servers. In all three I started to get the issues. I entered a PTR record on all three servers and it checks well with google toolbox https://toolbox.googleapps.com/apps/dig/#PTR/ and with https://www.whatsmydns.net/#PTR.

The IPv6 addresses set for the PTR record are the ones indicated in the email error messages which is the same (I've just checked) than the ones indicated by curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip (and the same detected in the Cloudron Network Setting if IPv6 is activated).

On one of the server I kept getting straight bounce after having set the PTR.

On the two other servers I thought the issue was solved as I wasn't getting bounce anymore but when giving a closer look I saw that some messages would still get errors like:
Delivery failure. Will retry in Xs. Upstream error: 421 4.7.23 [2a03:xxxx:xx:xxx:xxxx:7fff:fe49:51af] The IP address sending this 4.7.23 message does not have a PTR record, or the corresponding forward DNS 4.7.23 entry does not match the sending IP. To protect our users from spam, 4.7.23 mail has been temporarily rate limited. To learn more about IP 4.7.23 address requirements for sending to Gmail, visit 4.7.23 https://support.google.com/a?p=sender-guidelines-ip 4.7.23 To learn more about Gmail requirements for bulk senders, visit 4.7.23 https://support.google.com/a?p=sender-guidelines. 4fb4d7f45d1cf-5d807030e25si25472762a12.537 - gsmtp",

Most would get delivered after a couple of tries by the mail server while some would stay in the retry loop indefinitely.

[avatar1024]

Anyone else experiencing this?

One more thing. On the server where I'm still getting the straight bounce, I've tried to activate IPv6 in the Settings on Cloudron, it worked and the IP is corrected detected. However trying to install apps doesn't work anymore, it stays stuck on Waiting for DNS propagation. So it seems like something is up with the IPv6 set-up on that domain. Any clues on what I need to do? I use wildcard DNS on that domain, do I need to set-up anything manually DNS wise for that domain to work with IPv6?

[nebulon]

If you use wildcard DNS then you also have to setup the AAAA (ipv6) wildcard DNS record on your own

[avatar1024]

@nebulon thank you, I thought so but wasn't sure. I've done that and it first glance it seems to have solved both the app install and email bounce issue!

I'll reactivate IPv6 and try those settings on the two other servers and see if all email delivery problem also disappear.

Do I need to also create a AAAA record for the bare domain?