Email sending broken after updating to 8.2.x (due to IPv6 issues)

[girish]

To summarize the situation:

• starting 8.2, it seems the mail server has started to prefer using IPv6 for gmail. This wasn't a change in Cloudron consciously at least. I have looked into the Haraka changes and cannot find anything specific there either. I do see that gmail has IPv6 mail servers now, not sure if they were there before or not.

• To fix the situation, you simply have to set IPv6 PTR record . Cloudron has not implemented a IPv6 PTR check in 8.2 but a check is implemented for next release. The PTR record is set in the VPS provider. Usually, IPv6 is allocated a block of addresses and not a single address like IPv4.

• If you run curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip, this will give you the specific IPv6 address that Cloudron is using to connect to gmail. You have to set the PTR for this specific IPv6 address.

• If your VPS provider does not allow you to set IPv6 PTR , then just disable IPv6 in the interfaces. sysctl -w net.ipv6.conf.ens18.disable_ipv6=1 for example . You have to put net.ipv6.conf.ens18.disable_ipv6=1 in your /etc/sysctl.conf for this to persist reboots. After you do this, also disable IPv6 in Cloudron, Network -> IPv6 -> Disable.

[avatar1024]

@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

To fix the situation, you simply have to set IPv6 PTR record .

For me at least this hasn't worked well. I have three servers. In all three I started to get the issues. I entered a PTR record on all three servers and it checks well with google toolbox https://toolbox.googleapps.com/apps/dig/#PTR/ and with https://www.whatsmydns.net/#PTR.

The IPv6 addresses set for the PTR record are the ones indicated in the email error messages which is the same (I've just checked) than the ones indicated by curl https://ipv6.api.cloudron.io/api/v1/helper/public_ip (and the same detected in the Cloudron Network Setting if IPv6 is activated).

On one of the server I kept getting straight bounce after having set the PTR.

On the two other servers I thought the issue was solved as I wasn't getting bounce anymore but when giving a closer look I saw that some messages would still get errors like:
Delivery failure. Will retry in Xs. Upstream error: 421 4.7.23 [2a03:xxxx:xx:xxx:xxxx:7fff:fe49:51af] The IP address sending this 4.7.23 message does not have a PTR record, or the corresponding forward DNS 4.7.23 entry does not match the sending IP. To protect our users from spam, 4.7.23 mail has been temporarily rate limited. To learn more about IP 4.7.23 address requirements for sending to Gmail, visit 4.7.23 https://support.google.com/a?p=sender-guidelines-ip 4.7.23 To learn more about Gmail requirements for bulk senders, visit 4.7.23 https://support.google.com/a?p=sender-guidelines. 4fb4d7f45d1cf-5d807030e25si25472762a12.537 - gsmtp",

Most would get delivered after a couple of tries by the mail server while some would stay in the retry loop indefinitely.

[avatar1024]

Anyone else experiencing this?

One more thing. On the server where I'm still getting the straight bounce, I've tried to activate IPv6 in the Settings on Cloudron, it worked and the IP is corrected detected. However trying to install apps doesn't work anymore, it stays stuck on Waiting for DNS propagation. So it seems like something is up with the IPv6 set-up on that domain. Any clues on what I need to do? I use wildcard DNS on that domain, do I need to set-up anything manually DNS wise for that domain to work with IPv6?

[nebulon]

If you use wildcard DNS then you also have to setup the AAAA (ipv6) wildcard DNS record on your own

[avatar1024]

@nebulon thank you, I thought so but wasn't sure. I've done that and it first glance it seems to have solved both the app install and email bounce issue!

I'll reactivate IPv6 and try those settings on the two other servers and see if all email delivery problem also disappear.

Do I need to also create a AAAA record for the bare domain?

[jdaviescoates]

@girish said in Email sending broken after updating to 8.2.x (due to IPv6 issues):

To fix the situation, you simply have to set IPv6 PTR record .

And enable IPv6 in Network settings, and then (for good measure) Sync DNS in Domains

[privsec]

Sooo.... Assume that someone doesn't do this every day.

What does one have to do to get email sending to work again?
I'm using netcup and this is the IPV6 screen

[jdaviescoates]

@privsec could be wrong, but it doesn't look like there is anywhere for you to and rDNS/ PTR/ reverse dns record for your IPv6. I'd contact netcup support.

ah, seems you should be able to do so but they've just got a crappy unclear UI

https://helpcenter.netcup.com/en/wiki/server/network-server

I'd try whacking your mail url (my
cloudron.domain unless you've changed it) into that empty field with the disk image to the right and then clicking the disk

Then check if it worked with

dig -x <your ipv6 address> +short

[avatar1024]

@privsec you can do it. In the bottom section (i.e the IPv6 section), enter the full IPv6 address in the field on the left and the PTR record (my.yourdomain.xx) on the right, press save.

You get the full IPv6 address with the command Girish gave above (or by activating IPv6 in Cloudron settings it will show the IPv6 address automatically detected).

[privsec]

OK, I grabbed the IPv6 addy from cloudron and pasted it in netcup and used the same rDNS name addy for IPv4.

Nwtcup now says to wait 48 hrs

[avatar1024]

@privsec yeah netcup says that but it may only takes a few minutes. You can check your PTR record propagated in various ways, for example:

• https://toolbox.googleapps.com/apps/dig/#PTR/
• https://www.whatsmydns.net/#PTR

[potemkin_ai]

Got the same issue again, with Outlook servers again. IPv6 is disabled on Cloudron settings and on OS level.

[potemkin_ai]

fixed by removing IPv6 IP address from Hetzner completely and cleaning up old AAA entries from DNS - they seems like confused Outlook servers.

[privsec]

I set this up and it worked on netcup for about a week.

It’s giving me Al the error again about gmails ipv6 not being set up correctly.

Is there an in-depth how to guide to correcting this on netcup?

[joseph]

@privsec if you update to 8.2.4, the ipv6 check is already in place. Check the status tab after update. What does it say?

[jdaviescoates]

@privsec not netcup specific but the most in depth guide is this post by @avatar1024 :

https://forum.cloudron.io/topic/13072/gmail-ipv6-anyone-else-with-this-experience/22?_=1738857946551

[sponch]

Also got massive problems sending mails for 2 days now. Possible that 8.2.4 was released that day?

[jdaviescoates]

@sponch have you sorted out your IPv6 stuff?

[sponch]

Yes. Worked well after doing so for some days. Then „out of the blue“ sending not possible anymore on both of my instances.
„Email not configured properly“ errors in notifications when I go to email-overview page it takes 30-40 seconds until the domains get green. All values are set correctly for every single domain…
Log says: Delivery failure, will retry in 65536s.. DNS lookup failure: Error: queryMx ESERVFAIL

[joseph]

@sponch they might be old emails. You can just delete the old mail queue (to check if these are fresh failures) from /home/yellowtent/boxdata/mail/haraka-queue/ (files inside it). Restart mail container after deleting files.