Solved Frequent "/mail was restarted (OOM)" on several servers

[msbt]

Since a few weeks I'm getting more and more random OOM notifications from the mail service. I ignored it at first, because one of the machines where it happened is a bit overprovisioned and doesn't have a lot of free memory. But lately also machines that have almost nothing on it and don't even use mail are affected. Any chance you can take a look at it? What logs and access can I offer?

Prior to the restart, a lot of these are happening (mail.log):

...
2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<very@mydomain.com> retval=DENY msg="No such address"
2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":41}
2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <very@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<promise@mydomain.com> retval=DENY msg="No such address"
2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":42}
2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <promise@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<pocket@mydomain.com> retval=DENY msg="No such address"
2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":43}
2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <pocket@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<receive@mydomain.com> retval=DENY msg="No such address"
2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":44}
2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <receive@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
2021-05-06T22:17:34.000Z [NOTICE] [-] [core] worker 2 killed by signal SIGKILL
2021-05-06T22:17:34.000Z [NOTICE] [-] [core] worker started worker=3 pid=469
2021-05-06T22:17:34.000Z loaded TLD files:
2021-05-06T22:17:34.000Z   1=1504
2021-05-06T22:17:34.000Z   2=8569
2021-05-06T22:17:34.000Z   3=2448
2021-05-06T22:17:34.000Z loaded 9105 Public Suffixes
2021-05-06T22:17:35.000Z loglevel: INFO
2021-05-06T22:17:35.000Z invalid log format: undefined defaulting to DEFAULT
2021-05-06T22:17:35.000Z [WARN] [-] [core] smtp.ini.nodes unset, using 1, see https://github.com/haraka/Haraka/wiki/Performance-Tuning
2021-05-06T22:17:35.000Z Starting up Haraka version 2.8.27
...

Thanks, M

[jdaviescoates]

@msbt same. I've had 4 mail oom since 25 April on my Cloudron that doesn't even really use mail at all.

I wonder if there was an update around then with some sort of regression or memory leak.

[timconsidine]

@jdaviescoates Same here. I upped the memory limit on the rainloop instance, just in case. But I don't think that was the issue.

[jdaviescoates]

Just having a look and I think perhaps in my case it's just that my VPS is constantly running close to the 8GB RAM it's got.

I guess I either need to uninstall some unused apps or increase the power of my server...

[d19dotca]

For what it's worth, I haven't had this issue yet and I run email pretty heavily in Cloudron. I also have it with a large memory cap though at 3 GB, so maybe that's why I haven't encountered it yet. Might be worth increasing the memory cap for it if possible, as a workaround at least.

[fbartels]

@msbt is full text search enabled for these mail apps?

[girish]

To increase memory for mail, you have to go to Services -> Mail -> increase the limit. Changing memory for rainloop/roundcube/sogo won't help since they are just frontends to the mail server.

[msbt]

@fbartels no fts, on some servers there aren't even any mailboxes

@girish I know that, but since I'm not using mail in most of those cases, I would rather want to find out what spam-methods create the memory spike instead of allowing more attempts

[jdaviescoates]

@jdaviescoates said in Frequent "/mail was restarted (OOM)" on several servers:

Just having a look and I think perhaps in my case it's just that my VPS is constantly running close to the 8GB RAM it's got.

I guess I either need to uninstall some unused apps or increase the power of my server...

Actually, that wasn't it.

I had 8GB RAM and was running at a fairly constant 7GB RAM, so I figured a little spike somewhere might make the server run out of RAM and perhaps that was why Mail (which I'm barely using - I've only got it enabled on 4 domains with a total of 8 barely used mailboxes) kept running out of memory.

But I've since upped my VPS to 16GB and it's still happening.

@Staff any ideas as to why a barely used Mail service has been having fairly regular memory crashes (twice a week) since 25 April? Feels to me like perhaps some update somewhere on or before 25 April has caused a memory leak or something.

Or perhaps the 512MB default just isn't enough? Although whenever I actually look in services it never looks like it's anywhere near to running out of memory:

So why does it fall over twice a week?

[nebulon]

@jdaviescoates the mail container also handles outbound emails, not just mailboxes as such. Do you happen to send out lots of mails from apps? Also is the memory consumption rather stable for some time or linearly growing?

[jdaviescoates]

@nebulon said in Frequent "/mail was restarted (OOM)" on several servers:

Do you happen to send out lots of mails from apps?

No, I don't. Well, I mean Wordfence sends out alters from various WordPress sites fairly regularly. But it looks like less than 100 emails have been delivered across my whole Cloudron in the last 2 days. So not many.

@nebulon said in Frequent "/mail was restarted (OOM)" on several servers:

Also is the memory consumption rather stable for some time or linearly growing?

I don't keep that much of a close eye on how much memory is being used, but as I mentioned above, every time I do happen to look it's hardly using any memory, so I'd say mostly a stable low amount and then occasionally (twice a week since 25 April) there must be something causing a spike. No obvious clues in the Email event log.

[msbt]

@nebulon as I said, this happens without any emails being sent, it seems to be all attempts from spammers in a short period of time: https://paste.cloudron.io/okumavovil.coffeescript - eventually the mail service runs out of memory, restarts and a day later they try again. And this is on multiple cloudrons, happy to share more logs/access

[nebulon]

@msbt not a real fix, but at least the IP shown in those logs are always the same, so may be worth putting that in the blocklist. Of course the OOM should still not happening. Not sure how to debug this further.

[msbt]

@nebulon I could enable ssh on the cloudrons in question and you could take a look, should I do that and write down some timestamps of the restarts? maybe you see something that I don't, email is not my strongsuit and there is a lot happening that I don't know

[girish]

@msbt yes, sure. can you please drop a mail on support@cloudron.io ?

[msbt]

@girish sure thing, will do when they pop up the next time. For whatever reason the OOMs stopped since Tuesday, I haven't changed anything on any machine...

[jdaviescoates]

This is still happening to me. It would appear that receiving a single email 8 minutes ago to one of the 8 mailboxes (most of which have never sent nor received any mail) caused mail with 512MB (on a server with over 8GB spare RAM) to run out of memory. It really seems like there must be something wrong somewhere. I'll up Mail to 1GB now, but this is making me worried about actually starting to properly use Cloudron mail if it keeps falling over with basically no use

[msbt]

@jdaviescoates it started happening again here as well and I've already sent a message to support with access to the corresponding servers. It also happened on one server which has 768MB reserved for the mail service with almost no traffic, so the problem is probably somewhere else.

[girish]

@msbt yes, thanks for reaching out, will take a look.