Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Frequent "/mail was restarted (OOM)" on several servers

    Support
    mail oom
    7
    19
    694
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      msbt App Dev last edited by

      Since a few weeks I'm getting more and more random OOM notifications from the mail service. I ignored it at first, because one of the machines where it happened is a bit overprovisioned and doesn't have a lot of free memory. But lately also machines that have almost nothing on it and don't even use mail are affected. Any chance you can take a look at it? What logs and access can I offer?

      Prior to the restart, a lot of these are happening (mail.log):

      ...
      2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<very@mydomain.com> retval=DENY msg="No such address"
      2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":41}
      2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <very@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
      2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<promise@mydomain.com> retval=DENY msg="No such address"
      2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":42}
      2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <promise@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
      2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<pocket@mydomain.com> retval=DENY msg="No such address"
      2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":43}
      2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <pocket@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
      2021-05-06T22:17:34.000Z [INFO] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core]  hook=rcpt plugin=cloudron function=translate_rcpt_to params=<receive@mydomain.com> retval=DENY msg="No such address"
      2021-05-06T22:17:34.000Z [INFO] [-] [cloudron] skipped logging 5.188.206.171. {"time":"2021-05-06T22:17:33.494Z","count":44}
      2021-05-06T22:17:34.000Z [NOTICE] [1B4E9486-C94C-4A6E-8AB0-6A5D0D61BD08.1] [core] recipient <receive@mydomain.com> code=DENY msg="No such address" sender=vvyevrdvjovq@marychapmanchildrensbooks.co.uk
      2021-05-06T22:17:34.000Z [NOTICE] [-] [core] worker 2 killed by signal SIGKILL
      2021-05-06T22:17:34.000Z [NOTICE] [-] [core] worker started worker=3 pid=469
      2021-05-06T22:17:34.000Z loaded TLD files:
      2021-05-06T22:17:34.000Z   1=1504
      2021-05-06T22:17:34.000Z   2=8569
      2021-05-06T22:17:34.000Z   3=2448
      2021-05-06T22:17:34.000Z loaded 9105 Public Suffixes
      2021-05-06T22:17:35.000Z loglevel: INFO
      2021-05-06T22:17:35.000Z invalid log format: undefined defaulting to DEFAULT
      2021-05-06T22:17:35.000Z [WARN] [-] [core] smtp.ini.nodes unset, using 1, see https://github.com/haraka/Haraka/wiki/Performance-Tuning
      2021-05-06T22:17:35.000Z Starting up Haraka version 2.8.27
      ...
      

      Thanks, M

      jdaviescoates fbartels 2 Replies Last reply Reply Quote 1
      • jdaviescoates
        jdaviescoates @msbt last edited by jdaviescoates

        @msbt same. I've had 4 mail oom since 25 April on my Cloudron that doesn't even really use mail at all.

        I wonder if there was an update around then with some sort of regression or memory leak.

        I use Cloudron with Gandi & Hetzner

        timconsidine 1 Reply Last reply Reply Quote 2
        • timconsidine
          timconsidine @jdaviescoates last edited by

          @jdaviescoates Same here. I upped the memory limit on the rainloop instance, just in case. But I don't think that was the issue.

          jdaviescoates 1 Reply Last reply Reply Quote 0
          • jdaviescoates
            jdaviescoates @timconsidine last edited by

            Just having a look and I think perhaps in my case it's just that my VPS is constantly running close to the 8GB RAM it's got.

            I guess I either need to uninstall some unused apps or increase the power of my server...

            I use Cloudron with Gandi & Hetzner

            jdaviescoates 1 Reply Last reply Reply Quote 0
            • d19dotca
              d19dotca last edited by

              For what it's worth, I haven't had this issue yet and I run email pretty heavily in Cloudron. I also have it with a large memory cap though at 3 GB, so maybe that's why I haven't encountered it yet. Might be worth increasing the memory cap for it if possible, as a workaround at least.

              --
              Dustin Dauncey
              www.d19.ca

              1 Reply Last reply Reply Quote 0
              • fbartels
                fbartels App Dev @msbt last edited by

                @msbt is full text search enabled for these mail apps?

                M 1 Reply Last reply Reply Quote 0
                • girish
                  girish Staff last edited by

                  To increase memory for mail, you have to go to Services -> Mail -> increase the limit. Changing memory for rainloop/roundcube/sogo won't help since they are just frontends to the mail server.

                  1 Reply Last reply Reply Quote 3
                  • M
                    msbt App Dev @fbartels last edited by msbt

                    @fbartels no fts, on some servers there aren't even any mailboxes

                    @girish I know that, but since I'm not using mail in most of those cases, I would rather want to find out what spam-methods create the memory spike instead of allowing more attempts 😉

                    1 Reply Last reply Reply Quote 1
                    • jdaviescoates
                      jdaviescoates @jdaviescoates last edited by

                      @jdaviescoates said in Frequent "/mail was restarted (OOM)" on several servers:

                      Just having a look and I think perhaps in my case it's just that my VPS is constantly running close to the 8GB RAM it's got.

                      I guess I either need to uninstall some unused apps or increase the power of my server...

                      Actually, that wasn't it.

                      I had 8GB RAM and was running at a fairly constant 7GB RAM, so I figured a little spike somewhere might make the server run out of RAM and perhaps that was why Mail (which I'm barely using - I've only got it enabled on 4 domains with a total of 8 barely used mailboxes) kept running out of memory.

                      But I've since upped my VPS to 16GB and it's still happening.

                      @Staff any ideas as to why a barely used Mail service has been having fairly regular memory crashes (twice a week) since 25 April? Feels to me like perhaps some update somewhere on or before 25 April has caused a memory leak or something.

                      Or perhaps the 512MB default just isn't enough? Although whenever I actually look in services it never looks like it's anywhere near to running out of memory:

                      hardly_using_any.png

                      So why does it fall over twice a week?

                      I use Cloudron with Gandi & Hetzner

                      nebulon 1 Reply Last reply Reply Quote 1
                      • nebulon
                        nebulon Staff @jdaviescoates last edited by

                        @jdaviescoates the mail container also handles outbound emails, not just mailboxes as such. Do you happen to send out lots of mails from apps? Also is the memory consumption rather stable for some time or linearly growing?

                        jdaviescoates M 2 Replies Last reply Reply Quote 0
                        • jdaviescoates
                          jdaviescoates @nebulon last edited by

                          @nebulon said in Frequent "/mail was restarted (OOM)" on several servers:

                          Do you happen to send out lots of mails from apps?

                          No, I don't. Well, I mean Wordfence sends out alters from various WordPress sites fairly regularly. But it looks like less than 100 emails have been delivered across my whole Cloudron in the last 2 days. So not many.

                          @nebulon said in Frequent "/mail was restarted (OOM)" on several servers:

                          Also is the memory consumption rather stable for some time or linearly growing?

                          I don't keep that much of a close eye on how much memory is being used, but as I mentioned above, every time I do happen to look it's hardly using any memory, so I'd say mostly a stable low amount and then occasionally (twice a week since 25 April) there must be something causing a spike. No obvious clues in the Email event log.

                          I use Cloudron with Gandi & Hetzner

                          1 Reply Last reply Reply Quote 0
                          • M
                            msbt App Dev @nebulon last edited by

                            @nebulon as I said, this happens without any emails being sent, it seems to be all attempts from spammers in a short period of time: https://paste.cloudron.io/okumavovil.coffeescript - eventually the mail service runs out of memory, restarts and a day later they try again. And this is on multiple cloudrons, happy to share more logs/access

                            nebulon 1 Reply Last reply Reply Quote 1
                            • nebulon
                              nebulon Staff @msbt last edited by

                              @msbt not a real fix, but at least the IP shown in those logs are always the same, so may be worth putting that in the blocklist. Of course the OOM should still not happening. Not sure how to debug this further.

                              M 1 Reply Last reply Reply Quote 0
                              • M
                                msbt App Dev @nebulon last edited by

                                @nebulon I could enable ssh on the cloudrons in question and you could take a look, should I do that and write down some timestamps of the restarts? maybe you see something that I don't, email is not my strongsuit and there is a lot happening that I don't know 😉

                                girish 1 Reply Last reply Reply Quote 1
                                • girish
                                  girish Staff @msbt last edited by

                                  @msbt yes, sure. can you please drop a mail on support@cloudron.io ?

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    msbt App Dev @girish last edited by

                                    @girish sure thing, will do when they pop up the next time. For whatever reason the OOMs stopped since Tuesday, I haven't changed anything on any machine... 🙄

                                    1 Reply Last reply Reply Quote 1
                                    • jdaviescoates
                                      jdaviescoates last edited by

                                      This is still happening to me. It would appear that receiving a single email 8 minutes ago to one of the 8 mailboxes (most of which have never sent nor received any mail) caused mail with 512MB (on a server with over 8GB spare RAM) to run out of memory. It really seems like there must be something wrong somewhere. I'll up Mail to 1GB now, but this is making me worried about actually starting to properly use Cloudron mail if it keeps falling over with basically no use 😞

                                      I use Cloudron with Gandi & Hetzner

                                      M 1 Reply Last reply Reply Quote 1
                                      • M
                                        msbt App Dev @jdaviescoates last edited by

                                        @jdaviescoates it started happening again here as well and I've already sent a message to support with access to the corresponding servers. It also happened on one server which has 768MB reserved for the mail service with almost no traffic, so the problem is probably somewhere else.

                                        girish 1 Reply Last reply Reply Quote 2
                                        • girish
                                          girish Staff @msbt last edited by

                                          @msbt yes, thanks for reaching out, will take a look.

                                          1 Reply Last reply Reply Quote 1
                                          • First post
                                            Last post
                                          Powered by NodeBB