Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    When increasing memory to MySQL service, I get "updateServiceConfig(mail) exited with code 1 signal null" error.

    Support
    memory
    2
    5
    156
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • d19dotca
      d19dotca last edited by girish

      Hello,

      Due to a large increase in traffic to a few client websites since they were listed in BCCDC database of COVID-19 testing centres, I'm wanting to increase MySQL from 2 GB to 4 GB in terms of the memory limit.

      Unfortunately I tried to do that just a bit ago but get this very bizarre error when doing so:
      2939495d-2284-4bbb-b7d1-cc9fc3218eb3-image.png

      Why would increasing memory for MySQL raise an error around mail? What's even stranger is that after I hit cancel, it seemed like it was reporting 4 GB despite it failing in the window above. Logs are below of the event, looks like a 500 server error, or busy resource.

      2020-09-22T23:58:58.198Z box:shell updateServiceConfig(mysql) spawn: /usr/bin/docker update --memory 2147483648 --memory-swap 4294967296 mysql
      2020-09-22T23:58:59.095Z box:shell updateServiceConfig(mysql) (stdout): mysql
      
      2020-09-22T23:58:59.100Z box:shell updateServiceConfig(postgresql) spawn: /usr/bin/docker update --memory 134217728 --memory-swap 268435456 postgresql
      2020-09-22T23:58:59.370Z box:shell updateServiceConfig(postgresql) (stdout): postgresql
      
      2020-09-22T23:58:59.374Z box:shell updateServiceConfig(mail) spawn: /usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail
      2020-09-22T23:58:59.592Z box:shell updateServiceConfig(mail) (stdout): Error response from daemon: Cannot update container 69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590: runc did not terminate sucessfully: failed to write 805306368 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590/memory.limit_in_bytes: device or resource busy
      : unknown
      
      2020-09-22T23:58:59.594Z box:shell updateServiceConfig(mail) code: 1, signal: null
      Box POST /api/v1/services/mysql 500 Internal Server Error updateServiceConfig(mail) exited with code 1 signal null 1408.334 ms - 112
      2020-09-22T23:59:00.253Z box:apphealthmonitor app health: 33 alive / 1 dead.
      2020-09-22T23:59:10.380Z box:apphealthmonitor app health: 33 alive / 1 dead.
      2020-09-22T23:59:19.355Z box:shell updateServiceConfig(mysql) spawn: /usr/bin/docker update --memory 2147483648 --memory-swap 4294967296 mysql
      2020-09-22T23:59:19.479Z box:shell updateServiceConfig(mysql) (stdout): mysql
      
      2020-09-22T23:59:19.483Z box:shell updateServiceConfig(postgresql) spawn: /usr/bin/docker update --memory 134217728 --memory-swap 268435456 postgresql
      2020-09-22T23:59:19.573Z box:shell updateServiceConfig(postgresql) (stdout): postgresql
      
      2020-09-22T23:59:19.575Z box:shell updateServiceConfig(mail) spawn: /usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail
      2020-09-22T23:59:19.686Z box:shell updateServiceConfig(mail) (stdout): Error response from daemon: Cannot update container 69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590: runc did not terminate sucessfully: failed to write 805306368 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590/memory.limit_in_bytes: device or resource busy
      : unknown
      
      2020-09-22T23:59:19.689Z box:shell updateServiceConfig(mail) code: 1, signal: null
      Box POST /api/v1/services/mysql 500 Internal Server Error updateServiceConfig(mail) exited with code 1 signal null 340.502 ms - 112
      2020-09-22T23:59:20.790Z box:apphealthmonitor app health: 33 alive / 1 dead.
      2020-09-22T23:59:28.514Z box:addons Getting logs for mysql
      2020-09-22T23:59:30.337Z box:apphealthmonitor app health: 33 alive / 1 dead.
      2020-09-22T23:59:34.164Z box:shell startTask (stdout): Finished with result: success
      

      PS - In case anyone is curious, it's not a ton of traffic but not a tiny amount either. One clinic went from ~2,000 visits per month to ~50,000 visits per month after being listed in the BCCDC for COVID-19 testing locations, and that was practically overnight too. This occurred about two months ago (and hit a max of ~80,000 earlier), but now I have two other clinics under the same banner going online with the same services so I'm gearing up for much more traffic to the server as a result, thus the increase to 4 GB from 2 GB as an initial test.

      --
      Dustin Dauncey
      www.d19.ca

      1 Reply Last reply Reply Quote 0
      • girish
        girish Staff last edited by

        @d19dotca Ah, I think I have seen this error before. This happens when the kernel is unable to update the memory constraints of a container. I don't really know the root cause since it's somewhere low level.

        Can you try running this on the server:

        /usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail
        

        I guess you get the above error? (failed to write ...). What is the free -m output? Maybe there is not enough memory to allocate for all the containers or something.

        If you feel like it, can you tinker with the above values a bit to see if anything works? The math is 805306368 (memory) is 768MB (76810241024) and 1610612736 is twice of that number (memory+swap).

        d19dotca 1 Reply Last reply Reply Quote 0
        • d19dotca
          d19dotca @girish last edited by d19dotca

          @girish said in When increasing memory to MySQL service, I get "updateServiceConfig(mail) exited with code 1 signal null" error.:

          /usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail

          That command actually works fine. However right before hand I tried with "Reset to defaults" button on the MySQL container and it happened again. Is it because the memory is too high on the service already so it can't go any lower without needing to restart the service perhaps? Although if that's the case it doesn't explain why increasing the memory still caused the same error.
          8e9615ee-9fae-4460-ad6f-0d44ee813e3d-Screen Shot 2020-09-22 at 7.27.13 PM.png

          I just think it's really weird too that the error is about mail when I'm trying to manage MySQL.

          Btw, I think memory is okay. Here's the graphs of system memory in Cloudron.
          b063362b-a36c-4b35-a2d7-8ca0761e071a-image.png

          --
          Dustin Dauncey
          www.d19.ca

          girish 1 Reply Last reply Reply Quote 0
          • girish
            girish Staff @d19dotca last edited by

            @d19dotca said in When increasing memory to MySQL service, I get "updateServiceConfig(mail) exited with code 1 signal null" error.:

            I just think it's really weird too that the error is about mail when I'm trying to manage MySQL.

            Yes, that's fine. It's because there is a single API to set the size of all the services together. Some day, we will fix the backend to make it granular to change only one service (like how the UI exposes it).

            That command actually works fine

            This seems to suggest that maybe we should set the memory limits more slowly or maybe retry more. I actually worked around a similar error when the server restarts by retrying. You will see a similar error in the comments here - https://git.cloudron.io/cloudron/box/-/blob/master/src/platform.js#L94 . I will make a patch.

            1 Reply Last reply Reply Quote 1
            • girish
              girish Staff last edited by

              @d19dotca You can try https://git.cloudron.io/cloudron/box/-/commit/3c565defca000feb21c47e9d8c9ada4c62c11cef and then systemctl restart box.

              Alternately, you just use the docker CLI until the next release:

              usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail
              

              The containers are named mysql, postgresql, mail, mongodb.

              1 Reply Last reply Reply Quote 0
              • First post
                Last post
              Powered by NodeBB