When increasing memory to MySQL service, I get "updateServiceConfig(mail) exited with code 1 signal null" error.

[d19dotca]

Hello,

Due to a large increase in traffic to a few client websites since they were listed in BCCDC database of COVID-19 testing centres, I'm wanting to increase MySQL from 2 GB to 4 GB in terms of the memory limit.

Unfortunately I tried to do that just a bit ago but get this very bizarre error when doing so:

Why would increasing memory for MySQL raise an error around mail? What's even stranger is that after I hit cancel, it seemed like it was reporting 4 GB despite it failing in the window above. Logs are below of the event, looks like a 500 server error, or busy resource.

2020-09-22T23:58:58.198Z box:shell updateServiceConfig(mysql) spawn: /usr/bin/docker update --memory 2147483648 --memory-swap 4294967296 mysql
2020-09-22T23:58:59.095Z box:shell updateServiceConfig(mysql) (stdout): mysql

2020-09-22T23:58:59.100Z box:shell updateServiceConfig(postgresql) spawn: /usr/bin/docker update --memory 134217728 --memory-swap 268435456 postgresql
2020-09-22T23:58:59.370Z box:shell updateServiceConfig(postgresql) (stdout): postgresql

2020-09-22T23:58:59.374Z box:shell updateServiceConfig(mail) spawn: /usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail
2020-09-22T23:58:59.592Z box:shell updateServiceConfig(mail) (stdout): Error response from daemon: Cannot update container 69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590: runc did not terminate sucessfully: failed to write 805306368 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590/memory.limit_in_bytes: device or resource busy
: unknown

2020-09-22T23:58:59.594Z box:shell updateServiceConfig(mail) code: 1, signal: null
Box POST /api/v1/services/mysql 500 Internal Server Error updateServiceConfig(mail) exited with code 1 signal null 1408.334 ms - 112
2020-09-22T23:59:00.253Z box:apphealthmonitor app health: 33 alive / 1 dead.
2020-09-22T23:59:10.380Z box:apphealthmonitor app health: 33 alive / 1 dead.
2020-09-22T23:59:19.355Z box:shell updateServiceConfig(mysql) spawn: /usr/bin/docker update --memory 2147483648 --memory-swap 4294967296 mysql
2020-09-22T23:59:19.479Z box:shell updateServiceConfig(mysql) (stdout): mysql

2020-09-22T23:59:19.483Z box:shell updateServiceConfig(postgresql) spawn: /usr/bin/docker update --memory 134217728 --memory-swap 268435456 postgresql
2020-09-22T23:59:19.573Z box:shell updateServiceConfig(postgresql) (stdout): postgresql

2020-09-22T23:59:19.575Z box:shell updateServiceConfig(mail) spawn: /usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail
2020-09-22T23:59:19.686Z box:shell updateServiceConfig(mail) (stdout): Error response from daemon: Cannot update container 69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590: runc did not terminate sucessfully: failed to write 805306368 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/69d0c668883d02bd3b397ed3e7ea931276fdb05eaaf04a0ceb4f02a48a82d590/memory.limit_in_bytes: device or resource busy
: unknown

2020-09-22T23:59:19.689Z box:shell updateServiceConfig(mail) code: 1, signal: null
Box POST /api/v1/services/mysql 500 Internal Server Error updateServiceConfig(mail) exited with code 1 signal null 340.502 ms - 112
2020-09-22T23:59:20.790Z box:apphealthmonitor app health: 33 alive / 1 dead.
2020-09-22T23:59:28.514Z box:addons Getting logs for mysql
2020-09-22T23:59:30.337Z box:apphealthmonitor app health: 33 alive / 1 dead.
2020-09-22T23:59:34.164Z box:shell startTask (stdout): Finished with result: success

PS - In case anyone is curious, it's not a ton of traffic but not a tiny amount either. One clinic went from ~2,000 visits per month to ~50,000 visits per month after being listed in the BCCDC for COVID-19 testing locations, and that was practically overnight too. This occurred about two months ago (and hit a max of ~80,000 earlier), but now I have two other clinics under the same banner going online with the same services so I'm gearing up for much more traffic to the server as a result, thus the increase to 4 GB from 2 GB as an initial test.

[girish]

@d19dotca Ah, I think I have seen this error before. This happens when the kernel is unable to update the memory constraints of a container. I don't really know the root cause since it's somewhere low level.

Can you try running this on the server:

/usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail

I guess you get the above error? (failed to write ...). What is the free -m output? Maybe there is not enough memory to allocate for all the containers or something.

If you feel like it, can you tinker with the above values a bit to see if anything works? The math is 805306368 (memory) is 768MB (76810241024) and 1610612736 is twice of that number (memory+swap).

[d19dotca]

@girish said in When increasing memory to MySQL service, I get "updateServiceConfig(mail) exited with code 1 signal null" error.:

/usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail

That command actually works fine. However right before hand I tried with "Reset to defaults" button on the MySQL container and it happened again. Is it because the memory is too high on the service already so it can't go any lower without needing to restart the service perhaps? Although if that's the case it doesn't explain why increasing the memory still caused the same error.

I just think it's really weird too that the error is about mail when I'm trying to manage MySQL.

Btw, I think memory is okay. Here's the graphs of system memory in Cloudron.

[girish]

@d19dotca said in When increasing memory to MySQL service, I get "updateServiceConfig(mail) exited with code 1 signal null" error.:

I just think it's really weird too that the error is about mail when I'm trying to manage MySQL.

Yes, that's fine. It's because there is a single API to set the size of all the services together. Some day, we will fix the backend to make it granular to change only one service (like how the UI exposes it).

That command actually works fine

This seems to suggest that maybe we should set the memory limits more slowly or maybe retry more. I actually worked around a similar error when the server restarts by retrying. You will see a similar error in the comments here - https://git.cloudron.io/cloudron/box/-/blob/master/src/platform.js#L94 . I will make a patch.

[girish]

@d19dotca You can try https://git.cloudron.io/cloudron/box/-/commit/3c565defca000feb21c47e9d8c9ada4c62c11cef and then systemctl restart box.

Alternately, you just use the docker CLI until the next release:

usr/bin/docker update --memory 805306368 --memory-swap 1610612736 mail

The containers are named mysql, postgresql, mail, mongodb.