Apps using MySQL are not working

girish

The mysql addon fails to start sometimes and this results in mysql based apps not working.

First check the status of the mysql addon using docker logs -f mysql. If it says Waiting for MySQL to come up, then that is the root the problem.

docker exec -ti mysql /bin/bash
root@f483661487a5:/# mysqld --datadir="${MYSQL_DATA_DIR}" --user=mysql
mysqld: error while loading shared libraries: libaio.so.1: cannot stat shared object: Permission denied

This appears to be something related to docker. There are some existing bug reports/pages for this:

Not sure what caused the problem but simply recreating the addon infrastructure fixed it (this essentially recreates the docker containers). This can done by:

Bump up the patch version in /home/yellowtent/platformdata/INFRA_VERSION. This has to be done very carefully. Please do not do this without a complete backup.
systemctl restart cloudron.target

M-arcus

Is this affecting new docker builds (cloudron build )?

girish

In another server instance hit by https://forum.cloudron.io/topic/1382/error-for-multiple-apps/12, recreating docker containers did not help. I even disabled apparmor altogether using sudo systemctl stop apparmor and sudo systemctl disable apparmor. We tried server restart as well and that didn't help either.

After disabling apparmor, I saw that mysqld would just quit without showing any error message or log output As a final thing, I just ended up moving platformdata/mysql to mysql-old and recreated containers again. That made mysql come up and then we restored all the apps.

Still a bit crazy/worrying that we hit this bug atleast twice now and there is no clear resolution.