Apps using MySQL are not working

[girish]

The mysql addon fails to start sometimes and this results in mysql based apps not working.

First check the status of the mysql addon using docker logs -f mysql. If it says Waiting for MySQL to come up, then that is the root the problem.

docker exec -ti mysql /bin/bash
root@f483661487a5:/# mysqld --datadir="${MYSQL_DATA_DIR}" --user=mysql
mysqld: error while loading shared libraries: libaio.so.1: cannot stat shared object: Permission denied

This appears to be something related to docker. There are some existing bug reports/pages for this:

• https://stackoverflow.com/questions/22473830/docker-and-mysql-libz-so-1-cannot-open-shared-object-file-permission-denied
• https://github.com/moby/moby/issues/7512
• https://github.com/moby/moby/issues/5430

Not sure what caused the problem but simply recreating the addon infrastructure fixed it (this essentially recreates the docker containers). This can done by:

• Bump up the patch version in /home/yellowtent/platformdata/INFRA_VERSION. This has to be done very carefully. Please do not do this without a complete backup.
• systemctl restart cloudron.target

[M-arcus]

Is this affecting new docker builds (cloudron build )?

[girish]

In another server instance hit by https://forum.cloudron.io/topic/1382/error-for-multiple-apps/12, recreating docker containers did not help. I even disabled apparmor altogether using sudo systemctl stop apparmor and sudo systemctl disable apparmor. We tried server restart as well and that didn't help either.

After disabling apparmor, I saw that mysqld would just quit without showing any error message or log output As a final thing, I just ended up moving platformdata/mysql to mysql-old and recreated containers again. That made mysql come up and then we restored all the apps.

Still a bit crazy/worrying that we hit this bug atleast twice now and there is no clear resolution.