Apps using MySQL are not working

  • Themysql addon fails to start sometimes and this results in mysql based apps not working.

    First check the status of the mysql addon usingdocker logs -f mysql. If it saysWaiting for MySQL to come up, then that is the root the problem.

    docker exec -ti mysql /bin/bash
    root@f483661487a5:/# mysqld --datadir="${MYSQL_DATA_DIR}" --user=mysql
    mysqld: error while loading shared libraries: cannot stat shared object: Permission denied

    This appears to be something related to docker. There are some existing bug reports/pages for this:

    Not sure what caused the problem but simply recreating the addon infrastructure fixed it (this essentially recreates the docker containers). This can done by:

    • Bump up the patch version in/home/yellowtent/platformdata/INFRA_VERSION. This has to be done very carefully. Please do not do this without a complete backup.
    • systemctl restart

  • Is this affecting new docker builds (cloudron build )?

  • In another server instance hit by, recreating docker containers did not help. I even disabled apparmor altogether usingsudo systemctl stop apparmor andsudo systemctl disable apparmor. We tried server restart as well and that didn't help either.

    After disabling apparmor, I saw that mysqld would just quit without showing any error message or log output :( As a final thing, I just ended up moving platformdata/mysql to mysql-old and recreated containers again. That made mysql come up and then we restored all the apps.

    Still a bit crazy/worrying that we hit this bug atleast twice now and there is no clear resolution.