mysql won't start upon cloudron upgrade

[81ewlska]

Hey all,

I'm seeing in /home/yellowtent/platformdata/logs/box.log:

2024-06-07T12:58:29.738Z box:platform BoxError: startMysql errored with code 125 message Command failed: docker run --restart=always -d --name=mysql                 --hostname mysql                 --net cloudron                 --net-alias mysql                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mysql                 --dns 172.38.some.IP                 --dns-search=.                 --ip 172.32.some.IP                 -e CLOUDRON_MYSQL_TOKEN=somestuff                 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1                 -e CLOUDRON_MYSQL_ROOT_PASSWORD=somestuff                 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql                 --label isCloudronManaged=true                 --cap-add SYS_NICE                 --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.4.2@sha256:some_hash 
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/yellowtent/platformdata/mysql" to rootfs at "/var/lib/mysql": mount /home/yellowtent/platformdata/mysql:/var/lib/mysql (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

    at /home/yellowtent/box/src/shell.js:71:23
    at ChildProcess.exithandler (node:child_process:427:5)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at ChildProcess._handle.onexit (node:internal/child_process:302:5)

When mysql starts after upgrading. This means a bunch of stuff doesn't work.

• I've tried upgrading from Ubuntu 20.04 to 22.04, that seemed to work but the problem persists.
• I've tried docker container prune and docker volume prune, restart, same problem.
• I've tried the mysql troubleshooting instructions here: https://docs.cloudron.io/troubleshooting/#mysql-addon

Any ideas? It seems like it's a permissions thing, but I don't want to tinker with Cloudron's guts too much.

[girish]

@81ewlska did you get this sorted out? It looks like the docker directory is corrupt. Maybe because the server ran out of disk space at some point ?

[81ewlska]

@girish Thanks for checking in. I have not gotten it sorted out...if I revert to a previous Cloudron version everything is fine! Which leads me to believe something's going wrong with the upgrade.

[girish]

@81ewlska I suspect something in docker is already corrupt and this is hit only when you upgrade. The upgrade downloads a newer docker image for mysql .

On Cloudron, there is no data as it can always recreate docker containers and images without any data loss. There is currently no instructions on how to recreate docker storage. I will add a doc for this.

[81ewlska]

@girish great, thanks!

FWIW I followed the Ubuntu upgrade instructions in https://docs.cloudron.io/guides/upgrade-ubuntu-22/
and the recreate-containers script never finishes, same error I believe.

[girish]

@81ewlska you can get the latest support script from here https://git.cloudron.io/cloudron/box/-/raw/master/scripts/cloudron-support .

Before you run it on the server, go to Services and check if all the services are running.

If they are running, then run the script on the server:

wget https://git.cloudron.io/cloudron/box/-/raw/master/scripts/cloudron-support -O /usr/bin/cloudron-support
cloudron-support --recreate-docker

After the above, you can update Cloudron and it should work.

[81ewlska]

Thank you @girish , this worked.

For those coming through in the future note that cloudron-support may ask you to reboot and run again, keep an eye out for it.