MySQL addon not starting

[stratus41298]

When I first noticed the problem that and mysql were down. I was able to restart postgres but not mysql. Today I was able to restart all services, but there was no effect to the environment. I resumed the docker recreate and am now waiting for a result. I'll let you know. Thank you.

[stratus41298]

Alas, still seeing the same thing after about six hours. I noticed that the recreate-docker shut down a lot of services so I bounced them again and they are showing up in the event log as rebuilt, but it's seemed to have no effect in the terminal; just endless dots. There is a reboot required so i'm tempted to do that and see how it goes, but also don't want to interrupt the process if it's actually doing something.

[BrutalBirdie]

I've seen something like this before.
=> https://forum.cloudron.io/post/94377

Can you please share the box.log of that failed attempt?

[stratus41298]

Much less in there this time:

@cloudron:~$ sudo cat /home/yellowtent/platformdata/logs/box.log
[sudo] password for :
2025-04-06T00:18:41.199Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-04-06T00:18:41.849Z box:shell services: grep -q avx /proc/cpuinfo
2025-04-06T00:18:42.065Z box:shell services: systemctl is-active unbound
2025-04-06T00:18:42.132Z box:shell services: systemctl is-active nginx
2025-04-06T00:18:51.922Z box:shell support /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remotesupport.sh is-enabled /home/cloudron-support/.ssh/authorized_keys
2025-04-06T00:18:51.933Z box:shell grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory

2025-04-06T00:18:51.933Z box:shell false

[stratus41298]

Did that reboot it was asking for. Came back and the same services were down again.

New box.log after reboot:

2025-04-06T00:30:35.868Z box:server ==========================================
2025-04-06T00:30:35.870Z box:server            Cloudron 8.3.1
2025-04-06T00:30:35.870Z box:server ==========================================
2025-04-06T00:30:35.870Z box:platform initialize: start platform
2025-04-06T00:30:36.079Z box:tasks stopAllTasks: stopping all tasks
2025-04-06T00:30:36.079Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-04-06T00:30:36.268Z box:shell All tasks stopped

2025-04-06T00:30:37.082Z box:locks releaseAll: all locks released
2025-04-06T00:30:37.158Z box:reverseproxy writeDashboardConfig: writing dashboard config for hopefullylevelheaded.com
2025-04-06T00:30:37.227Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/_.hopefullylevelheaded.com.cert -noout -ocsp_uri
2025-04-06T00:30:37.265Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-04-06T00:30:39.353Z box:platform onActivated: starting post activation services
2025-04-06T00:30:39.353Z box:platform startInfra: checking infrastructure
2025-04-06T00:30:39.365Z box:platform startInfra: updating infrastructure from 48.0.0 to 49.8.0
2025-04-06T00:30:39.365Z box:platform markApps: reconfiguring apps
2025-04-06T00:30:39.366Z box:reverseproxy removeAppConfigs: removing app nginx configs
2025-04-06T00:30:39.479Z box:platform removeAllContainers: removing all containers for infra upgrade
2025-04-06T00:30:39.479Z box:shell platform: docker ps -qa --filter label=isCloudronManaged
2025-04-06T00:30:39.502Z box:platform removeAllContainers: stopping and removing e77cfa1fb71d
2025-04-06T00:30:39.502Z box:shell platform: docker stop e77cfa1fb71d
2025-04-06T00:30:44.009Z box:shell platform: docker rm -f e77cfa1fb71d
2025-04-06T00:30:44.967Z box:platform removeAllContainers: stopping and removing 3af0b28f8899
2025-04-06T00:30:44.967Z box:shell platform: docker stop 3af0b28f8899
2025-04-06T00:30:48.236Z box:shell platform: docker rm -f 3af0b28f8899
2025-04-06T00:30:49.250Z box:platform removeAllContainers: stopping and removing 254b1c19f3af
2025-04-06T00:30:49.250Z box:shell platform: docker stop 254b1c19f3af
2025-04-06T00:30:53.888Z box:shell platform: docker rm -f 254b1c19f3af
2025-04-06T00:30:54.687Z box:platform removeAllContainers: stopping and removing 3ca2ffe09606
2025-04-06T00:30:54.687Z box:shell platform: docker stop 3ca2ffe09606
2025-04-06T00:31:04.508Z box:shell platform: docker rm -f 3ca2ffe09606
2025-04-06T00:31:06.291Z box:platform removeAllContainers: stopping and removing 197cd2129268
2025-04-06T00:31:06.292Z box:shell platform: docker stop 197cd2129268
2025-04-06T00:31:09.682Z box:shell platform: docker rm -f 197cd2129268
2025-04-06T00:31:10.643Z box:platform removeAllContainers: stopping and removing 0bfbc0e4cc90
2025-04-06T00:31:10.643Z box:shell platform: docker stop 0bfbc0e4cc90
2025-04-06T00:31:16.299Z box:shell platform: docker rm -f 0bfbc0e4cc90
2025-04-06T00:31:17.164Z box:platform removeAllContainers: stopping and removing c87aa4f6d6e9
2025-04-06T00:31:17.164Z box:shell platform: docker stop c87aa4f6d6e9
2025-04-06T00:31:29.125Z box:shell platform: docker rm -f c87aa4f6d6e9

[stratus41298]

2025-04-06T00:31:29.650Z box:platform removeAllContainers: stopping and removing 82ff73d0f943
2025-04-06T00:31:29.651Z box:shell platform: docker stop 82ff73d0f943
2025-04-06T00:31:40.939Z box:shell platform: docker rm -f 82ff73d0f943
2025-04-06T00:31:41.380Z box:platform removeAllContainers: stopping and removing b8ebac43901c
2025-04-06T00:31:41.380Z box:shell platform: docker stop b8ebac43901c
2025-04-06T00:31:47.702Z box:shell platform: docker rm -f b8ebac43901c
2025-04-06T00:31:48.170Z box:platform createDockerNetwork: recreating docker network
2025-04-06T00:31:48.170Z box:shell platform: docker network rm -f cloudron
2025-04-06T00:31:48.942Z box:shell platform: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway=172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron
2025-04-06T00:31:49.408Z box:services startServices: 48.0.0 -> 49.8.0. starting all services
2025-04-06T00:31:49.408Z box:mailserver startMail: starting
2025-04-06T00:31:49.411Z box:mailserver restart: restarting mail container with mailFqdn:my.hopefullylevelheaded.com mailDomain:hopefullylevelheaded.com
2025-04-06T00:31:49.507Z box:locks write: current locks: {"mail_restart":null}
2025-04-06T00:31:49.507Z box:locks acquire: mail_restart
2025-04-06T00:31:49.518Z box:mailserver configureMail: stopping and deleting previous mail container
2025-04-06T00:31:49.537Z box:mailserver createMailConfig: generating mail config with my.hopefullylevelheaded.com
2025-04-06T00:31:49.612Z box:mailserver configureMail: starting mail container
2025-04-06T00:31:49.613Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail                 --net cloudron                 --net-alias mail                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mail                 -m 671088640                 --memory-swap -1                 --dns 172.18.0.1                 --dns-search=.                 --ip 172.18.30.4                 -e CLOUDRON_MAIL_TOKEN=b1323c8162862c3c9f73cfddd65cdc37761b1316b80ea7b626a257b3aec90e077007ac33255b0855191299f33b490228b9904af26b251ed6876fd9e56d54c4c88ec1111884c6d9559e598d3785b2de4f0f89f96297a980b406e5ea6a5078f4727023f2886f34e5726624be27c9d2b1fd73770d126c740f035ce6bbf37e4d697e                 -e CLOUDRON_RELAY_TOKEN=3c622e0c2f602488debbfaab91e101a164ddc8058b08bd83eac9f4c011ede0c5c5804d63999f72e44aa5bd694b44c44776d47b5e025a9917dc3e5c6b2a024c36ce3829c2626d6223bcfb26b8b47491721cd2ecc7d8493407139d92a294ae05d26379f197d948f0086069ad65e59fa16544e415d40c139d584dddeb115d2d9502                 -e LOGLEVEL=info                 -v /home/yellowtent/boxdata/mail:/app/data                 -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro                 -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995                 --label isCloudronManaged=true                 --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536
2025-04-06T00:31:58.071Z box:locks write: current locks: {}
2025-04-06T00:31:58.071Z box:locks release: mail_restart
2025-04-06T00:31:58.071Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e
2025-04-06T00:31:58.079Z box:services startTurn: stopping and deleting previous turn container
2025-04-06T00:31:58.085Z box:services startTurn: starting turn container
2025-04-06T00:31:58.085Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn                 --hostname turn                 --net host                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=turn                 -m 268435456                 --memory-swap -1                 -e CLOUDRON_TURN_SECRET=a07d85effc3c5732cde5341587d878d                 -e CLOUDRON_REALM=my.hopefullylevelheaded.com                 --label isCloudronManaged=true                 --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570
2025-04-06T00:32:02.815Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-04-06T00:32:03.400Z box:shell services: grep -q avx /proc/cpuinfo
2025-04-06T00:32:03.720Z box:shell services: systemctl is-active unbound
2025-04-06T00:32:03.728Z box:shell services: systemctl is-active nginx
2025-04-06T00:32:06.333Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d
2025-04-06T00:32:06.337Z box:services startMysql: stopping and deleting previous mysql container
2025-04-06T00:32:06.343Z box:services startMysql: starting mysql container
2025-04-06T00:32:06.344Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql                 --hostname mysql                 --net cloudron                 --net-alias mysql                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mysql                 --ip 172.18.30.1                 -e CLOUDRON_MYSQL_TOKEN=985d03b1df1c62feffc9fe7064da8300b7793d9ebf437ad4b33b3bca6f3b0cb21a88cda345c2e2edd6a4f03d100ad0f1d52239f4dd2d3615164ae0d31b420cc9ce336ab325c887a972df953d557ad9ce08e4f7bb31ec4bd2f544bc9695fa888bd9e65b641d91401f0cb7e8d1e98cb67f7247770a7a20c2c91c42d51375463f64                 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1                 -e CLOUDRON_MYSQL_ROOT_PASSWORD=31803f70cf6bfdc77e859e7d182c52633bf2e6de21b6e494e52c4d1fbb47703f3a71b54f05e041117e0e6c9694c88c416f79522ecc0111585da86b74ffb9c567c45cf095a26de5396bf7ec0187b2bb722f16ed5afe9cdf97dab8b494492acded7818de789efb10833d498839d1e8c06616fc9c0f08b9901bfad13c5b7d7bb657                 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql                 --label isCloudronManaged=true                 --cap-add SYS_NICE                 --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280
2025-04-06T00:32:10.503Z box:services Waiting for mysql
2025-04-06T00:32:10.509Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-06T00:32:25.511Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-06T00:32:40.753Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306
2025-04-06T00:32:56.076Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2
2025-04-06T00:32:56.080Z box:services startPostgresql: postgresql will be upgraded
2025-04-06T00:32:56.080Z box:services exportDatabase: Exporting postgresql
2025-04-06T00:32:56.085Z box:services exportDatabase: Exporting addon postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c
2025-04-06T00:32:56.086Z box:services Backing up postgresql
2025-04-06T00:32:56.089Z box:services exportDatabase: Error exporting postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c. BoxError: Unable to find container postgresql at Object.inspect (/home/yellowtent/box/src/docker.js:607:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20) at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20) { reason: 'Not found', details: {} }
2025-04-06T00:32:56.089Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Unable to find container postgresql
2025-04-06T00:32:56.089Z box:platform BoxError: Unable to find container postgresql
    at Object.inspect (/home/yellowtent/box/src/docker.js:607:50)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20)
    at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20)

[BrutalBirdie]

I see a pattern that everytime it postgresql tries to touch app 3147d06c-a773-4d59-b8d7-170517ef116c it errors.
What app is 3147d06c-a773-4d59-b8d7-170517ef116c on your system?

[stratus41298]

That would be my nextcloud app. I could maybe try to archive the app to preserve the whole system. I don't appear to have much recourse from the GUI due to the app being busy, but if there's some other CLI command I could run, let me know.

I also see these connection refused errors with redis, which also stay down upon reboot:

2025-04-06T16:48:40.302Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-3147d06c-a773-4d59-b8d7-170517ef116c: connect ECONNREFUSED 172.18.0.2:3000
2025-04-06T16:48:43.538Z box:services Waiting for redis-6deb5899-a69c-4c85-a050-7fe5dbdcac8a
2025-04-06T16:48:43.543Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-6deb5899-a69c-4c85-a050-7fe5dbdcac8a: connect ECONNREFUSED 172.18.0.3:3000

Is this normal behavior? I've never had a firewall issue with this instance before. Seems like all of this started happening after 8.3.1.

[stratus41298]

Okay, so it seems like postgresql is actually running fine, but cloudron/docker can't seem to pass a health check:

[GET] /healthcheck
healthcheck: disconnected
[GET] /healthcheck
healthcheck: disconnected

[robi]

@stratus41298 for that you need a web service that will have a 200 response on that /healthcheck endpoint

[stratus41298]

@robi I'm not really sure where you're going with that so I can't comment.

As for the continued troubleshooting, I was able to get postgresql up manually with extensive help of ChatGPT. We ran into a DNS issue trying to do an APT reinstall of docker even after adjusting the resolv.conf. If anyone is interested I can give a full writeup, but i'll summarize here:

---

🧩 Cloudron 8.3.1 Upgrade – PostgreSQL & DNS Issues Summary

1. PostgreSQL Socket Failure

• After updating to Cloudron 8.3.1, the PostgreSQL service showed as down in the Cloudron dashboard.

• However, logs showed the database was running fine and accepting connections over TCP.

• Cloudron healthchecks (and dependent services) failed due to the absence of the expected Unix socket at /var/run/postgresql/.s.PGSQL.5432.

• Investigation showed:

  • The actual socket was created in /run/postgresql/
  • The symlink /var/run/postgresql → /run/postgresql was missing inside the PostgreSQL container

• Fix: Re-created the symlink inside the container:

  ln -s /run/postgresql /var/run/postgresql
  

  Followed by:

  supervisorctl restart postgresql
  

• After this, PostgreSQL healthcheck passed, and dependent services began recovering.

---

2. APT Repository DNS/TLS Failure

• While attempting to reinstall the Cloudron CLI to recover from a broken docker recreate, the APT repo at https://apt.cloudron.io was unreachable.

• DNS (dig apt.cloudron.io) returned SOA records but no A records — even after:

  • Disabling systemd-resolved
  • Stopping unbound
  • Overriding /etc/resolv.conf to use 8.8.8.8
  • Confirming DNS worked for other domains (cloudron.io, google.com)

• Attempts to work around with /etc/hosts resulted in a TLS handshake failure:

  error:0A000410:SSL routines::sslv3 alert handshake failure
  

• Root cause appears to be Cloudflare (which fronts apt.cloudron.io) rejecting connections when:

  • DNS is bypassed via /etc/hosts, or
  • Something in the server environment breaks SNI or TLS negotiation

• Result: Unable to reinstall cloudron CLI using APT despite proper DNS config and working CA certs.

---

🧠 Context

• This all occurred during/after a failed cloudron docker recreate attempt (as suggested on the forums).
• That left the system partially repaired, with missing CLI tools and broken DNS.

---

So that's pretty much it. At this point i'm going to just set up a restore either through VM backup or Cloudron backup. I haven't decided yet. I won't do it until tomorrow though in case there's anything you'd like me to provide for future maintenance or research.

[nebulon]

So whatever you have posted here, makes no sense in the cloudron context, some of those commands don't even exist, nor is there any cloudron apt repository (apt.cloudron.io does and should not exist). Much AI hallucination happening here it seems

Looks like we have to start from first principles, first mysql didn't work then postgres and maybe redis...? Also it is probably not clear by now in which state the server is and what would have to be undone or not. Given all that, if you have recent backups a fresh start is probably the fastest and safest option to get to some well known state.

[stratus41298]

That makes sense that there was no apt repo, but the fact was that we couldn't even get out to apt due to the DNS issue so i'd caution an immediate dismissal of AI hallucination. We were also able to get the healthcheck up and running for postgresql after it had gotten stuck during the docker recreate (see forked post) using AI as well so i'm not sure how many of the commands were inappropriate as they all went through. We did indeed disable resolved, stopped unbound, overrode resolv.conf (which i've done before for pihole many times), digged DNS, bypassed DNS using hosts, and I myself had previously performed a docker recreate. All of those commands exist except I guess the apt reinstall, but that's something I myself might have tried, if i'm being honest as it's a fairly common command on other apps.

After the 8.3.1 update many services wouldn't come up and nothing worked on this instance. I guess if you have all the information you care about on this issue, i'll go ahead and restore since i've been down for quite a while at this point.

I get the whole "AI = bad" thing, but I spent about 2 hours troubleshooting with GPT 4o (the paid version) and made significant progress until we hit a wall that we couldn't get around, which at least gave me satisfaction that we tried everything we could.

Take care.

[nebulon]

I am generally not in the group of AI = bad, however it likely just lacks lots of knowledge about the Cloudron internals, which make recommendations often not very useful in this area (yet).

Overall I am still not sure based on this thread why DNS was involved here and given that you hit 3 different addons with apparently diverse issues, maybe something unrelated to them individually caused the trouble.

We will add a few more helpers to the cloudron-support script for the future, to hopefully get down to such issues faster.

If you haven't deleted the server yet, maybe we could take a closer look via SSH, since you already invested much time in debugging. If you want that, please enable remote ssh support and send a mail to support@cloudron.io with your Cloudron details.