MySQL addon not starting
-
I've seen something like this before.
=> https://forum.cloudron.io/post/94377Can you please share the box.log of that failed attempt?
-
Much less in there this time:
@cloudron:~$ sudo cat /home/yellowtent/platformdata/logs/box.log [sudo] password for : 2025-04-06T00:18:41.199Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name 2025-04-06T00:18:41.849Z box:shell services: grep -q avx /proc/cpuinfo 2025-04-06T00:18:42.065Z box:shell services: systemctl is-active unbound 2025-04-06T00:18:42.132Z box:shell services: systemctl is-active nginx 2025-04-06T00:18:51.922Z box:shell support /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remotesupport.sh is-enabled /home/cloudron-support/.ssh/authorized_keys 2025-04-06T00:18:51.933Z box:shell grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory 2025-04-06T00:18:51.933Z box:shell false
-
Did that reboot it was asking for. Came back and the same services were down again.
New box.log after reboot:
2025-04-06T00:30:35.868Z box:server ========================================== 2025-04-06T00:30:35.870Z box:server Cloudron 8.3.1 2025-04-06T00:30:35.870Z box:server ========================================== 2025-04-06T00:30:35.870Z box:platform initialize: start platform 2025-04-06T00:30:36.079Z box:tasks stopAllTasks: stopping all tasks 2025-04-06T00:30:36.079Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2025-04-06T00:30:36.268Z box:shell All tasks stopped 2025-04-06T00:30:37.082Z box:locks releaseAll: all locks released 2025-04-06T00:30:37.158Z box:reverseproxy writeDashboardConfig: writing dashboard config for hopefullylevelheaded.com 2025-04-06T00:30:37.227Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/_.hopefullylevelheaded.com.cert -noout -ocsp_uri 2025-04-06T00:30:37.265Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx 2025-04-06T00:30:39.353Z box:platform onActivated: starting post activation services 2025-04-06T00:30:39.353Z box:platform startInfra: checking infrastructure 2025-04-06T00:30:39.365Z box:platform startInfra: updating infrastructure from 48.0.0 to 49.8.0 2025-04-06T00:30:39.365Z box:platform markApps: reconfiguring apps 2025-04-06T00:30:39.366Z box:reverseproxy removeAppConfigs: removing app nginx configs 2025-04-06T00:30:39.479Z box:platform removeAllContainers: removing all containers for infra upgrade 2025-04-06T00:30:39.479Z box:shell platform: docker ps -qa --filter label=isCloudronManaged 2025-04-06T00:30:39.502Z box:platform removeAllContainers: stopping and removing e77cfa1fb71d 2025-04-06T00:30:39.502Z box:shell platform: docker stop e77cfa1fb71d 2025-04-06T00:30:44.009Z box:shell platform: docker rm -f e77cfa1fb71d 2025-04-06T00:30:44.967Z box:platform removeAllContainers: stopping and removing 3af0b28f8899 2025-04-06T00:30:44.967Z box:shell platform: docker stop 3af0b28f8899 2025-04-06T00:30:48.236Z box:shell platform: docker rm -f 3af0b28f8899 2025-04-06T00:30:49.250Z box:platform removeAllContainers: stopping and removing 254b1c19f3af 2025-04-06T00:30:49.250Z box:shell platform: docker stop 254b1c19f3af 2025-04-06T00:30:53.888Z box:shell platform: docker rm -f 254b1c19f3af 2025-04-06T00:30:54.687Z box:platform removeAllContainers: stopping and removing 3ca2ffe09606 2025-04-06T00:30:54.687Z box:shell platform: docker stop 3ca2ffe09606 2025-04-06T00:31:04.508Z box:shell platform: docker rm -f 3ca2ffe09606 2025-04-06T00:31:06.291Z box:platform removeAllContainers: stopping and removing 197cd2129268 2025-04-06T00:31:06.292Z box:shell platform: docker stop 197cd2129268 2025-04-06T00:31:09.682Z box:shell platform: docker rm -f 197cd2129268 2025-04-06T00:31:10.643Z box:platform removeAllContainers: stopping and removing 0bfbc0e4cc90 2025-04-06T00:31:10.643Z box:shell platform: docker stop 0bfbc0e4cc90 2025-04-06T00:31:16.299Z box:shell platform: docker rm -f 0bfbc0e4cc90 2025-04-06T00:31:17.164Z box:platform removeAllContainers: stopping and removing c87aa4f6d6e9 2025-04-06T00:31:17.164Z box:shell platform: docker stop c87aa4f6d6e9 2025-04-06T00:31:29.125Z box:shell platform: docker rm -f c87aa4f6d6e9
-
2025-04-06T00:31:29.650Z box:platform removeAllContainers: stopping and removing 82ff73d0f943 2025-04-06T00:31:29.651Z box:shell platform: docker stop 82ff73d0f943 2025-04-06T00:31:40.939Z box:shell platform: docker rm -f 82ff73d0f943 2025-04-06T00:31:41.380Z box:platform removeAllContainers: stopping and removing b8ebac43901c 2025-04-06T00:31:41.380Z box:shell platform: docker stop b8ebac43901c 2025-04-06T00:31:47.702Z box:shell platform: docker rm -f b8ebac43901c 2025-04-06T00:31:48.170Z box:platform createDockerNetwork: recreating docker network 2025-04-06T00:31:48.170Z box:shell platform: docker network rm -f cloudron 2025-04-06T00:31:48.942Z box:shell platform: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway=172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron 2025-04-06T00:31:49.408Z box:services startServices: 48.0.0 -> 49.8.0. starting all services 2025-04-06T00:31:49.408Z box:mailserver startMail: starting 2025-04-06T00:31:49.411Z box:mailserver restart: restarting mail container with mailFqdn:my.hopefullylevelheaded.com mailDomain:hopefullylevelheaded.com 2025-04-06T00:31:49.507Z box:locks write: current locks: {"mail_restart":null} 2025-04-06T00:31:49.507Z box:locks acquire: mail_restart 2025-04-06T00:31:49.518Z box:mailserver configureMail: stopping and deleting previous mail container 2025-04-06T00:31:49.537Z box:mailserver createMailConfig: generating mail config with my.hopefullylevelheaded.com 2025-04-06T00:31:49.612Z box:mailserver configureMail: starting mail container 2025-04-06T00:31:49.613Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail --net cloudron --net-alias mail --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mail -m 671088640 --memory-swap -1 --dns 172.18.0.1 --dns-search=. --ip 172.18.30.4 -e CLOUDRON_MAIL_TOKEN=b1323c8162862c3c9f73cfddd65cdc37761b1316b80ea7b626a257b3aec90e077007ac33255b0855191299f33b490228b9904af26b251ed6876fd9e56d54c4c88ec1111884c6d9559e598d3785b2de4f0f89f96297a980b406e5ea6a5078f4727023f2886f34e5726624be27c9d2b1fd73770d126c740f035ce6bbf37e4d697e -e CLOUDRON_RELAY_TOKEN=3c622e0c2f602488debbfaab91e101a164ddc8058b08bd83eac9f4c011ede0c5c5804d63999f72e44aa5bd694b44c44776d47b5e025a9917dc3e5c6b2a024c36ce3829c2626d6223bcfb26b8b47491721cd2ecc7d8493407139d92a294ae05d26379f197d948f0086069ad65e59fa16544e415d40c139d584dddeb115d2d9502 -e LOGLEVEL=info -v /home/yellowtent/boxdata/mail:/app/data -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995 --label isCloudronManaged=true --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536 2025-04-06T00:31:58.071Z box:locks write: current locks: {} 2025-04-06T00:31:58.071Z box:locks release: mail_restart 2025-04-06T00:31:58.071Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e 2025-04-06T00:31:58.079Z box:services startTurn: stopping and deleting previous turn container 2025-04-06T00:31:58.085Z box:services startTurn: starting turn container 2025-04-06T00:31:58.085Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn --hostname turn --net host --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=turn -m 268435456 --memory-swap -1 -e CLOUDRON_TURN_SECRET=a07d85effc3c5732cde5341587d878d -e CLOUDRON_REALM=my.hopefullylevelheaded.com --label isCloudronManaged=true --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570 2025-04-06T00:32:02.815Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name 2025-04-06T00:32:03.400Z box:shell services: grep -q avx /proc/cpuinfo 2025-04-06T00:32:03.720Z box:shell services: systemctl is-active unbound 2025-04-06T00:32:03.728Z box:shell services: systemctl is-active nginx 2025-04-06T00:32:06.333Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d 2025-04-06T00:32:06.337Z box:services startMysql: stopping and deleting previous mysql container 2025-04-06T00:32:06.343Z box:services startMysql: starting mysql container 2025-04-06T00:32:06.344Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mysql --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN=985d03b1df1c62feffc9fe7064da8300b7793d9ebf437ad4b33b3bca6f3b0cb21a88cda345c2e2edd6a4f03d100ad0f1d52239f4dd2d3615164ae0d31b420cc9ce336ab325c887a972df953d557ad9ce08e4f7bb31ec4bd2f544bc9695fa888bd9e65b641d91401f0cb7e8d1e98cb67f7247770a7a20c2c91c42d51375463f64 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=31803f70cf6bfdc77e859e7d182c52633bf2e6de21b6e494e52c4d1fbb47703f3a71b54f05e041117e0e6c9694c88c416f79522ecc0111585da86b74ffb9c567c45cf095a26de5396bf7ec0187b2bb722f16ed5afe9cdf97dab8b494492acded7818de789efb10833d498839d1e8c06616fc9c0f08b9901bfad13c5b7d7bb657 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280 2025-04-06T00:32:10.503Z box:services Waiting for mysql 2025-04-06T00:32:10.509Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000 2025-04-06T00:32:25.511Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000 2025-04-06T00:32:40.753Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306 2025-04-06T00:32:56.076Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2 2025-04-06T00:32:56.080Z box:services startPostgresql: postgresql will be upgraded 2025-04-06T00:32:56.080Z box:services exportDatabase: Exporting postgresql 2025-04-06T00:32:56.085Z box:services exportDatabase: Exporting addon postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c 2025-04-06T00:32:56.086Z box:services Backing up postgresql 2025-04-06T00:32:56.089Z box:services exportDatabase: Error exporting postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c. BoxError: Unable to find container postgresql at Object.inspect (/home/yellowtent/box/src/docker.js:607:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20) at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20) { reason: 'Not found', details: {} } 2025-04-06T00:32:56.089Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Unable to find container postgresql 2025-04-06T00:32:56.089Z box:platform BoxError: Unable to find container postgresql at Object.inspect (/home/yellowtent/box/src/docker.js:607:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20) at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20)
-
I see a pattern that everytime it postgresql tries to touch app
3147d06c-a773-4d59-b8d7-170517ef116c
it errors.
What app is3147d06c-a773-4d59-b8d7-170517ef116c
on your system? -
That would be my nextcloud app. I could maybe try to archive the app to preserve the whole system. I don't appear to have much recourse from the GUI due to the app being busy, but if there's some other CLI command I could run, let me know.
I also see these connection refused errors with redis, which also stay down upon reboot:
2025-04-06T16:48:40.302Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-3147d06c-a773-4d59-b8d7-170517ef116c: connect ECONNREFUSED 172.18.0.2:3000 2025-04-06T16:48:43.538Z box:services Waiting for redis-6deb5899-a69c-4c85-a050-7fe5dbdcac8a 2025-04-06T16:48:43.543Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-6deb5899-a69c-4c85-a050-7fe5dbdcac8a: connect ECONNREFUSED 172.18.0.3:3000
Is this normal behavior? I've never had a firewall issue with this instance before. Seems like all of this started happening after 8.3.1.
-
Okay, so it seems like postgresql is actually running fine, but cloudron/docker can't seem to pass a health check:
[GET] /healthcheck healthcheck: disconnected [GET] /healthcheck healthcheck: disconnected
-
Okay, so it seems like postgresql is actually running fine, but cloudron/docker can't seem to pass a health check:
[GET] /healthcheck healthcheck: disconnected [GET] /healthcheck healthcheck: disconnected
@stratus41298 for that you need a web service that will have a 200 response on that /healthcheck endpoint
-
@stratus41298 for that you need a web service that will have a 200 response on that /healthcheck endpoint
@robi I'm not really sure where you're going with that so I can't comment.
As for the continued troubleshooting, I was able to get postgresql up manually with extensive help of ChatGPT. We ran into a DNS issue trying to do an APT reinstall of docker even after adjusting the resolv.conf. If anyone is interested I can give a full writeup, but i'll summarize here:
🧩 Cloudron 8.3.1 Upgrade – PostgreSQL & DNS Issues Summary
1. PostgreSQL Socket Failure
-
After updating to Cloudron 8.3.1, the PostgreSQL service showed as down in the Cloudron dashboard.
-
However, logs showed the database was running fine and accepting connections over TCP.
-
Cloudron healthchecks (and dependent services) failed due to the absence of the expected Unix socket at
/var/run/postgresql/.s.PGSQL.5432
. -
Investigation showed:
- The actual socket was created in
/run/postgresql/
- The symlink
/var/run/postgresql
→/run/postgresql
was missing inside the PostgreSQL container
- The actual socket was created in
-
Fix: Re-created the symlink inside the container:
ln -s /run/postgresql /var/run/postgresql
Followed by:
supervisorctl restart postgresql
-
After this, PostgreSQL healthcheck passed, and dependent services began recovering.
2. APT Repository DNS/TLS Failure
-
While attempting to reinstall the Cloudron CLI to recover from a broken
docker recreate
, the APT repo athttps://apt.cloudron.io
was unreachable. -
DNS (
dig apt.cloudron.io
) returned SOA records but no A records — even after:- Disabling
systemd-resolved
- Stopping
unbound
- Overriding
/etc/resolv.conf
to use8.8.8.8
- Confirming DNS worked for other domains (
cloudron.io
,google.com
)
- Disabling
-
Attempts to work around with
/etc/hosts
resulted in a TLS handshake failure:error:0A000410:SSL routines::sslv3 alert handshake failure
-
Root cause appears to be Cloudflare (which fronts
apt.cloudron.io
) rejecting connections when:- DNS is bypassed via
/etc/hosts
, or - Something in the server environment breaks SNI or TLS negotiation
- DNS is bypassed via
-
Result: Unable to reinstall
cloudron
CLI using APT despite proper DNS config and working CA certs.
🧠Context
- This all occurred during/after a failed
cloudron docker recreate
attempt (as suggested on the forums). - That left the system partially repaired, with missing CLI tools and broken DNS.
So that's pretty much it. At this point i'm going to just set up a restore either through VM backup or Cloudron backup. I haven't decided yet. I won't do it until tomorrow though in case there's anything you'd like me to provide for future maintenance or research.
-
-
So whatever you have posted here, makes no sense in the cloudron context, some of those commands don't even exist, nor is there any cloudron apt repository (apt.cloudron.io does and should not exist). Much AI hallucination happening here it seems
Looks like we have to start from first principles, first mysql didn't work then postgres and maybe redis...? Also it is probably not clear by now in which state the server is and what would have to be undone or not. Given all that, if you have recent backups a fresh start is probably the fastest and safest option to get to some well known state.
-
That makes sense that there was no apt repo, but the fact was that we couldn't even get out to apt due to the DNS issue so i'd caution an immediate dismissal of AI hallucination. We were also able to get the healthcheck up and running for postgresql after it had gotten stuck during the docker recreate (see forked post) using AI as well so i'm not sure how many of the commands were inappropriate as they all went through. We did indeed disable resolved, stopped unbound, overrode resolv.conf (which i've done before for pihole many times), digged DNS, bypassed DNS using hosts, and I myself had previously performed a docker recreate. All of those commands exist except I guess the apt reinstall, but that's something I myself might have tried, if i'm being honest as it's a fairly common command on other apps.
After the 8.3.1 update many services wouldn't come up and nothing worked on this instance. I guess if you have all the information you care about on this issue, i'll go ahead and restore since i've been down for quite a while at this point.
I get the whole "AI = bad" thing, but I spent about 2 hours troubleshooting with GPT 4o (the paid version) and made significant progress until we hit a wall that we couldn't get around, which at least gave me satisfaction that we tried everything we could.
Take care.
-
-
I am generally not in the group of AI = bad, however it likely just lacks lots of knowledge about the Cloudron internals, which make recommendations often not very useful in this area (yet).
Overall I am still not sure based on this thread why DNS was involved here and given that you hit 3 different addons with apparently diverse issues, maybe something unrelated to them individually caused the trouble.
We will add a few more helpers to the
cloudron-support
script for the future, to hopefully get down to such issues faster.If you haven't deleted the server yet, maybe we could take a closer look via SSH, since you already invested much time in debugging. If you want that, please enable remote ssh support and send a mail to support@cloudron.io with your Cloudron details.