MySQL addon not starting

stratus41298

2025-04-01T01:05:37.462Z box:shell platform: docker stop 913d0e63ddaf
2025-04-01T01:05:43.061Z box:shell platform: docker rm -f 913d0e63ddaf
2025-04-01T01:05:43.496Z box:platform createDockerNetwork: recreating docker network
2025-04-01T01:05:43.497Z box:shell platform: docker network rm -f cloudron
2025-04-01T01:05:44.274Z box:shell platform: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway=172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron
2025-04-01T01:05:44.763Z box:services startServices: 48.0.0 -> 49.8.0. starting all services
2025-04-01T01:05:44.764Z box:mailserver startMail: starting
2025-04-01T01:05:44.768Z box:mailserver restart: restarting mail container with mailFqdn:my.hopefullylevelheaded.com mailDomain:hopefullylevelheaded.com
2025-04-01T01:05:44.872Z box:locks write: current locks: {"mail_restart":null}
2025-04-01T01:05:44.872Z box:locks acquire: mail_restart
2025-04-01T01:05:44.886Z box:mailserver configureMail: stopping and deleting previous mail container
2025-04-01T01:05:44.912Z box:mailserver createMailConfig: generating mail config with my.hopefullylevelheaded.com
2025-04-01T01:05:44.922Z box:mailserver configureMail: starting mail container
2025-04-01T01:05:44.922Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail                 --net cloudron                 --net-alias mail                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424  --log-opt tag=mail                 -m 671088640                 --memory-swap -1                 --dns 172.18.0.1            --dns-search=.                 --ip 172.18.30.4                 -e CLOUDRON_MAIL_TOKEN=c05f8214c1146e22a46316d68fe6874190ca2a180fc8b9aa3130fdd3585a62129c67620b24bef204078186a619bc0929272a6e57b3f66886fee7ed9312b64d6243dd07a374e6af3b584a9b815283d4de46b9e2ededc00c74f820100f2056933c63d6599add108cf525d8d6d710d8f3586205b5049d84ffa86866d78a4a8d8ea6            -e CLOUDRON_RELAY_TOKEN=ca6423deb379ceeb825a09fc1a636586bd0a4fa223c8ffd6e0e48e666c721a85e255ab0a7d538ab0d2f942da4af54fb0832369327e6d7ea467088809cb2312c8313d0d4306377c2692c89afbb735cec1a8c3cb045f5fc5af325a14f9787118b5e7a1c3f86fdad3ae20035fce5be5c66733b73ba493418dce18261e150c64fb28                 -e LOGLEVEL=info                 -v /home/yellowtent/boxdata/mail:/app/data                 -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro                 -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995                 --label isCloudronManaged=true          --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536
2025-04-01T01:05:49.604Z box:locks write: current locks: {}
2025-04-01T01:05:49.604Z box:locks release: mail_restart
2025-04-01T01:05:49.605Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e
2025-04-01T01:05:49.614Z box:services startTurn: stopping and deleting previous turn container
2025-04-01T01:05:49.624Z box:services startTurn: starting turn container
2025-04-01T01:05:49.624Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn                 --hostname turn                 --net host                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=turn                 -m 268435456                 --memory-swap -1                 -e CLOUDRON_TURN_SECRET=a07d85effc3c5732cde5341587d878d                 -e CLOUDRON_REALM=my.hopefullylevelheaded.com                 --label isCloudronManaged=true                 --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570
2025-04-01T01:05:53.563Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d
2025-04-01T01:05:53.569Z box:services startMysql: stopping and deleting previous mysql container
2025-04-01T01:05:53.582Z box:services startMysql: starting mysql container
2025-04-01T01:05:53.582Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql                 --hostname mysql                 --net cloudron                 --net-alias mysql                 --log-driver syslog            --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mysql                 --ip 172.18.30.1                 -e CLOUDRON_MYSQL_TOKEN=5f41ef1daee14c3a5d31b5dc12122dd7211de3fd802bbd4292a254fe2766fd684db952e57f4bbecc3f03b40edc831c25e0b1e1d02e4499ea94ead7c64d635158db9d0ba9d4235a801bbfa23c6a7d9b80294d84df3d8670eebf94044eeca50b678cc392073df05501d99f30651185e1525a708c60c43d2bc07b4766c8aecc21ba                 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1                 -e CLOUDRON_MYSQL_ROOT_PASSWORD=ed8e8295784b922e63e25e84dafc9553283970779f1681d89067bb8565d5f446490a939ebcbbb426b38987a394ce9946d3c466d07efbc6b82cf3a581a0bbdbf8d01696e129dfe7cb58258866114851a6414e4c04defb45f12c7d17b2dd8107a58978ede718699d8b842bef11880ab88b760fdc7d83118e1764280cf8129aed45                 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql                 --label isCloudronManaged=true                 --cap-add SYS_NICE                 --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280
2025-04-01T01:05:57.358Z box:services Waiting for mysql
2025-04-01T01:05:57.367Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-01T01:06:12.375Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-01T01:06:27.631Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306
2025-04-01T01:06:42.874Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2
2025-04-01T01:06:42.877Z box:services startPostgresql: postgresql will be upgraded
2025-04-01T01:06:42.878Z box:services exportDatabase: Exporting postgresql
2025-04-01T01:06:42.885Z box:services exportDatabase: Exporting addon postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c
2025-04-01T01:06:42.885Z box:services Backing up postgresql
2025-04-01T01:06:42.888Z box:services exportDatabase: Error exporting postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c. BoxError: Unable to find container postgresql at Object.inspect (/home/yellowtent/box/src/docker.js:607:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20) at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20) { reason: 'Not found', details: {} }
2025-04-01T01:06:42.888Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Unable to find container postgresql
2025-04-01T01:06:42.888Z box:platform BoxError: Unable to find container postgresql
    at Object.inspect (/home/yellowtent/box/src/docker.js:607:50)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20)
    at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20)
2025-04-01T01:07:06.871Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-04-01T01:07:07.304Z box:shell services: grep -q avx /proc/cpuinfo
2025-04-01T01:07:07.396Z box:shell services: systemctl is-active unbound
2025-04-01T01:07:07.509Z box:shell services: systemctl is-active nginx
2025-04-01T01:07:27.807Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-04-01T01:07:28.722Z box:shell mounts: mountpoint -q -- /mnt/cloudronbackup
2025-04-01T01:07:52.957Z box:shell support /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remotesupport.sh is-enabled /home/cloudron-support/.ssh/authorized_keys
2025-04-01T01:07:52.968Z box:shell grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory

2025-04-01T01:07:52.969Z box:shell false

2025-04-01T01:13:29.280Z box:user notifyLoginLocation: uid-dc448a93-1ce7-4d9a-ac7a-e1faba413c8d 192.168.50.2 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36
2025-04-01T01:14:15.784Z box:shell support /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remotesupport.sh is-enabled /home/cloudron-support/.ssh/authorized_keys
2025-04-01T01:14:15.795Z box:shell grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory

2025-04-01T01:14:15.795Z box:shell false

2025-04-01T01:14:58.298Z box:box Received SIGTERM. Shutting down.
2025-04-01T01:14:58.298Z box:platform uninitializing platform
2025-04-01T01:14:58.300Z box:platform onDeactivated: stopping post activation services
2025-04-01T01:14:58.300Z box:tasks stopAllTasks: stopping all tasks
2025-04-01T01:14:58.300Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-04-01T01:14:58.328Z box:shell All tasks stopped

2025-04-01T01:15:02.313Z box:server ==========================================
2025-04-01T01:15:02.313Z box:server            Cloudron 8.3.1
2025-04-01T01:15:02.313Z box:server ==========================================
2025-04-01T01:15:02.314Z box:platform initialize: start platform
2025-04-01T01:15:02.348Z box:tasks stopAllTasks: stopping all tasks
2025-04-01T01:15:02.348Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-04-01T01:15:02.383Z box:shell All tasks stopped

2025-04-01T01:15:02.850Z box:locks releaseAll: all locks released
2025-04-01T01:15:02.856Z box:reverseproxy writeDashboardConfig: writing dashboard config for hopefullylevelheaded.com
2025-04-01T01:15:02.863Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/_.hopefullylevelheaded.com.cert -noout -ocsp_uri
2025-04-01T01:15:02.874Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-04-01T01:15:03.278Z box:platform onActivated: starting post activation services
2025-04-01T01:15:03.278Z box:platform startInfra: checking infrastructure
2025-04-01T01:15:03.279Z box:platform startInfra: updating infrastructure from 48.0.0 to 49.8.0
2025-04-01T01:15:03.298Z box:platform markApps: reconfiguring apps
2025-04-01T01:15:03.298Z box:reverseproxy removeAppConfigs: removing app nginx configs
2025-04-01T01:15:03.330Z box:platform removeAllContainers: removing all containers for infra upgrade
2025-04-01T01:15:03.330Z box:shell platform: docker ps -qa --filter label=isCloudronManaged
2025-04-01T01:15:03.360Z box:platform removeAllContainers: stopping and removing 5b25b2c59cbe
2025-04-01T01:15:03.360Z box:shell platform: docker stop 5b25b2c59cbe
2025-04-01T01:15:08.502Z box:shell platform: docker rm -f 5b25b2c59cbe
2025-04-01T01:15:09.039Z box:platform removeAllContainers: stopping and removing 3af5ff670ea1
2025-04-01T01:15:09.039Z box:shell platform: docker stop 3af5ff670ea1
2025-04-01T01:15:20.257Z box:shell platform: docker rm -f 3af5ff670ea1
2025-04-01T01:15:20.708Z box:platform removeAllContainers: stopping and removing 1c45f903edf4
2025-04-01T01:15:20.709Z box:shell platform: docker stop 1c45f903edf4
2025-04-01T01:15:26.517Z box:shell platform: docker rm -f 1c45f903edf4
2025-04-01T01:15:26.902Z box:platform createDockerNetwork: recreating docker network
2025-04-01T01:15:26.902Z box:shell platform: docker network rm -f cloudron
2025-04-01T01:15:27.765Z box:shell platform: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway=172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron
2025-04-01T01:15:28.203Z box:services startServices: 48.0.0 -> 49.8.0. starting all services
2025-04-01T01:15:28.203Z box:mailserver startMail: starting
2025-04-01T01:15:28.207Z box:mailserver restart: restarting mail container with mailFqdn:my.hopefullylevelheaded.com mailDomain:hopefullylevelheaded.com
2025-04-01T01:15:28.313Z box:locks write: current locks: {"mail_restart":null}
2025-04-01T01:15:28.313Z box:locks acquire: mail_restart
2025-04-01T01:15:28.324Z box:mailserver configureMail: stopping and deleting previous mail container
2025-04-01T01:15:28.343Z box:mailserver createMailConfig: generating mail config with my.hopefullylevelheaded.com
2025-04-01T01:15:28.347Z box:mailserver configureMail: starting mail container
2025-04-01T01:15:28.347Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail                 --net cloudron                 --net-alias mail                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424  --log-opt tag=mail                 -m 671088640                 --memory-swap -1                 --dns 172.18.0.1            --dns-search=.                 --ip 172.18.30.4                 -e CLOUDRON_MAIL_TOKEN=e193c77f0c7fbaea669901e75397e2eb802f1b6b4d3195d1e60a8be234636f49620c3121561c6757a09f7afb9e5825a57aee74e6370124f7b539621f983bcd4ae587be6d1acdf8fc67b15832aba0354fecc37458bc5d56fccbb9bdee25f994583c2cba28f6f9d48353d141b0b44f44134b1fdfcb748a1b59daa1bceff9c1f503            -e CLOUDRON_RELAY_TOKEN=d1749634d5113fe1c4c09cdeaa0ddf6cdf473080c6a7893cd9c36dd64707d548020fa65859ea4f65339e2b2d34b0e7e004fc4f3aa7af0a03de2d4fcf57b3d25d778ab279fd11f46fc38015f395c97682693e0086a0c6b6d4f8555828a68259ad26d8e1f409171a343ed1eb30189abe7ddb4de9ec64d472b5389e307c911be2ab                 -e LOGLEVEL=info                 -v /home/yellowtent/boxdata/mail:/app/data                 -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro                 -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995                 --label isCloudronManaged=true          --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536
2025-04-01T01:15:32.783Z box:locks write: current locks: {}
2025-04-01T01:15:32.783Z box:locks release: mail_restart
2025-04-01T01:15:32.784Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e
2025-04-01T01:15:32.791Z box:services startTurn: stopping and deleting previous turn container
2025-04-01T01:15:32.796Z box:services startTurn: starting turn container
2025-04-01T01:15:32.796Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn                 --hostname turn                 --net host                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=turn                 -m 268435456                 --memory-swap -1                 -e CLOUDRON_TURN_SECRET=a07d85effc3c5732cde5341587d878d                 -e CLOUDRON_REALM=my.hopefullylevelheaded.com                 --label isCloudronManaged=true                 --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570
2025-04-01T01:15:36.909Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d
2025-04-01T01:15:36.926Z box:services startMysql: stopping and deleting previous mysql container
2025-04-01T01:15:36.933Z box:services startMysql: starting mysql container
2025-04-01T01:15:36.933Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql                 --hostname mysql                 --net cloudron                 --net-alias mysql                 --log-driver syslog            --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mysql                 --ip 172.18.30.1                 -e CLOUDRON_MYSQL_TOKEN=1885ed4ab1851e763171680d3b7443077f43a345fc4b8222d183bcd21437ae56d3a09c502becd8bdb23dcd25f2e4c97bf1ae52eb47ee8e83298b1547e828d1bd23c4620cd2978742ab9bd0c95e9a555fb912c9690cbb745f2dd1e88b4b325ce6f138d0f05a9c2ce9454cb78799fe9f4a032f6eb68fb3b27063f88742cb6e715a                 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1                 -e CLOUDRON_MYSQL_ROOT_PASSWORD=3e8ce027c04caea01a1a956d46d7c9fad00d00b79d26d27f9439662e96fbaf38bf86f7fb6fd3191426cf29969a0b06cab4a9bf5206fff01de0abc3ae450631881868bb2475f628c61f3335c4c427a8ce16dee38e4b7a61c1936b6c75575c77933cf18fafc1a3dbb99fce4e44dcba57ad7b0cbf5a08d9c54a88e21aa7eadb5629                 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql                 --label isCloudronManaged=true                 --cap-add SYS_NICE                 --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280
2025-04-01T01:15:42.220Z box:services Waiting for mysql
2025-04-01T01:15:42.230Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-01T01:15:57.233Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-01T01:16:12.471Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306
2025-04-01T01:16:28.345Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2
2025-04-01T01:16:28.348Z box:services startPostgresql: postgresql will be upgraded
2025-04-01T01:16:28.349Z box:services exportDatabase: Exporting postgresql
2025-04-01T01:16:28.354Z box:services exportDatabase: Exporting addon postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c
2025-04-01T01:16:28.354Z box:services Backing up postgresql
2025-04-01T01:16:28.357Z box:services exportDatabase: Error exporting postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c. BoxError: Unable to find container postgresql at Object.inspect (/home/yellowtent/box/src/docker.js:607:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20) at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20) { reason: 'Not found', details: {} }
2025-04-01T01:16:28.358Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Unable to find container postgresql
2025-04-01T01:16:28.358Z box:platform BoxError: Unable to find container postgresql
    at Object.inspect (/home/yellowtent/box/src/docker.js:607:50)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20)
    at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20)
2025-04-02T22:37:24.205Z box:shell support /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remotesupport.sh is-enabled /home/cloudron-support/.ssh/authorized_keys
2025-04-02T22:37:24.216Z box:shell grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory

2025-04-02T22:37:24.216Z box:shell false

2025-04-02T22:37:44.881Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
HttpError: Unknown error: 400 <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

    at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  status: 500,
  internalError: BoxError: Unknown error: 400 <html>
  <head><title>400 Bad Request</title></head>
  <body>
  <center><h1>400 Bad Request</h1></center>
  <hr><center>nginx</center>
  </body>
  </html>

      at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
    reason: 'External Error',
    details: {}
  },
  details: {}
}

I'll be away all day tomorrow so take your time. Thank you for your help!

nebulon

Thanks for the detailed logs. For a start it seems mysql is actually not the issue but the postgresql addon causes trouble. Is that service up and running?

The Expected "actual" to be strictly unequal to: null error can be ignored here, as it just happened while the process was winding down and the mysql (host instance, not service addon, Cloudron has two) connection was already terminated. Not nice to assert here, but also harmless.

stratus41298

When I first noticed the problem that and mysql were down. I was able to restart postgres but not mysql. Today I was able to restart all services, but there was no effect to the environment. I resumed the docker recreate and am now waiting for a result. I'll let you know. Thank you.

stratus41298

Alas, still seeing the same thing after about six hours. I noticed that the recreate-docker shut down a lot of services so I bounced them again and they are showing up in the event log as rebuilt, but it's seemed to have no effect in the terminal; just endless dots. There is a reboot required so i'm tempted to do that and see how it goes, but also don't want to interrupt the process if it's actually doing something.
Screenshot 2025-04-05 163423.png

BrutalBirdie

I've seen something like this before.
=> https://forum.cloudron.io/post/94377

Can you please share the box.log of that failed attempt?

stratus41298

Much less in there this time:

@cloudron:~$ sudo cat /home/yellowtent/platformdata/logs/box.log
[sudo] password for :
2025-04-06T00:18:41.199Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-04-06T00:18:41.849Z box:shell services: grep -q avx /proc/cpuinfo
2025-04-06T00:18:42.065Z box:shell services: systemctl is-active unbound
2025-04-06T00:18:42.132Z box:shell services: systemctl is-active nginx
2025-04-06T00:18:51.922Z box:shell support /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remotesupport.sh is-enabled /home/cloudron-support/.ssh/authorized_keys
2025-04-06T00:18:51.933Z box:shell grep: /home/cloudron-support/.ssh/authorized_keys: No such file or directory

2025-04-06T00:18:51.933Z box:shell false

stratus41298

Did that reboot it was asking for. Came back and the same services were down again.

New box.log after reboot:

Screenshot 2025-04-05 203212.png

2025-04-06T00:30:35.868Z box:server ==========================================
2025-04-06T00:30:35.870Z box:server            Cloudron 8.3.1
2025-04-06T00:30:35.870Z box:server ==========================================
2025-04-06T00:30:35.870Z box:platform initialize: start platform
2025-04-06T00:30:36.079Z box:tasks stopAllTasks: stopping all tasks
2025-04-06T00:30:36.079Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2025-04-06T00:30:36.268Z box:shell All tasks stopped

2025-04-06T00:30:37.082Z box:locks releaseAll: all locks released
2025-04-06T00:30:37.158Z box:reverseproxy writeDashboardConfig: writing dashboard config for hopefullylevelheaded.com
2025-04-06T00:30:37.227Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/_.hopefullylevelheaded.com.cert -noout -ocsp_uri
2025-04-06T00:30:37.265Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-04-06T00:30:39.353Z box:platform onActivated: starting post activation services
2025-04-06T00:30:39.353Z box:platform startInfra: checking infrastructure
2025-04-06T00:30:39.365Z box:platform startInfra: updating infrastructure from 48.0.0 to 49.8.0
2025-04-06T00:30:39.365Z box:platform markApps: reconfiguring apps
2025-04-06T00:30:39.366Z box:reverseproxy removeAppConfigs: removing app nginx configs
2025-04-06T00:30:39.479Z box:platform removeAllContainers: removing all containers for infra upgrade
2025-04-06T00:30:39.479Z box:shell platform: docker ps -qa --filter label=isCloudronManaged
2025-04-06T00:30:39.502Z box:platform removeAllContainers: stopping and removing e77cfa1fb71d
2025-04-06T00:30:39.502Z box:shell platform: docker stop e77cfa1fb71d
2025-04-06T00:30:44.009Z box:shell platform: docker rm -f e77cfa1fb71d
2025-04-06T00:30:44.967Z box:platform removeAllContainers: stopping and removing 3af0b28f8899
2025-04-06T00:30:44.967Z box:shell platform: docker stop 3af0b28f8899
2025-04-06T00:30:48.236Z box:shell platform: docker rm -f 3af0b28f8899
2025-04-06T00:30:49.250Z box:platform removeAllContainers: stopping and removing 254b1c19f3af
2025-04-06T00:30:49.250Z box:shell platform: docker stop 254b1c19f3af
2025-04-06T00:30:53.888Z box:shell platform: docker rm -f 254b1c19f3af
2025-04-06T00:30:54.687Z box:platform removeAllContainers: stopping and removing 3ca2ffe09606
2025-04-06T00:30:54.687Z box:shell platform: docker stop 3ca2ffe09606
2025-04-06T00:31:04.508Z box:shell platform: docker rm -f 3ca2ffe09606
2025-04-06T00:31:06.291Z box:platform removeAllContainers: stopping and removing 197cd2129268
2025-04-06T00:31:06.292Z box:shell platform: docker stop 197cd2129268
2025-04-06T00:31:09.682Z box:shell platform: docker rm -f 197cd2129268
2025-04-06T00:31:10.643Z box:platform removeAllContainers: stopping and removing 0bfbc0e4cc90
2025-04-06T00:31:10.643Z box:shell platform: docker stop 0bfbc0e4cc90
2025-04-06T00:31:16.299Z box:shell platform: docker rm -f 0bfbc0e4cc90
2025-04-06T00:31:17.164Z box:platform removeAllContainers: stopping and removing c87aa4f6d6e9
2025-04-06T00:31:17.164Z box:shell platform: docker stop c87aa4f6d6e9
2025-04-06T00:31:29.125Z box:shell platform: docker rm -f c87aa4f6d6e9

stratus41298

2025-04-06T00:31:29.650Z box:platform removeAllContainers: stopping and removing 82ff73d0f943
2025-04-06T00:31:29.651Z box:shell platform: docker stop 82ff73d0f943
2025-04-06T00:31:40.939Z box:shell platform: docker rm -f 82ff73d0f943
2025-04-06T00:31:41.380Z box:platform removeAllContainers: stopping and removing b8ebac43901c
2025-04-06T00:31:41.380Z box:shell platform: docker stop b8ebac43901c
2025-04-06T00:31:47.702Z box:shell platform: docker rm -f b8ebac43901c
2025-04-06T00:31:48.170Z box:platform createDockerNetwork: recreating docker network
2025-04-06T00:31:48.170Z box:shell platform: docker network rm -f cloudron
2025-04-06T00:31:48.942Z box:shell platform: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway=172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron
2025-04-06T00:31:49.408Z box:services startServices: 48.0.0 -> 49.8.0. starting all services
2025-04-06T00:31:49.408Z box:mailserver startMail: starting
2025-04-06T00:31:49.411Z box:mailserver restart: restarting mail container with mailFqdn:my.hopefullylevelheaded.com mailDomain:hopefullylevelheaded.com
2025-04-06T00:31:49.507Z box:locks write: current locks: {"mail_restart":null}
2025-04-06T00:31:49.507Z box:locks acquire: mail_restart
2025-04-06T00:31:49.518Z box:mailserver configureMail: stopping and deleting previous mail container
2025-04-06T00:31:49.537Z box:mailserver createMailConfig: generating mail config with my.hopefullylevelheaded.com
2025-04-06T00:31:49.612Z box:mailserver configureMail: starting mail container
2025-04-06T00:31:49.613Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail                 --net cloudron                 --net-alias mail                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mail                 -m 671088640                 --memory-swap -1                 --dns 172.18.0.1                 --dns-search=.                 --ip 172.18.30.4                 -e CLOUDRON_MAIL_TOKEN=b1323c8162862c3c9f73cfddd65cdc37761b1316b80ea7b626a257b3aec90e077007ac33255b0855191299f33b490228b9904af26b251ed6876fd9e56d54c4c88ec1111884c6d9559e598d3785b2de4f0f89f96297a980b406e5ea6a5078f4727023f2886f34e5726624be27c9d2b1fd73770d126c740f035ce6bbf37e4d697e                 -e CLOUDRON_RELAY_TOKEN=3c622e0c2f602488debbfaab91e101a164ddc8058b08bd83eac9f4c011ede0c5c5804d63999f72e44aa5bd694b44c44776d47b5e025a9917dc3e5c6b2a024c36ce3829c2626d6223bcfb26b8b47491721cd2ecc7d8493407139d92a294ae05d26379f197d948f0086069ad65e59fa16544e415d40c139d584dddeb115d2d9502                 -e LOGLEVEL=info                 -v /home/yellowtent/boxdata/mail:/app/data                 -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro                 -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995                 --label isCloudronManaged=true                 --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536
2025-04-06T00:31:58.071Z box:locks write: current locks: {}
2025-04-06T00:31:58.071Z box:locks release: mail_restart
2025-04-06T00:31:58.071Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e
2025-04-06T00:31:58.079Z box:services startTurn: stopping and deleting previous turn container
2025-04-06T00:31:58.085Z box:services startTurn: starting turn container
2025-04-06T00:31:58.085Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn                 --hostname turn                 --net host                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=turn                 -m 268435456                 --memory-swap -1                 -e CLOUDRON_TURN_SECRET=a07d85effc3c5732cde5341587d878d                 -e CLOUDRON_REALM=my.hopefullylevelheaded.com                 --label isCloudronManaged=true                 --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570
2025-04-06T00:32:02.815Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
2025-04-06T00:32:03.400Z box:shell services: grep -q avx /proc/cpuinfo
2025-04-06T00:32:03.720Z box:shell services: systemctl is-active unbound
2025-04-06T00:32:03.728Z box:shell services: systemctl is-active nginx
2025-04-06T00:32:06.333Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d
2025-04-06T00:32:06.337Z box:services startMysql: stopping and deleting previous mysql container
2025-04-06T00:32:06.343Z box:services startMysql: starting mysql container
2025-04-06T00:32:06.344Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql                 --hostname mysql                 --net cloudron                 --net-alias mysql                 --log-driver syslog                 --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock                 --log-opt syslog-format=rfc5424                 --log-opt tag=mysql                 --ip 172.18.30.1                 -e CLOUDRON_MYSQL_TOKEN=985d03b1df1c62feffc9fe7064da8300b7793d9ebf437ad4b33b3bca6f3b0cb21a88cda345c2e2edd6a4f03d100ad0f1d52239f4dd2d3615164ae0d31b420cc9ce336ab325c887a972df953d557ad9ce08e4f7bb31ec4bd2f544bc9695fa888bd9e65b641d91401f0cb7e8d1e98cb67f7247770a7a20c2c91c42d51375463f64                 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1                 -e CLOUDRON_MYSQL_ROOT_PASSWORD=31803f70cf6bfdc77e859e7d182c52633bf2e6de21b6e494e52c4d1fbb47703f3a71b54f05e041117e0e6c9694c88c416f79522ecc0111585da86b74ffb9c567c45cf095a26de5396bf7ec0187b2bb722f16ed5afe9cdf97dab8b494492acded7818de789efb10833d498839d1e8c06616fc9c0f08b9901bfad13c5b7d7bb657                 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql                 --label isCloudronManaged=true                 --cap-add SYS_NICE                 --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280
2025-04-06T00:32:10.503Z box:services Waiting for mysql
2025-04-06T00:32:10.509Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-06T00:32:25.511Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000
2025-04-06T00:32:40.753Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306
2025-04-06T00:32:56.076Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2
2025-04-06T00:32:56.080Z box:services startPostgresql: postgresql will be upgraded
2025-04-06T00:32:56.080Z box:services exportDatabase: Exporting postgresql
2025-04-06T00:32:56.085Z box:services exportDatabase: Exporting addon postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c
2025-04-06T00:32:56.086Z box:services Backing up postgresql
2025-04-06T00:32:56.089Z box:services exportDatabase: Error exporting postgresql of app 3147d06c-a773-4d59-b8d7-170517ef116c. BoxError: Unable to find container postgresql at Object.inspect (/home/yellowtent/box/src/docker.js:607:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20) at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20) { reason: 'Not found', details: {} }
2025-04-06T00:32:56.089Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Unable to find container postgresql
2025-04-06T00:32:56.089Z box:platform BoxError: Unable to find container postgresql
    at Object.inspect (/home/yellowtent/box/src/docker.js:607:50)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async getContainerDetails (/home/yellowtent/box/src/services.js:307:20)
    at async Object.backupPostgreSql [as backup] (/home/yellowtent/box/src/services.js:1557:20)

BrutalBirdie

I see a pattern that everytime it postgresql tries to touch app 3147d06c-a773-4d59-b8d7-170517ef116c it errors.
What app is 3147d06c-a773-4d59-b8d7-170517ef116c on your system?

stratus41298

That would be my nextcloud app. I could maybe try to archive the app to preserve the whole system. I don't appear to have much recourse from the GUI due to the app being busy, but if there's some other CLI command I could run, let me know.

I also see these connection refused errors with redis, which also stay down upon reboot:

2025-04-06T16:48:40.302Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-3147d06c-a773-4d59-b8d7-170517ef116c: connect ECONNREFUSED 172.18.0.2:3000
2025-04-06T16:48:43.538Z box:services Waiting for redis-6deb5899-a69c-4c85-a050-7fe5dbdcac8a
2025-04-06T16:48:43.543Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-6deb5899-a69c-4c85-a050-7fe5dbdcac8a: connect ECONNREFUSED 172.18.0.3:3000

Is this normal behavior? I've never had a firewall issue with this instance before. Seems like all of this started happening after 8.3.1.

stratus41298

Okay, so it seems like postgresql is actually running fine, but cloudron/docker can't seem to pass a health check:

[GET] /healthcheck
healthcheck: disconnected
[GET] /healthcheck
healthcheck: disconnected

robi

@stratus41298 for that you need a web service that will have a 200 response on that /healthcheck endpoint

stratus41298

@robi I'm not really sure where you're going with that so I can't comment.

As for the continued troubleshooting, I was able to get postgresql up manually with extensive help of ChatGPT. We ran into a DNS issue trying to do an APT reinstall of docker even after adjusting the resolv.conf. If anyone is interested I can give a full writeup, but i'll summarize here:

🧩 Cloudron 8.3.1 Upgrade – PostgreSQL & DNS Issues Summary

1. PostgreSQL Socket Failure

After updating to Cloudron 8.3.1, the PostgreSQL service showed as down in the Cloudron dashboard.
However, logs showed the database was running fine and accepting connections over TCP.
Cloudron healthchecks (and dependent services) failed due to the absence of the expected Unix socket at /var/run/postgresql/.s.PGSQL.5432.
Investigation showed:
- The actual socket was created in /run/postgresql/
- The symlink /var/run/postgresql → /run/postgresql was missing inside the PostgreSQL container

Fix: Re-created the symlink inside the container:

ln -s /run/postgresql /var/run/postgresql

Followed by:

supervisorctl restart postgresql

After this, PostgreSQL healthcheck passed, and dependent services began recovering.

2. APT Repository DNS/TLS Failure

While attempting to reinstall the Cloudron CLI to recover from a broken docker recreate, the APT repo at https://apt.cloudron.io was unreachable.
DNS (dig apt.cloudron.io) returned SOA records but no A records — even after:
- Disabling systemd-resolved
- Stopping unbound
- Overriding /etc/resolv.conf to use 8.8.8.8
- Confirming DNS worked for other domains (cloudron.io, google.com)
Attempts to work around with /etc/hosts resulted in a TLS handshake failure:
```
error:0A000410:SSL routines::sslv3 alert handshake failure
```
Root cause appears to be Cloudflare (which fronts apt.cloudron.io) rejecting connections when:
- DNS is bypassed via /etc/hosts, or
- Something in the server environment breaks SNI or TLS negotiation
Result: Unable to reinstall cloudron CLI using APT despite proper DNS config and working CA certs.

🧠 Context

This all occurred during/after a failed cloudron docker recreate attempt (as suggested on the forums).
That left the system partially repaired, with missing CLI tools and broken DNS.

So that's pretty much it. At this point i'm going to just set up a restore either through VM backup or Cloudron backup. I haven't decided yet. I won't do it until tomorrow though in case there's anything you'd like me to provide for future maintenance or research.

nebulon

So whatever you have posted here, makes no sense in the cloudron context, some of those commands don't even exist, nor is there any cloudron apt repository (apt.cloudron.io does and should not exist). Much AI hallucination happening here it seems

Looks like we have to start from first principles, first mysql didn't work then postgres and maybe redis...? Also it is probably not clear by now in which state the server is and what would have to be undone or not. Given all that, if you have recent backups a fresh start is probably the fastest and safest option to get to some well known state.

stratus41298

That makes sense that there was no apt repo, but the fact was that we couldn't even get out to apt due to the DNS issue so i'd caution an immediate dismissal of AI hallucination. We were also able to get the healthcheck up and running for postgresql after it had gotten stuck during the docker recreate (see forked post) using AI as well so i'm not sure how many of the commands were inappropriate as they all went through. We did indeed disable resolved, stopped unbound, overrode resolv.conf (which i've done before for pihole many times), digged DNS, bypassed DNS using hosts, and I myself had previously performed a docker recreate. All of those commands exist except I guess the apt reinstall, but that's something I myself might have tried, if i'm being honest as it's a fairly common command on other apps.

After the 8.3.1 update many services wouldn't come up and nothing worked on this instance. I guess if you have all the information you care about on this issue, i'll go ahead and restore since i've been down for quite a while at this point.

I get the whole "AI = bad" thing, but I spent about 2 hours troubleshooting with GPT 4o (the paid version) and made significant progress until we hit a wall that we couldn't get around, which at least gave me satisfaction that we tried everything we could.

Take care.

nebulon

I am generally not in the group of AI = bad, however it likely just lacks lots of knowledge about the Cloudron internals, which make recommendations often not very useful in this area (yet).

Overall I am still not sure based on this thread why DNS was involved here and given that you hit 3 different addons with apparently diverse issues, maybe something unrelated to them individually caused the trouble.

We will add a few more helpers to the cloudron-support script for the future, to hopefully get down to such issues faster.

If you haven't deleted the server yet, maybe we could take a closer look via SSH, since you already invested much time in debugging. If you want that, please enable remote ssh support and send a mail to support@cloudron.io with your Cloudron details.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

MySQL addon not starting

🧩 Cloudron 8.3.1 Upgrade – PostgreSQL & DNS Issues Summary

1. PostgreSQL Socket Failure

2. APT Repository DNS/TLS Failure

🧠 Context