Cloudron Forum

That makes sense, I hadn't looked at it that way. The page cache gives the memory back and shared_buffers doesn't, which matters a lot more on a box with thirty containers than on a dedicated database server — and the 25% guidance assumes the latter. What might still be worth it is effective_cache_size, which doesn't allocate anything and only tells the planner how much cache it can expect. On three of my servers with 8, 8 and 10 GB limits it reads 4GB everywhere, straight from boot_val — it doesn't track the limit at all. Deriving that one from the memory limit would improve plan choices without reserving any RAM. No rush on this one, it's much smaller than the MySQL case. It just seemed inconsistent that the buffer pool now follows the memory limit while the planner hint doesn't.

Excellent! Thank you very much @joseph (and also @fbartels)! I was able to alter the mysql dump file in the appsdata directory on the server and restarted the restore. The app started and I was able to fix all other database bugs in the dump directly in the apps database which then was up and running. Now the app has been restored on the new server and everything is working again. Thanks for the support and especially the note about the /home/yellowtent/appsdata/<appid>/mysqldump location.

Yes, it sorted itself out and I believe it may have been related to an aging SSD that was preforming very slowly. Thank you!

Created an issue. https://github.com/haraka/Haraka/issues/3516

Good catch. nc is not part of the base image it seems (not explicitly). I have replaced that code with bash logic.

@ghosted said in Update 9.0.11 Broke Services: It was THOUSANDS of lines, currently had a production instance, so needed it back up asap. This is fixed now . Was a bug in the polling in the UI.

@joseph Here are some excerpts from the log around the time I noticed the problem ("XXXXX" were manually added): 025-03-31T14:41:08.552Z box:server ========================================== 2025-03-31T14:41:08.553Z box:server Cloudron 8.3.1 2025-03-31T14:41:08.553Z box:server ========================================== 2025-03-31T14:41:08.554Z box:platform initialize: start platform 2025-03-31T14:41:08.889Z box:tasks stopAllTasks: stopping all tasks 2025-03-31T14:41:08.889Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2025-03-31T14:41:09.138Z box:shell All tasks stopped 2025-03-31T14:41:09.443Z box:locks releaseAll: all locks released 2025-03-31T14:41:09.486Z box:reverseproxy writeDashboardConfig: writing dashboard config for sebastienvigneau.xyz 2025-03-31T14:41:09.645Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/my.sebastienvigneau.xyz.cert -noout -ocsp_uri 2025-03-31T14:41:09.757Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx 2025-03-31T14:41:12.464Z box:platform onActivated: starting post activation services 2025-03-31T14:41:12.464Z box:platform startInfra: checking infrastructure 2025-03-31T14:41:12.569Z box:platform startInfra: updating infrastructure from 49.8.0 to 49.8.0 2025-03-31T14:41:12.570Z box:platform markApps: changedAddons: ["mysql","postgresql","mongodb","redis"] 2025-03-31T14:41:12.736Z box:services startServices: existing infra. incremental service create ["start","startTurn","startMysql","startPostgresql","startMongodb","startRedis","startGraphite","start"] 2025-03-31T14:41:12.736Z box:mailserver startMail: starting 2025-03-31T14:41:12.739Z box:mailserver restart: restarting mail container with mailFqdn:my.sebastienvigneau.xyz mailDomain:sebastienvigneau.xyz 2025-03-31T14:41:12.882Z box:locks write: current locks: {"mail_restart":null} 2025-03-31T14:41:12.882Z box:locks acquire: mail_restart 2025-03-31T14:41:13.100Z box:mailserver configureMail: stopping and deleting previous mail container 2025-03-31T14:42:42.033Z box:mailserver createMailConfig: generating mail config with my.sebastienvigneau.xyz 2025-03-31T14:42:42.068Z box:mailserver configureMail: starting mail container 2025-03-31T14:42:42.068Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail --net cloudron --net-alias mail --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mail -m 536870912 --memory-swap -1 --dns 172.18.0.1 --dns-search=. --ip 172.18.30.4 -e CLOUDRON_MAIL_TOKEN=XXXXX -e CLOUDRON_RELAY_TOKEN=XXXXX -e LOGLEVEL=info -v /home/yellowtent/boxdata/mail:/app/data -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995 --label isCloudronManaged=true --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536 2025-03-31T14:43:33.399Z box:locks write: current locks: {} 2025-03-31T14:43:33.399Z box:locks release: mail_restart 2025-03-31T14:43:33.399Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e 2025-03-31T14:43:33.405Z box:services startTurn: stopping and deleting previous turn container 2025-03-31T14:44:04.243Z box:services startTurn: starting turn container 2025-03-31T14:44:04.243Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn --hostname turn --net host --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=turn -m 268435456 --memory-swap -1 -e CLOUDRON_TURN_SECRET=XXXXX -e CLOUDRON_REALM=my.sebastienvigneau.xyz --label isCloudronManaged=true --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570 2025-03-31T14:44:15.397Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d 2025-03-31T14:44:15.400Z box:services startMysql: stopping and deleting previous mysql container 2025-03-31T14:44:29.585Z box:services startMysql: starting mysql container 2025-03-31T14:44:29.585Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mysql --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN=XXXXX -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=XXXXX -v /home/yellowtent/platformdata/mysql:/var/lib/mysql --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280 2025-03-31T14:44:36.436Z box:services Waiting for mysql 2025-03-31T14:44:36.442Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000 2025-03-31T14:44:51.455Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000 2025-03-31T14:45:06.732Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306 2025-03-31T14:45:22.063Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2 2025-03-31T14:45:26.965Z box:services startPostgresql: postgresql will be upgraded 2025-03-31T14:45:26.965Z box:services exportDatabase: Exporting postgresql 2025-03-31T14:45:26.972Z box:services exportDatabase: Exporting addon postgresql of app 1bd225ff-a0e5-4f80-866f-99bd40115530 2025-03-31T14:45:26.973Z box:services Backing up postgresql 2025-03-31T14:45:26.976Z box:services exportDatabase: Error exporting postgresql of app 1bd225ff-a0e5-4f80-866f-99bd40115530. BoxError: Could not pipe http://172.18.30.2:3000/databases/db1bd225ffa0e54f80866f99bd40115530/backup?access_token=XXXXX to /home/yellowtent/appsdata/1bd225ff-a0e5-4f80-866f-99bd40115530/postgresqldump: connect ECONNREFUSED 172.18.30.2:3000 at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1350:47) at ClientRequest.emit (node:events:519:28) at emitErrorEvent (node:_http_client:101:11) at Socket.socketErrorListener (node:_http_client:504:5) at Socket.emit (node:events:519:28) at emitErrorNT (node:internal/streams/destroy:169:8) at emitErrorCloseNT (node:internal/streams/destroy:128:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { reason: 'Network Error', details: {} } 2025-03-31T14:45:26.976Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Could not pipe http://172.18.30.2:3000/databases/db1bd225ffa0e54f80866f99bd40115530/backup?access_token=XXXXX to /home/yellowtent/appsdata/1bd225ff-a0e5-4f80-866f-99bd40115530/postgresqldump: connect ECONNREFUSED 172.18.30.2:3000 2025-03-31T14:45:26.976Z box:platform BoxError: Could not pipe http://172.18.30.2:3000/databases/db1bd225ffa0e54f80866f99bd40115530/backup?access_token=XXXXX to /home/yellowtent/appsdata/1bd225ff-a0e5-4f80-866f-99bd40115530/postgresqldump: connect ECONNREFUSED 172.18.30.2:3000 at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1350:47) at ClientRequest.emit (node:events:519:28) at emitErrorEvent (node:_http_client:101:11) at Socket.socketErrorListener (node:_http_client:504:5) at Socket.emit (node:events:519:28) at emitErrorNT (node:internal/streams/destroy:169:8) at emitErrorCloseNT (node:internal/streams/destroy:128:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) 2025-03-31T18:15:19.191Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name 2025-03-31T18:15:19.505Z box:shell services: grep -q avx /proc/cpuinfo 2025-03-31T18:15:19.608Z box:shell services: systemctl is-active nginx 2025-03-31T18:15:19.610Z box:shell services: systemctl is-active unbound 2025-03-31T18:16:37.596Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name 2025-03-31T18:16:40.619Z box:shell system /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reboot.sh 2025-03-31T18:16:40.968Z box:box Received SIGTERM. Shutting down. 2025-03-31T18:16:40.969Z box:platform uninitializing platform 2025-03-31T18:16:40.969Z box:shell system: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reboot.sh errored BoxError: system exited with code null signal SIGTERM at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) at ChildProcess.emit (node:events:519:28) at ChildProcess.emit (node:domain:488:12) at ChildProcess._handle.onexit (node:internal/child_process:294:12) { reason: 'Shell Error', details: {}, code: null, signal: 'SIGTERM' } 2025-03-31T18:16:40.970Z box:system reboot: could not reboot. BoxError: system exited with code null signal SIGTERM at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) at ChildProcess.emit (node:events:519:28) at ChildProcess.emit (node:domain:488:12) at ChildProcess._handle.onexit (node:internal/child_process:294:12) { reason: 'Shell Error', details: {}, code: null, signal: 'SIGTERM' } 2025-03-31T18:16:40.977Z box:platform onDeactivated: stopping post activation services 2025-03-31T18:16:40.977Z box:tasks stopAllTasks: stopping all tasks 2025-03-31T18:16:40.978Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2025-03-31T18:16:41.456Z box:shell All tasks stopped Anyway, I ended up running cloudron-support --recreate-docker, which made the apps functional again, and I am now restoring them from the backup one by one.

@d19dotca said in Shouldn't we get an alert when a service container fails / is unhealthy?: checking so that it ignores any entries of the current Docker networking ranges such as 172.18.xxx.xxx addresses For the moment, I have added validation to not accept such addresses . Ignoring this properly is more complicated (since we have to filter it at apply time) but atleast it won't let you save the blocklist easily .

If the system runs out of disk space, a lot of side-effects can happen, so this could very well be the root cause then.

A temporary workaround: docker stop turn . Some platform updates might turn back the turn service on though (when updating the turn docker image).

Thank you very much @girish, you rock!

Looks good and prevents issues like looking wrong in a short sight. Thanks!

@girish It's up now, and all the errors have disappeared.

Closed due to inactivity

@bcdodgeme this usually only happens when the server runs out of disk space.

Found an issue: MongoDB needed up 600Mb of RAM (to be adjusted in services page).

@fossy As mugero said, you really have no reason to worry about the services. They use a negligible amount of resources when they are not actually used!

Amazing! And so we all get a confidence boost with a simple timeout fix! Thanks @girish & everyone who contributed on this thread! Love you all