Cloudron Forum

@mathieuepitech Great analysis! @staff will be very interested about this for further improvement to circumvent such scenarios.

@joseph Here are some excerpts from the log around the time I noticed the problem ("XXXXX" were manually added): 025-03-31T14:41:08.552Z box:server ========================================== 2025-03-31T14:41:08.553Z box:server Cloudron 8.3.1 2025-03-31T14:41:08.553Z box:server ========================================== 2025-03-31T14:41:08.554Z box:platform initialize: start platform 2025-03-31T14:41:08.889Z box:tasks stopAllTasks: stopping all tasks 2025-03-31T14:41:08.889Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2025-03-31T14:41:09.138Z box:shell All tasks stopped 2025-03-31T14:41:09.443Z box:locks releaseAll: all locks released 2025-03-31T14:41:09.486Z box:reverseproxy writeDashboardConfig: writing dashboard config for sebastienvigneau.xyz 2025-03-31T14:41:09.645Z box:shell reverseproxy: openssl x509 -in /home/yellowtent/platformdata/nginx/cert/my.sebastienvigneau.xyz.cert -noout -ocsp_uri 2025-03-31T14:41:09.757Z box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx 2025-03-31T14:41:12.464Z box:platform onActivated: starting post activation services 2025-03-31T14:41:12.464Z box:platform startInfra: checking infrastructure 2025-03-31T14:41:12.569Z box:platform startInfra: updating infrastructure from 49.8.0 to 49.8.0 2025-03-31T14:41:12.570Z box:platform markApps: changedAddons: ["mysql","postgresql","mongodb","redis"] 2025-03-31T14:41:12.736Z box:services startServices: existing infra. incremental service create ["start","startTurn","startMysql","startPostgresql","startMongodb","startRedis","startGraphite","start"] 2025-03-31T14:41:12.736Z box:mailserver startMail: starting 2025-03-31T14:41:12.739Z box:mailserver restart: restarting mail container with mailFqdn:my.sebastienvigneau.xyz mailDomain:sebastienvigneau.xyz 2025-03-31T14:41:12.882Z box:locks write: current locks: {"mail_restart":null} 2025-03-31T14:41:12.882Z box:locks acquire: mail_restart 2025-03-31T14:41:13.100Z box:mailserver configureMail: stopping and deleting previous mail container 2025-03-31T14:42:42.033Z box:mailserver createMailConfig: generating mail config with my.sebastienvigneau.xyz 2025-03-31T14:42:42.068Z box:mailserver configureMail: starting mail container 2025-03-31T14:42:42.068Z box:shell mailserver: /bin/bash -c docker run --restart=always -d --name=mail --net cloudron --net-alias mail --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mail -m 536870912 --memory-swap -1 --dns 172.18.0.1 --dns-search=. --ip 172.18.30.4 -e CLOUDRON_MAIL_TOKEN=XXXXX -e CLOUDRON_RELAY_TOKEN=XXXXX -e LOGLEVEL=info -v /home/yellowtent/boxdata/mail:/app/data -v /home/yellowtent/platformdata/addons/mail:/etc/mail:ro -p 587:2587 -p 993:9993 -p 4190:4190 -p 25:2587 -p 465:2465 -p 995:9995 --label isCloudronManaged=true --read-only -v /run -v /tmp registry.docker.com/cloudron/mail:3.15.0@sha256:c93b5a83fc4e775bda4e05010bd19e5a658936e7a09cf7e51281e3696fde4536 2025-03-31T14:43:33.399Z box:locks write: current locks: {} 2025-03-31T14:43:33.399Z box:locks release: mail_restart 2025-03-31T14:43:33.399Z box:docker deleteImage: removing registry.docker.com/cloudron/mail:3.14.9@sha256:c51a2ee20b2087e208084a9115cc6a525e3b3b227b52d8cfdac2d98a6409672e 2025-03-31T14:43:33.405Z box:services startTurn: stopping and deleting previous turn container 2025-03-31T14:44:04.243Z box:services startTurn: starting turn container 2025-03-31T14:44:04.243Z box:shell services: /bin/bash -c docker run --restart=always -d --name=turn --hostname turn --net host --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=turn -m 268435456 --memory-swap -1 -e CLOUDRON_TURN_SECRET=XXXXX -e CLOUDRON_REALM=my.sebastienvigneau.xyz --label isCloudronManaged=true --read-only -v /tmp -v /run registry.docker.com/cloudron/turn:1.8.0@sha256:cdbe83c3c83b8f25de3a5814b121eb941b457dca7127d2e6ff446c7a0cfa1570 2025-03-31T14:44:15.397Z box:docker deleteImage: removing registry.docker.com/cloudron/turn:1.7.2@sha256:9ed8da613c1edc5cb8700657cf6e49f0f285b446222a8f459f80919945352f6d 2025-03-31T14:44:15.400Z box:services startMysql: stopping and deleting previous mysql container 2025-03-31T14:44:29.585Z box:services startMysql: starting mysql container 2025-03-31T14:44:29.585Z box:shell services: /bin/bash -c docker run --restart=always -d --name=mysql --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mysql --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN=XXXXX -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=XXXXX -v /home/yellowtent/platformdata/mysql:/var/lib/mysql --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.5.0@sha256:969ea5b2f91861940ca6309c7676c52e479d2a864ba3aabd08a4266799707280 2025-03-31T14:44:36.436Z box:services Waiting for mysql 2025-03-31T14:44:36.442Z box:services Attempt 1 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000 2025-03-31T14:44:51.455Z box:services Attempt 2 failed. Will retry: Network error waiting for mysql: connect ECONNREFUSED 172.18.30.1:3000 2025-03-31T14:45:06.732Z box:services Attempt 3 failed. Will retry: Error waiting for mysql. Status code: 200 message: connect ECONNREFUSED 127.0.0.1:3306 2025-03-31T14:45:22.063Z box:docker deleteImage: removing registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2 2025-03-31T14:45:26.965Z box:services startPostgresql: postgresql will be upgraded 2025-03-31T14:45:26.965Z box:services exportDatabase: Exporting postgresql 2025-03-31T14:45:26.972Z box:services exportDatabase: Exporting addon postgresql of app 1bd225ff-a0e5-4f80-866f-99bd40115530 2025-03-31T14:45:26.973Z box:services Backing up postgresql 2025-03-31T14:45:26.976Z box:services exportDatabase: Error exporting postgresql of app 1bd225ff-a0e5-4f80-866f-99bd40115530. BoxError: Could not pipe http://172.18.30.2:3000/databases/db1bd225ffa0e54f80866f99bd40115530/backup?access_token=XXXXX to /home/yellowtent/appsdata/1bd225ff-a0e5-4f80-866f-99bd40115530/postgresqldump: connect ECONNREFUSED 172.18.30.2:3000 at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1350:47) at ClientRequest.emit (node:events:519:28) at emitErrorEvent (node:_http_client:101:11) at Socket.socketErrorListener (node:_http_client:504:5) at Socket.emit (node:events:519:28) at emitErrorNT (node:internal/streams/destroy:169:8) at emitErrorCloseNT (node:internal/streams/destroy:128:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { reason: 'Network Error', details: {} } 2025-03-31T14:45:26.976Z box:platform startInfra: Failed to start services. retry=false (attempt 0): Could not pipe http://172.18.30.2:3000/databases/db1bd225ffa0e54f80866f99bd40115530/backup?access_token=XXXXX to /home/yellowtent/appsdata/1bd225ff-a0e5-4f80-866f-99bd40115530/postgresqldump: connect ECONNREFUSED 172.18.30.2:3000 2025-03-31T14:45:26.976Z box:platform BoxError: Could not pipe http://172.18.30.2:3000/databases/db1bd225ffa0e54f80866f99bd40115530/backup?access_token=XXXXX to /home/yellowtent/appsdata/1bd225ff-a0e5-4f80-866f-99bd40115530/postgresqldump: connect ECONNREFUSED 172.18.30.2:3000 at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1350:47) at ClientRequest.emit (node:events:519:28) at emitErrorEvent (node:_http_client:101:11) at Socket.socketErrorListener (node:_http_client:504:5) at Socket.emit (node:events:519:28) at emitErrorNT (node:internal/streams/destroy:169:8) at emitErrorCloseNT (node:internal/streams/destroy:128:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) 2025-03-31T18:15:19.191Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name 2025-03-31T18:15:19.505Z box:shell services: grep -q avx /proc/cpuinfo 2025-03-31T18:15:19.608Z box:shell services: systemctl is-active nginx 2025-03-31T18:15:19.610Z box:shell services: systemctl is-active unbound 2025-03-31T18:16:37.596Z box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name 2025-03-31T18:16:40.619Z box:shell system /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reboot.sh 2025-03-31T18:16:40.968Z box:box Received SIGTERM. Shutting down. 2025-03-31T18:16:40.969Z box:platform uninitializing platform 2025-03-31T18:16:40.969Z box:shell system: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reboot.sh errored BoxError: system exited with code null signal SIGTERM at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) at ChildProcess.emit (node:events:519:28) at ChildProcess.emit (node:domain:488:12) at ChildProcess._handle.onexit (node:internal/child_process:294:12) { reason: 'Shell Error', details: {}, code: null, signal: 'SIGTERM' } 2025-03-31T18:16:40.970Z box:system reboot: could not reboot. BoxError: system exited with code null signal SIGTERM at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) at ChildProcess.emit (node:events:519:28) at ChildProcess.emit (node:domain:488:12) at ChildProcess._handle.onexit (node:internal/child_process:294:12) { reason: 'Shell Error', details: {}, code: null, signal: 'SIGTERM' } 2025-03-31T18:16:40.977Z box:platform onDeactivated: stopping post activation services 2025-03-31T18:16:40.977Z box:tasks stopAllTasks: stopping all tasks 2025-03-31T18:16:40.978Z box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2025-03-31T18:16:41.456Z box:shell All tasks stopped Anyway, I ended up running cloudron-support --recreate-docker, which made the apps functional again, and I am now restoring them from the backup one by one.

@vadim said in Running a separate PostgreSQL instance alongside Cloudron: Is there any way to use pgvector via any cloudron app? The answer is a simple search for "pgvector" in the forum. It yields: https://forum.cloudron.io/topic/13139/consider-adding-support-for-pgvector-pgvectorscale-for-pgsql

@timconsidine You are so right. The best day for Team Cloudron is one in which there are no updates. I give @girish @nebulon @joseph an enormous amount of credit for the job they do. As someone who rolls out ~ monthly updates to users, every time that happens I pray that me and my team didn't miss something important. But it happens to all of us, despite good processes and best of intentions. But 8.3 will be replaced shortly with 8.3.1 and then 8.4 and then 9.0. And this difficult day will be replaced in everyone's memory by the great things that are yet to come!

I could not let go if it, so I kept digging a bit. I had previously reset the memory limit back to its default, but bumped it to 6gb now for another test. This time the restore of Nextcloud went through. So now everything is operational again (and I also cleaned out some apps that had errors, but I was not using anyways anymore). To get back to a working state I needed to restore each up using a postgres database to the state from before the upgrade to 8.3.0. Else the app would have no data in their databases (Mastodon users and toots gone, Matrix rooms gone, for example).

Thanks, I have put this info in https://docs.cloudron.io/apps/baserow/#log-retention

@luckow give postgresql more memory

The issue is now resolved. The problem was with the GitLab database, where we found 160 GB of logs. The database dump was failing because there wasn’t enough free disk space. We have purged the database, and backups are now successful. Thank you for your support.

@bennyv53 I recommend using ssh then (SSL support for internal postgres will break Cloudron's setup). Is the concern security related?

Turns out that Cloudron hit the same issue as https://forum.cloudron.io/topic/11276/cloudron-update-to-7-7-0-postgresql-does-not-start?_=1723244919333 and the fixed worked there also.

@girish perfect, thanks - appreciate it! I'm almost certain there could be ways to improve those queries, but it's always better to have a failsafe in place

The current addon uses 14.9 so I guess you could run the corresponding image from https://hub.docker.com/_/postgres

@girish thank you, I tried it several times and it didn't worked. I tried it a last time and it worked. I don't know why but thank you

This was a bug in 7.4.0 and is now fixed in 7.4.1. We updated node to v18 (from v16). There was a subtle but impact change hidden... In node 16, the requestTimeout was 0. This meant unlimited time. In node 18, the requestTImeout is now 5 mins. This meant that large restores were failing if the upload didn't finish within 5 mins.

@roofboard said in Chat woot cannot uninstall after breaking the backup system.: Mar 30 20:19:30 box:services Backing up postgresql Mar 30 20:29:30 box:services pipeRequestToFile: timeout - connect or post-connect idle timeout It seems somehow postgresql was not having a happy time. Maybe give it a bit more memory (Services -> PostgreSQL -> increase memory limit). The reboot probably just freed up some memory.

I have given these stable IPs now - https://git.cloudron.io/cloudron/box/-/commit/582994b9d646989b42e27dde1ae0c27bbb964834

@girish It's up now, and all the errors have disappeared.

Thanks, it works.

@girish many thanks !

Not sure why that slipped through. Is this sorted by now? We had a case where slow I/O caused the issue resulting in a timeout. Could this have been the case? https://forum.cloudron.io/topic/4780/postgresql-database-won-t-start/8