regular postgres crashes break some apps & backups until manual postgres restart

[Supaiku]

For some reason postgres keep crashing, and then mastodon and a few other apps, plus backups don't work until I go in an manually restart it.

It appears yellow in services.

backups error with:

Jan 15 03:05:54 box:tasks setCompleted - 6715: {"result":null,"error":{"stack":"BoxError: Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup?access_token=426047e4ce5009e88eff393b292fafb1e64fad4736a2d78661ffc48f61327cb03e5d8ba67bada47dad087675e7dbd293291e8d58a0efacff0197dc5941bd71f0898b0cebe584d0ce31e9c9571f558df61a775cfeda9f85a52d1d8775ad1a51bb5e7e8e5f6ecfa81088b2b677857fe2b3bafbb21d6a9b2c8b9c843ac10e15456d to /home/yellowtent/appsdata/04f262be-1a1a-40d1-beed-2fd1d95d0dfb/postgresqldump: status 500\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1283:60)\n at ClientRequest.emit (node:events:519:28)\n at ClientRequest.emit (node:domain:488:12)\n at HTTPParser.parserOnIncomingClient (node:_http_client:702:27)\n at HTTPParser.parserOnHeadersComplete (node:_http_common:118:17)\n at Socket.socketOnData (node:_http_client:544:22)\n at Socket.emit (node:events:519:28)\n at Socket.emit (node:domain:488:12)\n at addChunk (node:internal/streams/readable:559:12)\n at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)","name":"BoxError","reason":"Addons Error","details":{},"message":"Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup?access_token=426047e4ce5009e88eff393b292fafb1e64fad4736a2d78661ffc48f61327cb03e5d8ba67bada47dad087675e7dbd293291e8d58a0efacff0197dc5941bd71f0898b0cebe584d0ce31e9c9571f558df61a775cfeda9f85a52d1d8775ad1a51bb5e7e8e5f6ecfa81088b2b677857fe2b3bafbb21d6a9b2c8b9c843ac10e15456d to /home/yellowtent/appsdata/04f262be-1a1a-40d1-beed-2fd1d95d0dfb/postgresqldump: status 500"}}
Jan 15 03:05:54 box:tasks update 6715: {"percent":100,"result":null,"error":{"stack":"BoxError: Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup?access_token=426047e4ce5009e88eff393b292fafb1e64fad4736a2d78661ffc48f61327cb03e5d8ba67bada47dad087675e7dbd293291e8d58a0efacff0197dc5941bd71f0898b0cebe584d0ce31e9c9571f558df61a775cfeda9f85a52d1d8775ad1a51bb5e7e8e5f6ecfa81088b2b677857fe2b3bafbb21d6a9b2c8b9c843ac10e15456d to /home/yellowtent/appsdata/04f262be-1a1a-40d1-beed-2fd1d95d0dfb/postgresqldump: status 500\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1283:60)\n at ClientRequest.emit (node:events:519:28)\n at ClientRequest.emit (node:domain:488:12)\n at HTTPParser.parserOnIncomingClient (node:_http_client:702:27)\n at HTTPParser.parserOnHeadersComplete (node:_http_common:118:17)\n at Socket.socketOnData (node:_http_client:544:22)\n at Socket.emit (node:events:519:28)\n at Socket.emit (node:domain:488:12)\n at addChunk (node:internal/streams/readable:559:12)\n at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)","name":"BoxError","reason":"Addons Error","details":{},"message":"Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup?access_token=426047e4ce5009e88eff393b292fafb1e64fad4736a2d78661ffc48f61327cb03e5d8ba67bada47dad087675e7dbd293291e8d58a0efacff0197dc5941bd71f0898b0cebe584d0ce31e9c9571f558df61a775cfeda9f85a52d1d8775ad1a51bb5e7e8e5f6ecfa81088b2b677857fe2b3bafbb21d6a9b2c8b9c843ac10e15456d to /home/yellowtent/appsdata/04f262be-1a1a-40d1-beed-2fd1d95d0dfb/postgresqldump: status 500"}}
Jan 15 03:05:54 BoxError: Unexpected response code or HTTP error when piping http://172.18.30.2:3000/databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup?access_token=426047e4ce5009e88eff393b292fafb1e64fad4736a2d78661ffc48f61327cb03e5d8ba67bada47dad087675e7dbd293291e8d58a0efacff0197dc5941bd71f0898b0cebe584d0ce31e9c9571f558df61a775cfeda9f85a52d1d8775ad1a51bb5e7e8e5f6ecfa81088b2b677857fe2b3bafbb21d6a9b2c8b9c843ac10e15456d to /home/yellowtent/appsdata/04f262be-1a1a-40d1-beed-2fd1d95d0dfb/postgresqldump: status 500
Jan 15 03:05:54 at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1283:60)

error log in postgres says this (until restart):

backup: failed to backup database. Error: connect ECONNREFUSED ::1:5432
Jan 12 03:05:56 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
Jan 12 03:05:56 errno: -111,
Jan 12 03:05:56 code: 'ECONNREFUSED',
Jan 12 03:05:56 syscall: 'connect',
Jan 12 03:05:56 address: '::1',
Jan 12 03:05:56 port: 5432
Jan 12 03:05:56 }
Jan 13 03:03:18 [POST] /databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup
Jan 13 03:03:18 backup: failed to backup database. Error: connect ECONNREFUSED ::1:5432
Jan 13 03:03:18 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
Jan 13 03:03:18 errno: -111,
Jan 13 03:03:18 code: 'ECONNREFUSED',
Jan 13 03:03:18 syscall: 'connect',
Jan 13 03:03:18 address: '::1',
Jan 13 03:03:18 port: 5432
Jan 13 03:03:18 }
Jan 15 03:05:53 [POST] /databases/db04f262be1a1a40d1beed2fd1d95d0dfb/backup
Jan 15 03:05:54 backup: failed to backup database. Error: connect ECONNREFUSED ::1:5432
Jan 15 03:05:54 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
Jan 15 03:05:54 errno: -111,
Jan 15 03:05:54 code: 'ECONNREFUSED',
Jan 15 03:05:54 syscall: 'connect',
Jan 15 03:05:54 address: '::1',
Jan 15 03:05:54 port: 5432
Jan 15 03:05:54 }
Jan 17 13:23:35 [GET] /healthcheck
Jan 17 13:23:38 2025-01-17 18:23:38,590 WARN received SIGTERM indicating exit request
Jan 17 13:23:38 2025-01-17 18:23:38,602 INFO waiting for postgresql-service to die
Jan 17 13:23:39 2025-01-17 18:23:39,621 INFO stopped: postgresql-service (terminated by SIGTERM)```


What's up with this and how can i better troubleshoot address this?

[joseph]

Have you tried bumping the memory limit already? Services -> Postgres -> Edit -> Increase memory limit to something quite high

[Supaiku]

@joseph yes, it was 2GB plus - I haven't managed to catch it maxing out of memory since then though it's possible

[joseph]

@Supaiku the dbs take more memory during backup/restore , so it's hard to catch it at that exact point.

[Supaiku]

I did just catch it at 70%...
My MYSQL memory limit is set at 4.125GB - I bumped postgresql up to 5GB - is that likely to be enough or should I go higher?
The box has 32GB of memory and a big swap so I think it can handle pretty high memory usage.

[Supaiku]

it seems like maybe the issues is my device occasionally running out of space crashing postgres - does that check out?

I'll try moving my docker images to another drive to preven that issue as space use seems to fluctuate drastically...

[nebulon]

If the system runs out of disk space, a lot of side-effects can happen, so this could very well be the root cause then.