Postgresql not starting with "Resetting root password" infinite loop

[adrw]

My Postgresql seems to have an infinite loop and won't start, any ideas?

2022-01-24T20:21:05.000Z Resetting root password
2022-01-24T20:22:43.000Z Detected existing installation
2022-01-24T20:22:43.000Z Resetting root password
2022-01-24T20:24:22.000Z Detected existing installation
2022-01-24T20:24:22.000Z Resetting root password
2022-01-24T20:26:03.000Z Detected existing installation

[girish]

@adrw It seems postgres is not starting up. Do you have enough disk space? If you do docker restart postgresql , it should show any startup errors in the logs.

[adrw]

@girish thanks! disk space was fine but it seems like the memory was underprovisioned, I increased from default to 2GB and it started booting again, 1GB seemingly wasn't enough.

[adrw]

@girish I'm seeing this failure again, the disk usage in System tab shows 250GB free

[adrw]

@adrw It seem to again have required more memory, wondering if there'd be a way to add that to the logs or detect that reset root password loop to suggest increasing memory. I also encountered it with the SFTP service.

[nebulon]

@adrw did you get notifications about postgres running out of memory in your dashboard? Also the starting worked now after increasing the limit?

[girish]

Also, if there are no OOM messages in the notifications, can you also please check if dmesg output on the server has some OOM errors? Maybe we are not detecting OOM correctly.

[robi]

@girish There are at least two ways I can think of to get a warning that the container is running out of memory limit and is about to be OOM killed.

1. use --memory-reservation in containers set at 90% of --memory (limit) setting.

2. track high watermarks for app usage in kernel or graph data and raise warnings of imminent OOM kill possibility.

Perhaps add another graph somewhere where these are plotted visually and one can quick click to see which and bump up the memory config by 20%.

[adrw]

@nebulon I didn't get any notifications in the dashboard for OOM which is why debugging was fairly unintuitive.

@girish If I encounter it again, I'll check dmesg and let you know.