Docker and Postgres refuse to start

[timconsidine]

Bunch of apps showing as not responding.
Services screen shows Docker with red dot, and Postgres as orange with the restart symbol stuck circling.
Have rebooted the box.
Restarting Docker, status goes temporarily to orange then returns to red.
Any advice ?
Don't want to fiddle randomly on the box by SSH.
Logging in by SSH (and doing doing nothing) shows 43 updates pending, but no restart required message.
docker ps -a at terminal hangs.

EDIT :
postgres service logs show stuck in a loop

Nov 01 09:48:19 Detected existing installation
Nov 01 09:48:19 Resetting root password

[timconsidine]

All apps back up except for minio
EDIT : rolled minio back to v2.9.13 and working now.
Services now show green dot for docker and postgres

2nd time postgres has failed recently.
Running 7.2.5 and ubuntu 20.04

Worrying

Just noticed that docker ps -a shows a number of exited containers, but Cloudron portal only shows 1 app not responding. Is this normal ?

[girish]

@timconsidine the minio thing is a bug in their latest release. It seems they removed the fs backend and there is no migration forward See https://forum.cloudron.io/topic/7880/latest-update-seems-to-have-broken-my-minio

[girish]

@timconsidine said in Docker and Postgres refuse to start:

Running 7.2.5 and ubuntu 20.04

What caused this whole issue ? Just random?

[timconsidine]

@girish good question !
It seems random ... except 2nd time that postgres has failed.
I'll keep an eye on and monitor, trying to be more aware of different changes.
I'm running Cloudron on a Hetzner box, so I have reasonable confidence in the VPS.
Will keep an eye out and report any observations.
For now just carry on enjoying Cloudron !

[timconsidine]

@girish so it has happened again
Not aware of any cause - no intervention or fiddling done.
I just rebooted the box when prompted in the dashboard
Although I suspect it was failing before this
postgres service does not start
Cloudron dashboard for services shows orange dot
Logs show :

Nov 20 09:15:35 Resetting root password
Nov 20 09:15:57 Detected existing installation
Nov 20 09:15:59 Debug mode. Sleeping
Nov 20 09:16:16 Debug mode. Sleeping
Nov 20 09:16:54 Debug mode. Sleeping
Nov 20 09:19:24 Debug mode. Sleeping
Nov 20 09:28:39 Debug mode. Sleeping
Nov 20 09:30:36 Debug mode. Sleeping

Tried rebooting the box
As postgres is such a core service, I am reluctant to fiddle.
What would be the recommended steps to solve this ?

EDIT : I did try to start piostgres via command line :

root@Ubuntu-2004-focal-64-minimal ~ # sudo service postgresql start
Failed to start postgresql.service: Unit postgresql.service not found.

[nebulon]

@timconsidine from the logs, do you have recovery mode simply enabled for your postgres service? Click on the edit icon for postgres in the services view in your Cloudron dashboard to see that checkbox.

[timconsidine]

@nebulon I tried increasing the memory (no change)
I tried toggling the enable recovery but also no change
EDIT : recovery mode currently unticked, does not start

[girish]

@timconsidine said in Docker and Postgres refuse to start:

EDIT : recovery mode currently unticked, does not start

Do you see the same logs as before? Debug mode. Sleeping is printed when the service is in recovery mode.

[timconsidine]

@girish ah thanks
Now I just have the first 2 lines (resetting root password / existing instal).

[timconsidine]

@girish interestingly (for me!) there is no postgres service file

root@Ubuntu-2004-focal-64-minimal ~ # systemctl status postgresql
Unit postgresql.service could not be found.

Is this normal ? Or the cause of the issue ?

[girish]

@timconsidine postgresql is in a docker container. docker inspect postgresql for example should work. I am not sure why it's stuck, can you drop a note to support@cloudron.io ?

[timconsidine]

@girish ah good to know ref docker
will contact support@

[girish]

The issue was that pg_ctl has a default timeout of 60 seconds. On your server, it's taking a bit more than 60 seconds to start. So, I have bumped this value up and it's running now.

[timconsidine]

@girish many thanks !