How to stop apps via SSH command line

[nebulon]

Maybe check the logs at /home/yellowtent/platformdata/logs/box.log for more info to work with.

[bmann]

Looking at postgres logs, it has repeated FATAL: the database system is starting up, as I assume that apps that depend on it try to connect to it.

Hmmm... and now this?

CREATE EXTENSION
Nov 17 23:20:39 2024-11-18 07:20:39.460 UTC [1066] root@cloudronpgvectorupdate WARNING: pgvecto.rs is upgraded. Restart PostgreSQL to take effects.
Nov 17 23:20:56 DROP DATABASE
Nov 17 23:20:56 ==> Stopping postgresql
Nov 17 23:20:58 waiting for server to shut down

box.log has healthcheck and a couple of these for different containers:

box:scheduler BoxError: (HTTP code 409) unexpected - Conflict. The container name "/e16f7c6f-861f-427b-96a0-99dc7744e481-wpcron" is already in use by container "e710bd8be1653ba1231ea6c8cc5ea7406ce39cb7799c1e2b07a7cd766942a5c9". You have to remove (or rename) that container to be able to reuse that name.

[bmann]

I increased memory to Postgres and MySQL and restarted the VPS. Graphite, mail, mongodb, mysql, and Postgres are all yellow / listed as starting.

Just looking at the "top" command, I've got massive load, 47.07 / 51.62 / 36.56 (this is down from twice that).

I don't have massive traffic coming into the box.

AFAICT this is from self-updating to v8.1 Cloudron updated to v8.1.0 11/15/2024 -- it's the only thing.

Is there a way to roll back Cloudron?

[nebulon]

Indeed looks like the postgres service does not come up, which is then blocking the platform startup to succeed. Haven't seen those postgres logs like this so far. We had some hiccups long ago with that postgres extension. Do you have more logs if you manually restart postgres?

[joseph]

@bmann said in How to stop apps via SSH command line:

Is there a way to roll back Cloudron?

you have to do this using backup. https://docs.cloudron.io/backups/#restore-cloudron

[bmann]

@nebulon I haven’t had it successfully start. Just repeated FATAL lines as above.

I’ve restarted the VPS and it’s all under heavy load.

What’s next here? I’m stuck.

[nebulon]

Unless you want to restore the Cloudron itself, I guess we have to get down to this postgres issue. Maybe you can try to reset the postgres addon then following the steps at https://docs.cloudron.io/troubleshooting/#corrupt-addon

If that doesn't work we have to maybe get access to the server to see why it refused to start without further error messages.

[bmann]

I used docker stop to turn off all the apps to get load down. Now load average: 1.59, 5.06, 3.37

That corrupt addon doesn't have any information in it on what to do, unless you mean "Now, restore each app that uses the addon in the Cloudron dashboard."

I used docker stop to stop the PG container, then went ahead and did that for mongo and mysql and mail which were all yellow.

All redis instances also yellow. OK, let's stop them, too.

All containers are off other than core services and three surfer instances. Those all work, although slow to load. Load dipped down to 0.65, yay!

Ok, now I turn back on Postgres, it launches succesfully. I forget that MySQL is what I need for my Ghost website, so I launch that. Both of them turn back on (pressed restart from red in the Services screen in web dashboard).

I then go to Repair > Restart App for the site I'm trying to get up. It launches!

So here's where box logs are at: box:apphealthmonitor app health: 4 running / 16 stopped / 11 unresponsive

We're still at oad average: 1.20, 1.30, 1.93. That's with just one app running, where before 8.1 everything was running fine. I haven't turned back on mail or mongodb or redis yet.

Any ideas here on what is causing this load? What is the recommended path? Is there an 8.1 update soon that might improve things?

[girish]

Would be good to know what process is consuming the load. Is this obvious via top/htop ? 8.1 doesn't fix anything specific to this issue.

[bmann]

Node 15-30%, dockerd 40-80%, mysql 6 - 8% (Two ghost apps which use mysql are the only ones I've succesfully restarted). Docker shows up periodically at 20% and then drops off again.

My reference to 8.1 was a hope for a 8.1.x that magically fixes things :

This is what my CPU usage shows from the Hetzner control panel:

My logs tell me that Cloudron updated to 8.1 on 11/15, so at this point I assume that's the culprit.

Any other ideas?

[girish]

Node was updated to 20.18.0 . But looks like dockerd takes bulk of the CPU? AFAIK, dockerd is just the management daemon, I wonder why it's taking so much CPU.

[nebulon]

Are any containers constantly restarting causing maybe such a load on docker side? Is journalctl -u docker -f revealing anything?

[bmann]

CPU is back down to 5% for the last day with just a couple of Ghost websites running. Restarting mail, mongodb and slowly turning on each of the apps. Crossing my fingers that this works.