How to stop apps via SSH command line

[bmann]

Hmm. Feels like it would be useful to just have the cloudron CLI stuff run locally on the command line, too.

I've gotten into the dashboard and mail, postgres, mongo, and mysql are all listed as "starting". Well, this isn't good.

[nebulon]

Maybe check the logs at /home/yellowtent/platformdata/logs/box.log for more info to work with.

[bmann]

Looking at postgres logs, it has repeated FATAL: the database system is starting up, as I assume that apps that depend on it try to connect to it.

Hmmm... and now this?

CREATE EXTENSION
Nov 17 23:20:39 2024-11-18 07:20:39.460 UTC [1066] root@cloudronpgvectorupdate WARNING: pgvecto.rs is upgraded. Restart PostgreSQL to take effects.
Nov 17 23:20:56 DROP DATABASE
Nov 17 23:20:56 ==> Stopping postgresql
Nov 17 23:20:58 waiting for server to shut down

box.log has healthcheck and a couple of these for different containers:

box:scheduler BoxError: (HTTP code 409) unexpected - Conflict. The container name "/e16f7c6f-861f-427b-96a0-99dc7744e481-wpcron" is already in use by container "e710bd8be1653ba1231ea6c8cc5ea7406ce39cb7799c1e2b07a7cd766942a5c9". You have to remove (or rename) that container to be able to reuse that name.

[bmann]

I increased memory to Postgres and MySQL and restarted the VPS. Graphite, mail, mongodb, mysql, and Postgres are all yellow / listed as starting.

Just looking at the "top" command, I've got massive load, 47.07 / 51.62 / 36.56 (this is down from twice that).

I don't have massive traffic coming into the box.

AFAICT this is from self-updating to v8.1 Cloudron updated to v8.1.0 11/15/2024 -- it's the only thing.

Is there a way to roll back Cloudron?

[nebulon]

Indeed looks like the postgres service does not come up, which is then blocking the platform startup to succeed. Haven't seen those postgres logs like this so far. We had some hiccups long ago with that postgres extension. Do you have more logs if you manually restart postgres?

[joseph]

@bmann said in How to stop apps via SSH command line:

Is there a way to roll back Cloudron?

you have to do this using backup. https://docs.cloudron.io/backups/#restore-cloudron

[bmann]

@nebulon I haven’t had it successfully start. Just repeated FATAL lines as above.

I’ve restarted the VPS and it’s all under heavy load.

What’s next here? I’m stuck.

[nebulon]

Unless you want to restore the Cloudron itself, I guess we have to get down to this postgres issue. Maybe you can try to reset the postgres addon then following the steps at https://docs.cloudron.io/troubleshooting/#corrupt-addon

If that doesn't work we have to maybe get access to the server to see why it refused to start without further error messages.

[bmann]

I used docker stop to turn off all the apps to get load down. Now load average: 1.59, 5.06, 3.37

That corrupt addon doesn't have any information in it on what to do, unless you mean "Now, restore each app that uses the addon in the Cloudron dashboard."

I used docker stop to stop the PG container, then went ahead and did that for mongo and mysql and mail which were all yellow.

All redis instances also yellow. OK, let's stop them, too.

All containers are off other than core services and three surfer instances. Those all work, although slow to load. Load dipped down to 0.65, yay!

Ok, now I turn back on Postgres, it launches succesfully. I forget that MySQL is what I need for my Ghost website, so I launch that. Both of them turn back on (pressed restart from red in the Services screen in web dashboard).

I then go to Repair > Restart App for the site I'm trying to get up. It launches!

So here's where box logs are at: box:apphealthmonitor app health: 4 running / 16 stopped / 11 unresponsive

We're still at oad average: 1.20, 1.30, 1.93. That's with just one app running, where before 8.1 everything was running fine. I haven't turned back on mail or mongodb or redis yet.

Any ideas here on what is causing this load? What is the recommended path? Is there an 8.1 update soon that might improve things?

[girish]

Would be good to know what process is consuming the load. Is this obvious via top/htop ? 8.1 doesn't fix anything specific to this issue.

[bmann]

Node 15-30%, dockerd 40-80%, mysql 6 - 8% (Two ghost apps which use mysql are the only ones I've succesfully restarted). Docker shows up periodically at 20% and then drops off again.

My reference to 8.1 was a hope for a 8.1.x that magically fixes things :

This is what my CPU usage shows from the Hetzner control panel:

My logs tell me that Cloudron updated to 8.1 on 11/15, so at this point I assume that's the culprit.

Any other ideas?

[girish]

Node was updated to 20.18.0 . But looks like dockerd takes bulk of the CPU? AFAIK, dockerd is just the management daemon, I wonder why it's taking so much CPU.

[nebulon]

Are any containers constantly restarting causing maybe such a load on docker side? Is journalctl -u docker -f revealing anything?

[bmann]

CPU is back down to 5% for the last day with just a couple of Ghost websites running. Restarting mail, mongodb and slowly turning on each of the apps. Crossing my fingers that this works.

[bmann]

Happened again. Not sure what caused it, 3 days ago, had to manually kill everything with docker stop, then slowly bring things back up. Postgres definitely down. Brought things up again slowly and everything was fine.

This morning, things started dying again. Had to kill some things with docker stop, then managed to hit stop on linkding, mastodon, and nextcloud which seemed to be culprits.

Rebooted, database services struggling to come up fully. Various apps that rely on databases enter not responding.

Last week, capturing spikes and then bringing it down and it's fine, and then this morning everything gets pinned to 100%

Zoom in of load last 24 hours. I'm in there killing things again and it's just starting to shed load:

Snapshot of top

[nebulon]

Can you identify by stopping one app after the other which one is causing this?

[bmann]

@nebulon working on doing that. But I like have a handful of Surfer sites and one Ghost site that really needs to stay up. If Ghost or Surfer is the issue then I'm really in trouble.

Mail / Mongo / Postgres / MySQL are all at "starting" / showing as yellow...without any errors in their logs.

Box is responsive enough to work with a bit.

Hostinger CPU usage stats show stuff still pretty maxed out.

top at load average: 7.16, 9.80, 7.10

yellowtent seems to be spiking node processes over 100% CPU usage.

[bmann]

Well, apparently the only way to "fix" this, is to docker stop the yellow services Mail / Mongo / Postgres / MySQL and then restart them in the dashboard and they start up succesfully, and then you can start other apps that have been stopped.