Cloudron Forum

Hey all, thanks for recent notes. Still a good idea, probably need to find a developer who can do the packaging first.

There is an option to get email notification when apps go down.

I’ve been having degraded (yellow / restarting) core services - Postgres, MySQL, etc

Is there a way to get email notifications when those core services go down?

And — anyone have documentation on using Grafana or Prometheus to show stats for the Cloudron box where they are installled that might do something similar?

Well, apparently the only way to "fix" this, is to docker stop the yellow services Mail / Mongo / Postgres / MySQL and then restart them in the dashboard and they start up succesfully, and then you can start other apps that have been stopped.

@nebulon working on doing that. But I like have a handful of Surfer sites and one Ghost site that really needs to stay up. If Ghost or Surfer is the issue then I'm really in trouble.

Mail / Mongo / Postgres / MySQL are all at "starting" / showing as yellow...without any errors in their logs.

Box is responsive enough to work with a bit.

Hostinger CPU usage stats show stuff still pretty maxed out.

top at load average: 7.16, 9.80, 7.10

yellowtent seems to be spiking node processes over 100% CPU usage.

Happened again. Not sure what caused it, 3 days ago, had to manually kill everything with docker stop, then slowly bring things back up. Postgres definitely down. Brought things up again slowly and everything was fine.

This morning, things started dying again. Had to kill some things with docker stop, then managed to hit stop on linkding, mastodon, and nextcloud which seemed to be culprits.

Rebooted, database services struggling to come up fully. Various apps that rely on databases enter not responding.

Last week, capturing spikes and then bringing it down and it's fine, and then this morning everything gets pinned to 100%

Zoom in of load last 24 hours. I'm in there killing things again and it's just starting to shed load:

Snapshot of top

Hmm. Found another domain email.uscc.net

Thanks! yeah, that's JSON validation. Applied, appreciate the feedback.

@nebulon is that /app/data/config.production.json or somewhere else?

OK, I finally worked on this AGAIN and remembered how messed up this is conceptually.

username: groupname@domain
password: (any one of the passwords of the people that have access)

This is also makes Freescout work for me https://forum.cloudron.io/topic/12976/connection-could-not-be-established-with-host-mail-domain-com-connection-timed-out-110/43

A single "username" having multiple passwords is what messes me up every time.

@jdaviescoates it does not. Is +apple a setup thing in Cloudron? I've never heard of it.

So, I'm getting back to this, and logging into third party clients (notably Mail.app on MacOS), currently does not work.

email: groupname@otherdomain
user: boris@domain
password:

Mail app says "account already exists", presumably because I'm using the same user / pass.

I can put whatever I like under email address. Changing username to organizers@dwebyvr.org doesn't work.

(works fine in Snappy mail)

@girish

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = E6
verify return:1
depth=0 CN = *.commonscomputer.com
verify return:1
---
Certificate chain
 0 s:CN = *.commonscomputer.com
   i:C = US, O = Let's Encrypt, CN = E6
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Dec  9 07:12:07 2024 GMT; NotAfter: Mar  9 07:12:06 2025 GMT
 1 s:C = US, O = Let's Encrypt, CN = E6
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate

(several more screenfuls of certs and things)

read R BLOCK
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot (Ubuntu) ready.

@girish said in Connection could not be established with host mail.domain.com [Connection timed out #110]:

openssl s_client -connect <mailsever>:993 -crlf

Since I'm using the localhost settings that HD said, I used this (note host name and alt port):

openssl s_client -connect mail:9393 -crlf

Got this response:

CONNECTED(00000003)
40A74365227F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:354:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 293 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

So what's the process here? I guess I can toggle the "enable signups" feature, get people to sign up, and then toggle it off again?

I am on 8.2.3 on a Hostinger VPS and also having trouble fetching.

This is an old FS that I had stopped and just now turned on again. Previous settings worked (group shared mailbox) -- in that it fetched the IMAP folders and said the connection was working, and the email arrives in snappymail. Tried humptydumpty settings, also "worked", still not fetching.

Sending works fine from FS, from Snappy, etc.

@shanelord01 thanks for bringing this up Shane. I would like to see it packaged, it will probably need to start as a custom app.

I installed one myself taking over a full KVM-1 VPS at Hostinger ($4/month).

I'm an AT Protocol expert and maintain a tech talks Ghost blog at https://atprotocol.dev (hosted on my Cloudron server!).

There are some misunderstandings in this thread about architecture and frankly I'm not going to argue about decentralization. Running a PDS means user ownership of your data, which seems very on mission for Cloudron.

It also handles auth, acting as your OAuth server wherever you log into with your account. There are multiple non-Bluesky apps with different data types (called Lexicons). A few of my favourites:

• Smoke Signal - events & RSVP https://smokesignal.events
• White Wind - markdown blogs https://whtwnd.com
• Frontpage - link ranking & discussion https://frontpage.fyi
• Recipe Exchange - recipe storage https://recipe.exchange

When you log into other AT Prototocol powered apps, the data is written to your own PDS. Here's my (Bluesky hosted) account with all the data types: https://atproto-browser.vercel.app/at/bmann.ca

It is also extremely low resource usage. Runs on NodeJS and SQL lite and some web socket connections.

I haven't gotten far in learning Cloudron packaging myself yet, but I'll update this thread if I get something guying.

CPU is back down to 5% for the last day with just a couple of Ghost websites running. Restarting mail, mongodb and slowly turning on each of the apps. Crossing my fingers that this works.

Node 15-30%, dockerd 40-80%, mysql 6 - 8% (Two ghost apps which use mysql are the only ones I've succesfully restarted). Docker shows up periodically at 20% and then drops off again.

My reference to 8.1 was a hope for a 8.1.x that magically fixes things :

This is what my CPU usage shows from the Hetzner control panel:

My logs tell me that Cloudron updated to 8.1 on 11/15, so at this point I assume that's the culprit.

Any other ideas?

I used docker stop to turn off all the apps to get load down. Now load average: 1.59, 5.06, 3.37

That corrupt addon doesn't have any information in it on what to do, unless you mean "Now, restore each app that uses the addon in the Cloudron dashboard."

I used docker stop to stop the PG container, then went ahead and did that for mongo and mysql and mail which were all yellow.

All redis instances also yellow. OK, let's stop them, too.

All containers are off other than core services and three surfer instances. Those all work, although slow to load. Load dipped down to 0.65, yay!

Ok, now I turn back on Postgres, it launches succesfully. I forget that MySQL is what I need for my Ghost website, so I launch that. Both of them turn back on (pressed restart from red in the Services screen in web dashboard).

I then go to Repair > Restart App for the site I'm trying to get up. It launches!

So here's where box logs are at: box:apphealthmonitor app health: 4 running / 16 stopped / 11 unresponsive

We're still at oad average: 1.20, 1.30, 1.93. That's with just one app running, where before 8.1 everything was running fine. I haven't turned back on mail or mongodb or redis yet.

Any ideas here on what is causing this load? What is the recommended path? Is there an 8.1 update soon that might improve things?

@nebulon I haven’t had it successfully start. Just repeated FATAL lines as above.

I’ve restarted the VPS and it’s all under heavy load.

What’s next here? I’m stuck.