Smarter healthcheck behaviour on restart

[Robin]

Cloudron sends health check notifications for each app ("App xxx is down / back online"). In normal use, this is quite useful, but when working on my Cloudron machine (rebooting it, or whatever), I then get a flood of emails for each application, so my inbox looks like:

Aaa is back up
Bbb is back up
Ccc is back up
Aaa is down
Bbb is down
Ccc is down

Suggested improvements:

• Perhaps it might make sense to delay the activation of health check alerts until (host or app) uptime is over a certain value?
• Consider merging notifications for multiple applications into one (or at least, one mail)

[Lonkle]

Very much agree with this.

[nebulon]

I agree, this is far from ideal how it is done now, especially for Cloudrons with a lot of apps.

Thinking on how to solve this, I am wondering what good options are. Merging notifications over some time would delay sending out the first one, however maybe this is just fine by simply trying to collect such notifications within say 5min?

[msbt]

@nebulon some uptime monitors have the option, that only if the check is positive/negative for a second or third time it will actually trigger a notification, maybe that's something to consider.

something somewhat unrelated but still an issue: apps that start up too quickly (before e.g. mysql is running, like teamspeak) appear to be running, but don't have a db connection. maybe we can start the main services and only when they're all up the regular apps are getting started?

[girish]

I didn't realize this happens for every reboot, let me test. Atleast, the way I intended it was to only send notifications if the app is down for 10mins.

edit: Maybe it's because the reboot takes more than 10 mins, I think that's the bug. As in, if i keep the cloudron powered down, and power it back up after 10mins, I guess it will trigger this behavior.

[plusone-nick]

@girish I get many of the same emails not for every app but like 5-10 at a time, seems to happen daily now possibly when backups are performing or completed?

[ruihildt]

With more than 100 apps, you can guess I have an overload of emails I delete without even looking at them.

[plusone-nick]

@ruihildt app is down emails go brrrrr