Automatically repair app when the HealthCheck goes down (Not Responding)

[Lonkle]

@nebulon said in Automatically repair app when the HealthCheck goes down (Not Responding):

Can you imagine Cloudron just issuing a server restart automatically since we found that this often fixed issues in the past?

Well, of course not. That would defeat the entire purpose of this thread which is to have your apps have as much uptime as needed. Your opposition is, “we, developers need to make a perfect system and fix things so that our healthcheck isn’t even needed.” I agree that that’s the goal but you’re not there yet (both my and the other users on this thread attest to that) so why not protect users to these once off oddities that will never be reported if it doesn’t happen more than once? Because users are still going to hit repair and not do a single thing about it. But if in the notification there was an “ignore” button or “Send before and after logs to developers” then they may just press that and automate all of your manual steps. Because you could make this more complicated if you want - but in the end, it boils down to a user choosing to send you logs to fix an underlying issue. Even if you choose not to repair automatically once per “Non-responsive” / “Failed” app, you should then go further in your direction and make it easy for users to log these events and report to you. I still think a single courtesy attempted repair is best and Docker already has that built in so we’re gonna get it. Well, actually, I’m now making “Dot - The Repair Bot” to monitor the status of my apps and repair them if needed. This was always something I could do - even from inside a Cloudron app. And I intend to do it and release it for everyone. But this seems more suited as a built in feature, a checkbox that says “Attempt a single repair on app failure.”

And sure your steps weren’t exhaustive. But @ruihildt put it best. Average users don’t have time for any of that. I wasn’t advocating for automated sending of the logs. I was advocating for a button in the Notification Center that would send you, the developers, the relevant data you need - because users do not do what devs do (report bugs), but they would if it was a button click. Regardless of where you land on this matter, making it easier for devs to fix their apps is best if you’re going to say the only con to the pro of “more uptime” is “less developer communication.”

You guys for sure won’t be niche forever. So scaling ideas to reduce support requests should be considered heavily. Turns out for everyone in this thread, they repaired all of their apps and it never happened again. So no support requests when I could totally see someone complaining about having to manual repair 20 apps. The fact that you have a health check and this one-off repair has worked for all of us and we never reported it, means a singular automatic repair has no real-life con (aside from so far your theoretical con), but real life pros (uptime and reliability).

Coincidentally, this unintentionally happened to me again while rebooting the server. The app booted up not responding (1 of 4) and I repaired it and went on with my day. I’m a developer and didn’t think anything of it except that I wished it had been automatic in case I hadn’t noticed.

So I just want to ask, do you feel like my goal and your goal can possibly coexist? I can wait for the Docker update which adds this but I think I’ll create Dot to keep everything healthy automatically (which is something I would expect from an app manager). Average customer UX, saying this even as a developer, is more important than developer UX.

[nebulon]

Well really if you hit this just now on a reboot, I don't understand why this would not be investigated. We are happy to help. This is simply not acceptable having to run a repair after reboot. Repair in this case would just mean it recreates the containers after they were already just recreated after reboot. This makes no sense why it would fail first and then not, maybe there is some timing or resource issue lurking. I suspect there is an issue in the healthchecker state and the apps would have become healthy after some time. But all this is up in the air without further info. Repair may actually re-runs the last task! So after a reboot it may run some random unrelated app configuration task.

I have seen many times where users would restart/repair apps just because they felt it took too long for them to start up, such actions are sometimes even risky if an app is starting up. This is also why we don't show the task cancellation button immediately. Randomly interrupting processes is not a good idea if it can be avoided, as it heavily depends on how an app is written and how it can recover from for example interrupted database migrations.

[ruihildt]

@nebulon I totally understand the privacy aspect, so that could get logged, but only sent when needed.

The idea would be to have a snapshot of the logs at the time with all necessary informational to further debug even if the app is restarted for obvious reasons.

Cloudron promise is to make selfhosting easy, and such a system would make easier the bug reporting.

Maybe that's unrealistic.

[Lonkle]

@nebulon @nebulon We can 100% agree on that. But I have nothing to give you. There was just a down time notification until I Xed the notification and hit repair on the app. I tried rebooting again and it worked so my impression is it was some kind of race condition since I have four apps now and that’s the most I’ve ever had. I’ve rebooted plenty of times with nothing happening. But it’s kind of the point. Yes, this shouldn’t happen. But it does. It has. We shouldn’t be punishing users because we want a perfect system simply because we want more data. We should have safeguards for them and make logs easier for us to receive if something does happen. If I had a notification that auto-repaired my app and then I could hit “Send before and after logs to developers” with a description of what happened; I’d do it.

I’m making Dot the Repair Bot for me to accomplish this. Since I know we’re not going to agree on this. But why wouldn’t we be trying to find a win for you and win for users. Cloudron has become my favorite platform, I presented an issue and possible solution and it was rejected for being to “automated” but that’s the point of Cloudron, automating the redundant tasks of running apps on the web.

[Lonkle]

PS. @nebulon, do you ever sleep? #meneither

[Lonkle]

PSPS. I hope no one takes any disrespect when having opposing ideas on here. I always try to debate in good faith. I view us, the forum members, as needing to decide on important infrastructure aspects like this. So it’s us vs the problem. Not my idea vs anyone else’s. The problem is that sometimes users have to repair their apps manually. It usually happens after auto-updates / reboots (the only two times I’ve experienced it or read about others experiencing it on here). So what’s the solution? My solution is to auto-repair apps like that until we get enough data to fix the underlying problem. Meaning, we need not only an auto-repair so we’re not punishing users for a bad update (that will always happen), but also despite automatic repairs, easier log reporting of these errors to us.

So, it’s us vying for more logs to fix underlying issues, and automatic repair when something really is a one-ff situation which sometimes don’t have an explanation. Or they do but an average user couldn’t find it. But if they hit the “Send us crash logs” button then we might be able to fix it or find the issue.

[robi]

While this has STILL not been addressed by docker upstream, there is a proposed solution that works:
https://hub.docker.com/r/willfarrell/autoheal

Alternatively a simple cron script checking for unhealthy containers to restart them.

@staff what do you think?