Automatically repair app when the HealthCheck goes down (Not Responding)
-
@nebulon Would it be complicated to automate that 3 steps reporting? (For example, clicking repair would trigger this by default)
Like I said, if I have clients complaining and if I'm not in front of my computer with free time on hand, I'm going to hit that repair button to get back online ASAP.
This was my situation yesterday, I'm sick and stuck in bed with just my smartphone, it would have helped if it was automated, for you and for me.
-
@ruihildt said in Automatically repair app when the HealthCheck goes down (Not Responding):
@nebulon Would it be complicated to automate that 3 steps reporting? (For example, clicking repair would trigger this by default)
Like I said, if I have clients complaining and if I'm not in front of my computer with free time on hand, I'm going to hit that repair button to get back online ASAP.
This was my situation yesterday, I'm sick and stuck in bed with just my smartphone, it would have helped if it was automated, for you and for me.
Exactly, what @nebulon and @girish want could be automated and in doing so - it increases what they want (more data to fix underlying problems) while at the same time keeping the users apps uptime as high as possible. I want everyone to win in this scenario so I want to hear everyone’s pros and cons.
-
Well those steps are just generic debugging and investigation steps for sysadmins. They may or may not apply and are certainly not exhaustive, just what came to mind while writing this. Plus we don't generally just send information wholesome from your server to us also for privacy reasons. You can still manually issue a support ticket for that app from the support view, which will include the app logs.
The out-of-memory restart is something different though. The underlying issue is known here and restarting due to out-of-memory is the correct thing to do. There is also no solution code-wise, since we can't just up the memory limit automatically, risking over-provisioning the server. Nor can we add memory to the server automatically.
Again if you have concrete situations where a repair may solve the issue, we have to investigate. Keep in mind that in our experience this is not at all common across our users. Can you imagine Cloudron just issuing a server restart automatically since we found that this often fixed issues in the past?
-
@nebulon said in Automatically repair app when the HealthCheck goes down (Not Responding):
Can you imagine Cloudron just issuing a server restart automatically since we found that this often fixed issues in the past?
Well, of course not. That would defeat the entire purpose of this thread which is to have your apps have as much uptime as needed. Your opposition is, “we, developers need to make a perfect system and fix things so that our healthcheck isn’t even needed.” I agree that that’s the goal but you’re not there yet (both my and the other users on this thread attest to that) so why not protect users to these once off oddities that will never be reported if it doesn’t happen more than once? Because users are still going to hit repair and not do a single thing about it. But if in the notification there was an “ignore” button or “Send before and after logs to developers” then they may just press that and automate all of your manual steps. Because you could make this more complicated if you want - but in the end, it boils down to a user choosing to send you logs to fix an underlying issue. Even if you choose not to
repairautomatically once per “Non-responsive” / “Failed” app, you should then go further in your direction and make it easy for users to log these events and report to you. I still think a single courtesy attempted repair is best and Docker already has that built in so we’re gonna get it. Well, actually, I’m now making “Dot - The Repair Bot” to monitor the status of my apps and repair them if needed. This was always something I could do - even from inside a Cloudron app. And I intend to do it and release it for everyone. But this seems more suited as a built in feature, a checkbox that says “Attempt a single repair on app failure.”And sure your steps weren’t exhaustive. But @ruihildt put it best. Average users don’t have time for any of that. I wasn’t advocating for automated sending of the logs. I was advocating for a button in the Notification Center that would send you, the developers, the relevant data you need - because users do not do what devs do (report bugs), but they would if it was a button click. Regardless of where you land on this matter, making it easier for devs to fix their apps is best if you’re going to say the only con to the pro of “more uptime” is “less developer communication.”
You guys for sure won’t be niche forever. So scaling ideas to reduce support requests should be considered heavily. Turns out for everyone in this thread, they repaired all of their apps and it never happened again. So no support requests when I could totally see someone complaining about having to manual repair 20 apps. The fact that you have a health check and this one-off repair has worked for all of us and we never reported it, means a singular automatic repair has no real-life con (aside from so far your theoretical con), but real life pros (uptime and reliability).
Coincidentally, this unintentionally happened to me again while rebooting the server. The app booted up not responding (1 of 4) and I repaired it and went on with my day. I’m a developer and didn’t think anything of it except that I wished it had been automatic in case I hadn’t noticed.
So I just want to ask, do you feel like my goal and your goal can possibly coexist? I can wait for the Docker update which adds this but I think I’ll create Dot to keep everything healthy automatically (which is something I would expect from an app manager). Average customer UX, saying this even as a developer, is more important than developer UX.
-
Well really if you hit this just now on a reboot, I don't understand why this would not be investigated. We are happy to help. This is simply not acceptable having to run a repair after reboot. Repair in this case would just mean it recreates the containers after they were already just recreated after reboot. This makes no sense why it would fail first and then not, maybe there is some timing or resource issue lurking. I suspect there is an issue in the healthchecker state and the apps would have become healthy after some time. But all this is up in the air without further info. Repair may actually re-runs the last task! So after a reboot it may run some random unrelated app configuration task.
I have seen many times where users would restart/repair apps just because they felt it took too long for them to start up, such actions are sometimes even risky if an app is starting up. This is also why we don't show the task cancellation button immediately. Randomly interrupting processes is not a good idea if it can be avoided, as it heavily depends on how an app is written and how it can recover from for example interrupted database migrations.
-
@nebulon I totally understand the privacy aspect, so that could get logged, but only sent when needed.
The idea would be to have a snapshot of the logs at the time with all necessary informational to further debug even if the app is restarted for obvious reasons.
Cloudron promise is to make selfhosting easy, and such a system would make easier the bug reporting.
Maybe that's unrealistic.
-
@nebulon @nebulon We can 100% agree on that. But I have nothing to give you. There was just a down time notification until I Xed the notification and hit repair on the app. I tried rebooting again and it worked so my impression is it was some kind of race condition since I have four apps now and that’s the most I’ve ever had. I’ve rebooted plenty of times with nothing happening. But it’s kind of the point. Yes, this shouldn’t happen. But it does. It has. We shouldn’t be punishing users because we want a perfect system simply because we want more data. We should have safeguards for them and make logs easier for us to receive if something does happen. If I had a notification that auto-repaired my app and then I could hit “Send before and after logs to developers” with a description of what happened; I’d do it.
I’m making Dot the Repair Bot for me to accomplish this. Since I know we’re not going to agree on this. But why wouldn’t we be trying to find a win for you and win for users. Cloudron has become my favorite platform, I presented an issue and possible solution and it was rejected for being to “automated” but that’s the point of Cloudron, automating the redundant tasks of running apps on the web.
-
PSPS. I hope no one takes any disrespect when having opposing ideas on here. I always try to debate in good faith. I view us, the forum members, as needing to decide on important infrastructure aspects like this. So it’s us vs the problem. Not my idea vs anyone else’s. The problem is that sometimes users have to repair their apps manually. It usually happens after auto-updates / reboots (the only two times I’ve experienced it or read about others experiencing it on here). So what’s the solution? My solution is to auto-repair apps like that until we get enough data to fix the underlying problem. Meaning, we need not only an auto-repair so we’re not punishing users for a bad update (that will always happen), but also despite automatic repairs, easier log reporting of these errors to us.
So, it’s us vying for more logs to fix underlying issues, and automatic repair when something really is a one-ff situation which sometimes don’t have an explanation. Or they do but an average user couldn’t find it. But if they hit the “Send us crash logs” button then we might be able to fix it or find the issue.
-
While this has STILL not been addressed by docker upstream, there is a proposed solution that works:
https://hub.docker.com/r/willfarrell/autohealAlternatively a simple cron script checking for unhealthy containers to restart them.
@staff what do you think?
-
Z Zoldir referenced this topic on
#meneither