7.5.1 bug in event logs and unexpected boot
-
Yesterday evening I updated one of my Cloudrons from 7.5 to 7.5.1. All looked ok.
I just was reviewing the Event Log (see screenshot) and noticed two strange things:
- in the details line it says "logged in to undefined" where before the corresponding app or service was noted
- today at 15:10h it seems that Cloudron was suddenly restarted?!?!
-
@girish was already working on the
undefined
display issue. This comes up with the move to OpenID instead of LDAP. Will be fixed with next release.For the Cloudron started event, maybe just the
box
service restarted? You can check on the server withjournalctl --list-boots
over SSH when actual system reboots happened. -
@nebulon said in 7.5.1 bug in event logs and unexpected boot:
For the Cloudron started event, maybe just the box service restarted? You can check on the server with journalctl --list-boots over SSH when actual system reboots happened.
This is the result but the date/time is totally different and I'm not aware of a system reboot:
~# journalctl --list-boots 0 a7dbd8**********b3b43576519eb703 Fri 2023-08-25 05:44:24 UTC—Fri 2023-08-25 16:54:59 UTC
-
@girish not really a sign of the root cause, after many "box:apphealthmonitor app health:" it suddenly:
2023-08-25T13:09:50.139Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive 2023-08-25T13:10:00.965Z box:server ========================================== 2023-08-25T13:10:00.966Z box:platform initializing platform 2023-08-25T13:10:00.966Z box:server Cloudron 7.5.1 2023-08-25T13:10:00.966Z box:server ========================================== 2023-08-25T13:10:00.992Z box:tasks stopAllTasks: stopping all tasks 2023-08-25T13:10:00.993Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2023-08-25T13:10:01.023Z box:shell stopTask (stdout): All tasks stopped 2023-08-25T13:10:01.035Z box:reverseproxy writeDashboardConfig: writing admin config for ***.** 2023-08-25T13:10:01.055Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx 2023-08-25T13:10:01.263Z box:platform onActivated: starting post activation services 2023-08-25T13:10:01.264Z box:platform startInfra: checking infrastructure 2023-08-25T13:10:01.265Z box:apps schedulePendingTasks: scheduling app tasks 2023-08-25T13:10:01.265Z box:platform onInfraReady: platform is ready. infra changed: false 2023-08-25T13:10:01.265Z box:platform startInfra: infra is uptodate at version 49.5.0 2023-08-25T13:10:01.301Z box:cron startJobs: starting cron jobs with hour 13 and minute 55 2023-08-25T13:10:01.340Z box:cron backupPolicyChanged: schedule 00 00 8,20 * * * (Europe/Amsterdam) 2023-08-25T13:10:01.351Z box:cron autoupdatePatternChanged: pattern - 00 00 2,3,4 * * * (Europe/Amsterdam) 2023-08-25T13:10:01.354Z box:cron Dynamic DNS setting changed to false 2023-08-25T13:10:01.355Z box:dockerproxy startDockerProxy: started proxy on port 3003 2023-08-25T13:10:01.726Z box:oidc Using existing OIDC EdDSA key 2023-08-25T13:10:01.727Z box:oidc Using existing OIDC RS256 key 2023-08-25T13:10:01.730Z box:oidc start: create provider for my.****.** at /openid 2023-08-25T13:10:10.166Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive 2023-08-25T13:10:20.221Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive 2023-08-25T13:10:30.126Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive 2023-08-25T13:10:31.754Z box:reverseproxy writeDefaultConfig: writing configs for endpoint "ip" 2023-08-25T13:10:31.754Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
-
Today same time the reboot occurred again!
-
I've sent you an email and opened SSH for you to investigate
-
I've just spotted something similar on one of my Cloudrons too
This doesn't seem right. Seems to have started doing it with 7.5.0 too, only just noticing it...
Update, ssh'd into the server, see this:
# journalctl --list-boots 0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Sun 2023-08-27 23:36:47 UTC—Mon 2023-08-28 21:34:21 UTC
Which I think says that the whole system has rebooted twice recently?
-
I just looked into the issue and indeed once per day it crashes as the renew certificates call fails.
I've pushed a fix https://git.cloudron.io/cloudron/box/-/commit/24981e1f811cfa20eff6c63f8cf2397691d0976f will be included in the 7.5.2 then.
-
Just to be clear though, this will just restart the
box
process which just takes a few seconds and its not an actual server reboot. A slightly bigger issue is, that the certs won't renew automatically until the fix is out, this can be done manually until that using the renew all certs action in the domains view. So if you are on 7.5 at the moment just click that button in the next few days. -
-