7.5.1 bug in event logs and unexpected boot

[imc67]

Yesterday evening I updated one of my Cloudrons from 7.5 to 7.5.1. All looked ok.

I just was reviewing the Event Log (see screenshot) and noticed two strange things:

1. in the details line it says "logged in to undefined" where before the corresponding app or service was noted
2. today at 15:10h it seems that Cloudron was suddenly restarted?!?!

[nebulon]

@girish was already working on the undefined display issue. This comes up with the move to OpenID instead of LDAP. Will be fixed with next release.

For the Cloudron started event, maybe just the box service restarted? You can check on the server with journalctl --list-boots over SSH when actual system reboots happened.

[imc67]

@nebulon said in 7.5.1 bug in event logs and unexpected boot:

For the Cloudron started event, maybe just the box service restarted? You can check on the server with journalctl --list-boots over SSH when actual system reboots happened.

This is the result but the date/time is totally different and I'm not aware of a system reboot:

~# journalctl --list-boots
 0 a7dbd8**********b3b43576519eb703 Fri 2023-08-25 05:44:24 UTC—Fri 2023-08-25 16:54:59 UTC

[girish]

@imc67 the boot is referring to the box code starting and not to the system booting. Can you check if there is a crash in the logs around the time the event was raised?

[imc67]

@girish not really a sign of the root cause, after many "box:apphealthmonitor app health:" it suddenly:

2023-08-25T13:09:50.139Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive
2023-08-25T13:10:00.965Z box:server ==========================================
2023-08-25T13:10:00.966Z box:platform initializing platform
2023-08-25T13:10:00.966Z box:server            Cloudron 7.5.1  
2023-08-25T13:10:00.966Z box:server ==========================================
2023-08-25T13:10:00.992Z box:tasks stopAllTasks: stopping all tasks
2023-08-25T13:10:00.993Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2023-08-25T13:10:01.023Z box:shell stopTask (stdout): All tasks stopped
2023-08-25T13:10:01.035Z box:reverseproxy writeDashboardConfig: writing admin config for ***.**
2023-08-25T13:10:01.055Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2023-08-25T13:10:01.263Z box:platform onActivated: starting post activation services
2023-08-25T13:10:01.264Z box:platform startInfra: checking infrastructure
2023-08-25T13:10:01.265Z box:apps schedulePendingTasks: scheduling app tasks
2023-08-25T13:10:01.265Z box:platform onInfraReady: platform is ready. infra changed: false
2023-08-25T13:10:01.265Z box:platform startInfra: infra is uptodate at version 49.5.0
2023-08-25T13:10:01.301Z box:cron startJobs: starting cron jobs with hour 13 and minute 55
2023-08-25T13:10:01.340Z box:cron backupPolicyChanged: schedule 00 00 8,20 * * * (Europe/Amsterdam)
2023-08-25T13:10:01.351Z box:cron autoupdatePatternChanged: pattern - 00 00 2,3,4 * * * (Europe/Amsterdam)
2023-08-25T13:10:01.354Z box:cron Dynamic DNS setting changed to false
2023-08-25T13:10:01.355Z box:dockerproxy startDockerProxy: started proxy on port 3003
2023-08-25T13:10:01.726Z box:oidc Using existing OIDC EdDSA key
2023-08-25T13:10:01.727Z box:oidc Using existing OIDC RS256 key
2023-08-25T13:10:01.730Z box:oidc start: create provider for my.****.** at /openid
2023-08-25T13:10:10.166Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive
2023-08-25T13:10:20.221Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive
2023-08-25T13:10:30.126Z box:apphealthmonitor app health: 22 running / 3 stopped / 0 unresponsive
2023-08-25T13:10:31.754Z box:reverseproxy writeDefaultConfig: writing configs for endpoint "ip"
2023-08-25T13:10:31.754Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx

[imc67]

Today same time the reboot occurred again!

[nebulon]

Do you see any extra logs in the system logs, like journalctl -u box or journalctl --system around that time?

[imc67]

I've sent you an email and opened SSH for you to investigate

[jdaviescoates]

I've just spotted something similar on one of my Cloudrons too

This doesn't seem right. Seems to have started doing it with 7.5.0 too, only just noticing it...

Update, ssh'd into the server, see this:

# journalctl --list-boots
 0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Sun 2023-08-27 23:36:47 UTC—Mon 2023-08-28 21:34:21 UTC

Which I think says that the whole system has rebooted twice recently?

[nebulon]

I just looked into the issue and indeed once per day it crashes as the renew certificates call fails.

I've pushed a fix https://git.cloudron.io/cloudron/box/-/commit/24981e1f811cfa20eff6c63f8cf2397691d0976f will be included in the 7.5.2 then.

[nebulon]

Just to be clear though, this will just restart the box process which just takes a few seconds and its not an actual server reboot. A slightly bigger issue is, that the certs won't renew automatically until the fix is out, this can be done manually until that using the renew all certs action in the domains view. So if you are on 7.5 at the moment just click that button in the next few days.