Automated server reboot

[potemkin_ai]

As a follow up for a previously discussed feature request - I would like to have an option to automatically reboot the server at the updates time window.

At the meantime, I guess a simple shell script at cron will do the trick:

if test -f /var/run/reboot-required; then
    shutdown -r now
fi

If that looks right, there is one more thing that I would like to add to the script (and automated reboot procedure as well) - it's e-mail notification that the server is going to be rebooted and, ideally, that it came back on-line - could you please, assist me with simple command line, that will work inside of the server and send e-mail to the admin using configured e-mail?

Thanks in advance?

[girish]

@potemkin_ai Did I understand correctly that the reboot is tied to the ubuntu security updates requiring a restart? (i.e this is not a generic automated server reboot feature).

[potemkin_ai]

@girish in my case - it is for sure - the only reason why I'm ready to risk automated server restart, it's to ensure that my server is not exposed to some remote issue, that I will discover only after a few days and it will be too late.

On my ubuntu servers I usually configure unattended upgrades for security only and automatic updates, if required, at the least used time.

[girish]

Thanks for clarifying.

While it's easy to implement, have to think more about this one. After all, no VPS provider I know has this functionality. Maybe an idea is to help the sysadmin somehow automate this via API call (say via n8n).

[Robin]

I'm also a big fan of automatic reboots. Both to keep things updated, but also because in my opinion, the more often something restarts, the less likely you are to get bitten by somehow relying on stale state.

(Meaning, you're more likely to notice that something no longer restarts immediately after it breaks, rather than 3-4 months down the line when you no longer remember what you changed :))

[potemkin_ai]

@girish sorry, not sure I understood the problem here - if you use unattended-upgrade under the whood, it's just about adding more line of code.

If you don't, this could be very easily done via the script as I offered earlier.

If you don't plan to implement this anytime soon, I would highly appreciate the help with e-mail notification call to be added to the script, so that I will just install it on my servers and be happy about this

[nebulon]

@potemkin_ai ubuntu unattended updates are already enabled by default to get security updates independently of Cloudron updates. I think you only then miss the automatic reboot if required (like a kernel security update was applied)

Generally updating ubuntu otherwise manually is not advised as Cloudron requires specific versions of docker and the likes.

[potemkin_ai]

@nebulon that’s right - I exactly miss two options from unattended upgrades… or 3:

• reboot if required
• in specific time
• notify via email if any issues (on error)

Any chance?

[nebulon]

@potemkin_ai maybe we can take a closer look for the next release. So far the automatic reboot wasn't a pressing issue, so if you need this anytime soon, I guess a simple bashscript doing all this for you will be best. Would be nice if you could share it here, since I currently wouldn't even know how to figure out if any of the packages would have failed to notify.

Also we do not have any generic email route to just send out custom text to the admins via the mail configuration since there was no use-case for this either.

Further with Ubuntu 20 and ongoing I think there is also a possibility to avoid reboots altogether for kernel updates, so maybe this would be a better solution in the long run anyways?

[potemkin_ai]

@nebulon a possibility to live without reboots with software up to date is a much better option - that's for sure! And which feature do you have in mind, mentioning that?
From what I recall kernel patching without reboot was a paid feature, or am I missing something?

As for the script sharing - I guess it's all, actually... I would probably just add something like ssmtp and add it to cron to reboot at less use time.

[girish]

I think you can use swaks to send mail from Cloudron. Let me dig up the exact way to use it.

[girish]

You can use swaks like this:

First get the IP of the mail container:

# docker inspect --format '{{ .NetworkSettings.Networks.cloudron.IPAddress }}' mail
172.18.0.2

Then, you need the "token" which is the auth to send mails:

# docker inspect --format '{{ .Config.Env }}' mail | tr ' ' '\n' | grep CLOUDRON_RELAY_TOKEN | sed 's/^.*=//'
9e000daa7ba34f29ed975d54b880f30794c2cebcd795f124

Now, you can send using swaks (replace IP and token below):

# swaks -s 172.18.0.2 -p 2525 --au no-reply@cloudron.space --ap 9e000daa7ba34f29ed975d54b880f30794c2cebcd795f1241 -f 'no-reply@cloudron.space' -t 'test@cloudron.io' --h-Subject "Test mail"

[potemkin_ai]

@girish thank you so much!

I love automation and one-liners, so I guess it could be all combined into one command:

swaks -s `docker inspect --format '{{ .NetworkSettings.Networks.cloudron.IPAddress }}' mail` -p 2525 --au $MY_LOCAL_MAILBOX --ap `docker inspect --format '{{ .Config.Env }}' mail | tr ' ' '\n' | grep CLOUDRON_RELAY_TOKEN | sed 's/^.*=//'` -f '$MY_LOCAL_MAILBOX' -t '$MY_PERSONAL_MAIL' --h-Subject "Test mail"

The only problem - is that it doesn't work - please, find the output below:

=== Trying 172.18.0.5:2525...
=== Connected to 172.18.0.5.
<-  220 my.domain ESMTP Haraka/2.8.27 ready
 -> EHLO localhost
<-  250-my.domain Hello [172.18.0.1]Haraka is at your service.
<-  250-PIPELINING
<-  250-8BITMIME
<-  250-SMTPUTF8
<-  250-SIZE 26214400
<-  250 AUTH LOGIN PLAIN
 -> AUTH LOGIN
<-  334 VXNlcm5hbWU6
 -> YWxleEBwb3RlbWtpbi5jbw==
<-  334 UGFzc3dvcmQ6
 -> $TOKEN==
<** 535 5.7.8 Authentication failed
 -> AUTH PLAIN AGFsZXhAcG90ZW1raW4uY28A$TOKEN==
<** 535 5.7.8 Authentication failed
*** No authentication type succeeded
 -> QUIT
<-  221 my.domain closing connection. Have a jolly good day.
=== Connection closed with remote host.

Any thoughts what could went wrong?

[potemkin_ai]

@nebulon did you mean manual configuration for that service?
https://ubuntu.com/tutorials/enable-the-livepatch-service#1-overview

[nebulon]

@potemkin_ai yes I was referring to this, but I haven't really had a deeper look into that topic yet and to which extent this is free or a paid addon in ubuntu.

[girish]

@potemkin_ai said in Automated server reboot:

-> $TOKEN==
<** 535 5.7.8 Authentication failed
-> AUTH PLAIN AGFsZXhAcG90ZW1raW4uY28A$TOKEN==
<** 535 5.7.8 Authentication failed
*** No authentication type succeeded

Something wrong in the script. It's sending $TOKEN

[potemkin_ai]

@nebulon got it, makes sense, but from what I see it requires extra configuration anyway and most probably won't be available out of the box, so an automatic reboot would be of great help; especially for the next two years while 18.04LTS is supported

[potemkin_ai]

@girish thank you.

I re-did all the steps manually as per your original message and got the same error.

The only major difference between your and my setup - it's token length - it's 257 characters long in my case.

Any clue?

[girish]

@potemkin_ai command below works for me. One thing is the no-reply@mydomain.com is important! You cannot change the mailbox name. The mail server does not allow sending emails as any other name for security from localhost.

# swaks -s `docker inspect --format '{{ .NetworkSettings.Networks.cloudron.IPAddress }}' mail` -p 2525 --au no-reply@cloudron.space --ap `docker inspect --format '{{ .Config.Env }}' mail | tr ' ' '\n' | grep CLOUDRON_RELAY_TOKEN | sed 's/^.*=//'` -f no-reply@cloudron.space -t 'test@cloudron.io' --h-Subject "Test mail"
=== Trying 172.18.0.3:2525...
=== Connected to 172.18.0.3.
<-  220 my.cloudron.space ESMTP Haraka/2.8.28 ready
 -> EHLO upgrade-test
<-  250-my.cloudron.space Hello [172.18.0.1]Haraka is at your service.
<-  250-PIPELINING
<-  250-8BITMIME
<-  250-SMTPUTF8
<-  250-SIZE 26214400
<-  250 AUTH LOGIN PLAIN
 -> AUTH LOGIN
<-  334 VXNlcm5hbWU6
 -> bm8tcmVwbHlAY2xvdWRyb24uc3BhY2U=
<-  334 UGFzc3dvcmQ6
 -> OWZmOGU4MWI4NTJmNjMxOWVlNWI4NjM0NDQ1ZDE1ZTliYTI5ZWY3NjE5N2UwZWE0MmM5OGJlMTI0ZjQ4NjFhM2EzZmEyNDQ2Yzk2MzVmODYzMDNiNTUwNjRiMDRjOGYwNjc1M2U3NjIxZGJhOGFmZGY1YWJlODUyYmVmMTVmZWVjNmFiOTVjYmE1MDJhNGY5OGZiMjBjMWZiZDA3YjBlMDFjOTBhOTE4NTRhOGY4ZDNhYWI2ZmYyMGJkMjUzZjY1NDAwNmE0YmRjZGVhMmM2ZGYzYTY0YjcxMWRjZmQxODE5Y2FiZDJhMmMyMTY2YTJhYmE5MjA4NGNjNDhmN2E1ZQ==
<-  235 2.7.0 Authentication successful
 -> MAIL FROM:<no-reply@cloudron.space>
<-  250 sender <no-reply@cloudron.space> OK
 -> RCPT TO:<test@cloudron.io>
<-  250 recipient <test@cloudron.io> OK
 -> DATA
<-  354 go ahead, make my day
 -> Date: Fri, 22 Oct 2021 18:59:51 +0000
 -> To: test@cloudron.io
 -> From: no-reply@cloudron.space
 -> Subject: Test mail
 -> Message-Id: <20211022185951.169923@upgrade-test>
 -> X-Mailer: swaks v20190914.0 jetmore.org/john/code/swaks/
 -> 
 -> This is a test mailing
 -> 
 -> 
 -> .
<-  250 Message Queued (0741EB98-48BA-4A23-AB22-FB67C176AF7B.1)
 -> QUIT
<-  221 my.cloudron.space closing connection. Have a jolly good day.
=== Connection closed with remote host.

[potemkin_ai]

@girish thank you - tried that, but authentification is still failing at 334 step.

My primary concern remains token, because mine is muuuuch longer than yours (hope it sounds right in the context of token comparison....), and authentication is failing right after mail service gets it.

Any way to troubleshoot that?