Server not available after reboot

[SirFiChi]

Hello Cloudron Team,
after a reboot of our system the Cloudron instance cannot be reached anymore.
I am also not able to log in to the Cloudron site with one of the mail addresses because the mails were managed through the Cloudron server (@OS-Television.de).
What options do I have to either get our server started Cloudron again to retrieve mails, or log in to Cloudron.io?
Thank you thanks for the support.

Floh Klare

[girish]

@SirFiChi Have you see https://docs.cloudron.io/troubleshooting/ ? Let us know the status of the server following those steps.

[SirFiChi]

@girish
Expired domain - I restarted the server within the webinterface.
Nginx error - I'm not able to connect to the server since the reboot.
Failed upgrade - I'm not able to connect to the server since the reboot.
Unresponsive App- I'm not able to connect to the server since the reboot.
Unresponsive Service - I'm not able to connect to the server since the reboot.
Certificates - I'm not able to connect to the server since the reboot.
DNS - I'm not able to connect to the server since the reboot.

[subven]

@SirFiChi use the VNC console from your hosting provider to access the server if SSH is not accessible.

There is nothing more we can do for you since you have no access to your server and can't provide logs/errors or anything else.

[robi]

OS-Television.de seems to respond just fine.
my.OS-Television.de does not (if that is where your cloudron dashboard was installed)

Is the expired domain renewed?

[subven]

@robi if you ping both domains, you get different IPs which indicates that the root domain points to another server and not the one in question.

[SirFiChi]

@subven
Cloudron is installed on my.tvorga.de.
VNC of my provider isn't possible too. =/

But I got the Password-REset-Mail for one of the two possible Mailadresses.

I'll try to contact the provider.
Thanks for your help.

[SirFiChi]

Hello together,
The current status is that I found out that you can access the installation from the rescue system via chroot, and that the SSH port on Cloudron is not 22.
So now I can say that SSH works. I have reset the root password, but it asks for a "verification code".
The installation is still not accessible via https/http.

Can I set the "Verification code"?

[subven]

@SirFiChi the root password can only be set by your VPS provider. It has nothing to do with Cloudron. I guess by "verification code", your VPS provider is asking you for a code to verify that you have access to the mail address associated with your VPS account. This is to prevent intruders from taking over everything after a login is compromised. AFAIK, non of your current problems are Cloudron related.

The standard SSH port for Cloudron is indeed 22 but can be changed by the server administrator (you).

If you want to change the password for a Cloudron account, you can do so over SSH.

[SirFiChi]

@subven
All these professionals around me.
Thanks for the tips. I had GoogleAuthenticator installed, but no longer have the appropriate app on my phone (due to phone swap). I was able to restore access now and the nginx must have a broken configuration?

systemctl status nginx

● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/nginx.service.d
           └─cloudron.conf
   Active: failed (Result: exit-code) since Thu 2022-11-17 23:43:18 UTC; 6min ago
     Docs: http://nginx.org/en/docs/
  Process: 879 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)

Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Service hold-off time over, scheduling restart.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Scheduled restart job, restart counter is at 5.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: Stopped nginx - high performance web server.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Start request repeated too quickly.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Failed with result 'exit-code'.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: Failed to start nginx - high performance web server.

journalctl -u nginx -fa
-- Logs begin at Thu 2022-11-17 12:13:11 UTC. --
Nov 17 23:43:18 vmd52314.contaboserver.net nginx[879]: nginx: [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/old.os-television.de.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/home/yellowtent/platformdata/nginx/cert/old.os-television.de.cert','r') error:2006D080:BIO routines:BIO_new_file:no such file)
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Control process exited, code=exited status=1
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Failed with result 'exit-code'.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: Failed to start nginx - high performance web server.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Service hold-off time over, scheduling restart.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Scheduled restart job, restart counter is at 5.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: Stopped nginx - high performance web server.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Start request repeated too quickly.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: nginx.service: Failed with result 'exit-code'.
Nov 17 23:43:18 vmd52314.contaboserver.net systemd[1]: Failed to start nginx - high performance web server.

journalctl -xe
-- the configured Restart= setting for the unit.
Nov 17 23:57:10 vmd52314.contaboserver.net systemd[1]: Stopped nginx - high performance web server.
-- Subject: Unit nginx.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit nginx.service has finished shutting down.
Nov 17 23:57:10 vmd52314.contaboserver.net systemd[1]: nginx.service: Start request repeated too quickly.
Nov 17 23:57:10 vmd52314.contaboserver.net systemd[1]: nginx.service: Failed with result 'exit-code'.
Nov 17 23:57:10 vmd52314.contaboserver.net systemd[1]: Failed to start nginx - high performance web server.
-- Subject: Unit nginx.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit nginx.service has failed.
-- 
-- The result is RESULT.
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-networkd[362]: vetha0abfae: Link UP
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-timesyncd[426]: Network configuration changed, trying to establish connection.
Nov 17 23:57:32 vmd52314.contaboserver.net networkd-dispatcher[567]: WARNING:Unknown index 112 seen, reloading interface list
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-udevd[19988]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered blocking state
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered disabled state
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: device vetha0abfae entered promiscuous mode
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: IPv6: ADDRCONF(NETDEV_UP): vetha0abfae: link is not ready
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered blocking state
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered forwarding state
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered disabled state
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-udevd[19988]: Could not generate persistent MAC address for veth13cbccb: No such file or directory
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-udevd[19989]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-udevd[19989]: Could not generate persistent MAC address for vetha0abfae: No such file or directory
Nov 17 23:57:32 vmd52314.contaboserver.net containerd[574]: time="2022-11-17T23:57:32.242433654Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/c3d4617ef859a10a314a581d3d981
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-timesyncd[426]: Synchronized to time server 103.123.108.224:123 (2.ubuntu.pool.ntp.org).
Nov 17 23:57:32 vmd52314.contaboserver.net systemd-timesyncd[426]: Network configuration changed, trying to establish connection.
Nov 17 23:57:32 vmd52314.contaboserver.net kernel: eth0: renamed from veth13cbccb
Nov 17 23:57:33 vmd52314.contaboserver.net kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov 17 23:57:33 vmd52314.contaboserver.net systemd-networkd[362]: vetha0abfae: Gained carrier
Nov 17 23:57:33 vmd52314.contaboserver.net kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov 17 23:57:33 vmd52314.contaboserver.net kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 17 23:57:33 vmd52314.contaboserver.net kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vetha0abfae: link becomes ready
Nov 17 23:57:33 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered blocking state
Nov 17 23:57:33 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered forwarding state
Nov 17 23:57:33 vmd52314.contaboserver.net systemd-timesyncd[426]: Synchronized to time server 103.123.108.224:123 (2.ubuntu.pool.ntp.org).
Nov 17 23:57:34 vmd52314.contaboserver.net systemd-networkd[362]: vetha0abfae: Gained IPv6LL
Nov 17 23:57:34 vmd52314.contaboserver.net systemd-timesyncd[426]: Network configuration changed, trying to establish connection.
Nov 17 23:57:34 vmd52314.contaboserver.net systemd-timesyncd[426]: Synchronized to time server 103.123.108.224:123 (2.ubuntu.pool.ntp.org).
Nov 17 23:57:37 vmd52314.contaboserver.net dockerd[693]: time="2022-11-17T23:57:37.158048117Z" level=info msg="ignoring event" container=c3d4617ef859a10a314a581d3d981f9d6c6e1b5fac6e591230abbcb50d5ab64e module=libcontainerd namespace=mob
Nov 17 23:57:37 vmd52314.contaboserver.net containerd[574]: time="2022-11-17T23:57:37.159273912Z" level=info msg="shim disconnected" id=c3d4617ef859a10a314a581d3d981f9d6c6e1b5fac6e591230abbcb50d5ab64e
Nov 17 23:57:37 vmd52314.contaboserver.net containerd[574]: time="2022-11-17T23:57:37.159377507Z" level=warning msg="cleaning up after shim disconnected" id=c3d4617ef859a10a314a581d3d981f9d6c6e1b5fac6e591230abbcb50d5ab64e namespace=moby
Nov 17 23:57:37 vmd52314.contaboserver.net containerd[574]: time="2022-11-17T23:57:37.159406521Z" level=info msg="cleaning up dead shim"
Nov 17 23:57:37 vmd52314.contaboserver.net containerd[574]: time="2022-11-17T23:57:37.178279530Z" level=warning msg="cleanup warnings time=\"2022-11-17T23:57:37Z\" level=info msg=\"starting signal loop\" namespace=moby pid=20244\n"
Nov 17 23:57:37 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered disabled state
Nov 17 23:57:37 vmd52314.contaboserver.net kernel: veth13cbccb: renamed from eth0
Nov 17 23:57:37 vmd52314.contaboserver.net systemd-networkd[362]: vetha0abfae: Lost carrier
Nov 17 23:57:37 vmd52314.contaboserver.net systemd-timesyncd[426]: Network configuration changed, trying to establish connection.
Nov 17 23:57:37 vmd52314.contaboserver.net systemd-udevd[20269]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 17 23:57:37 vmd52314.contaboserver.net systemd-udevd[20269]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 17 23:57:37 vmd52314.contaboserver.net networkd-dispatcher[567]: WARNING:Unknown index 111 seen, reloading interface list
Nov 17 23:57:37 vmd52314.contaboserver.net systemd-networkd[362]: vetha0abfae: Link DOWN
Nov 17 23:57:37 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered disabled state
Nov 17 23:57:37 vmd52314.contaboserver.net kernel: device vetha0abfae left promiscuous mode
Nov 17 23:57:37 vmd52314.contaboserver.net kernel: br-398ed0848f39: port 20(vetha0abfae) entered disabled state
Nov 17 23:57:37 vmd52314.contaboserver.net systemd-timesyncd[426]: Synchronized to time server 103.123.108.224:123 (2.ubuntu.pool.ntp.org).
Nov 17 23:57:38 vmd52314.contaboserver.net kernel: Packet dropped: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:28:99:3a:4d:23:91:08:00 SRC=173.212.238.3 DST=255.255.255.255 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=33

systemctl restart nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.

[subven]

@SirFiChi

There is your error

Nov 17 23:43:18 vmd52314.contaboserver.net nginx[879]: nginx: [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/old.os-television.de.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/home/yellowtent/platformdata/nginx/cert/old.os-television.de.cert','r') error:2006D080:BIO routines:BIO_new_file:no such file)

I would suggest to copy an existend certificate from this folder and rename it to old.os-television.de.cert. I guess the app behind that URL was stopped (for a while)? In this case, Cloudron had(?) a bug where certs for stopped apps did not get renewal and could lead to the sitauation where NGINX does not come up after a restart.

After you are able to restart NGINX and bring up your Dashboard, you should start the stopped apps and renew their certificates. You can delete the fake old.os-television.de.cert we just created from the filesystem right before you trigger a certificate renew so it will be renewed 100% with a correct cert file.

[SirFiChi]

@subven

I found it ti, but I can't start the server:

systemctl start nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.
root@vmd52314:/etc/nginx/cert# systemctl status nginx.service
● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/nginx.service.d
           └─cloudron.conf
   Active: activating (start) since Fri 2022-11-18 00:51:28 UTC; 925ms ago
     Docs: http://nginx.org/en/docs/
Cntrl PID: 27071 (nginx)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/nginx.service
           └─27071 /usr/sbin/nginx -c /etc/nginx/nginx.conf

Nov 18 00:51:28 vmd52314.contaboserver.net systemd[1]: Starting nginx - high performance web server...
Nov 18 00:51:28 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Nov 18 00:51:28 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Nov 18 00:51:28 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Nov 18 00:51:28 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
Nov 18 00:51:29 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Nov 18 00:51:29 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Nov 18 00:51:29 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Nov 18 00:51:29 vmd52314.contaboserver.net nginx[27071]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)

journalctl -u nginx -fa


Nov 18 00:52:39 vmd52314.contaboserver.net nginx[27988]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Nov 18 00:52:39 vmd52314.contaboserver.net nginx[27988]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Nov 18 00:52:39 vmd52314.contaboserver.net nginx[27988]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Nov 18 00:52:39 vmd52314.contaboserver.net nginx[27988]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
Nov 18 00:52:40 vmd52314.contaboserver.net nginx[27988]: nginx: [emerg] still could not bind()
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: nginx.service: Control process exited, code=exited status=1
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: nginx.service: Failed with result 'exit-code'.
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: Failed to start nginx - high performance web server.
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: nginx.service: Service hold-off time over, scheduling restart.
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: nginx.service: Scheduled restart job, restart counter is at 32.
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: Stopped nginx - high performance web server.
Nov 18 00:52:40 vmd52314.contaboserver.net systemd[1]: Starting nginx - high performance web server...
Nov 18 00:52:40 vmd52314.contaboserver.net nginx[28012]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Nov 18 00:52:40 vmd52314.contaboserver.net nginx[28012]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Nov 18 00:52:40 vmd52314.contaboserver.net nginx[28012]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
Nov 18 00:52:40 vmd52314.contaboserver.net nginx[28012]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)

[SirFiChi]

Apache2 is not installed, but there are processes running for Apache2 with paths that are not present.

[SirFiChi]

Ahhh, but the cloudron-site is available. I try to fix the certificate...

[subven]

@SirFiChi said in Server not available after reboot:

Apache2 is not installed, but there are processes running for Apache2 with paths that are not present.

Hope you did not made the mistake to manually install packages on the server or do stuff like apt update/upgrade. Use systemctl restart box to restart Cloudrons services only. I also like systemctl list-units --failed

[SirFiChi]

@subven
I did this about 2-3 hours ago in recovery mode.... -.-

[SirFiChi]

@subven @girish @robi
Thank you for the renewed competent help and patience.
After I had restored the last backup, the old instance could be restarted and stopped.
After a reboot of the server everything runs as usual again.
I hope my system has now no lasting damage by apt-get update/upgrade/dist-upgrade and "apt autoremove".

[subven]

@SirFiChi said in Server not available after reboot:

I hope my system has now no lasting damage by apt-get update/upgrade/dist-upgrade and "apt autoremove".

I hope so too but to be sure you can make a backup and restore on a fresh and clean installed Ubuntu 20.04 (or whatever you're currently using). Just make sure to restore to the same OS and Cloudron version and you should be fine. There are also guides what to do after a (planned) dist-upgrade so maybe check them out.