Cloudron dash is stuck in reconnecting loop

[joseph]

@mpeterson0418 Can you run cloudron-support --troubleshoot on the server? That should give us a better idea. It looks like some networking issue. From what I can make out you are using Cloudflare. Have you tried disabling Cloudflare? Maybe cert renewal is failing? Did you change the API key of Cloudflare?

Also enabled SSH for cloudron-support if anyone wants to connect and check things out

Please don't do this automatically unless contacted you are asked by the support staff.

[mpeterson0418]

@joseph

Thanks for the feedback. Here is the output to the troubleshoot command -

michael@pcomp-webadmin:~$ sudo cloudron-support --troubleshoot
[sudo] password for michael: 
Vendor: VMware, Inc. Product: VMware Virtual Platform
Linux: 5.15.0-58-generic
Ubuntu: focal 20.04
Processor: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz x 4
RAM: 3983200KB
Disk: /dev/sda5        19G
[OK]	node version is correct
[OK]	docker is running
[OK]	MySQL is running
[OK]	nginx is running
sed: can't read /home/yellowtent/platformdata/nginx/applications/dashboard/my..conf: No such file or directory
Can't open  for reading, No such file or directory
140280590427456:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('','r')
140280590427456:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load certificate
[FAIL]	Certificate has expired. Certificate expired at 
	Please check /home/yellowtent/platformdata/logs/tasks/.log for last cert renewal logs
	Common issues include expiry of domain's API key OR incoming http port 80 not being open

For the record, I am able to locate the file on the OS -

Although I do see the cert error, I am not entirely sure that's the only problem here. Granted yes I got the below error to check the yellowtent directory for logs.... but when doing so I am seeing issues with MySQL:

2024-09-04T04:00:13.051Z box:server ==========================================
2024-09-04T04:00:13.052Z box:server            Cloudron 8.0.3  
2024-09-04T04:00:13.052Z box:server ==========================================
2024-09-04T04:00:13.052Z box:platform initialize: start platform
2024-09-04T04:00:13.073Z box:tasks stopAllTasks: stopping all tasks
2024-09-04T04:00:13.073Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
All tasks stopped
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client","nestedError":{"code":"ER_NOT_SUPPORTED_AUTH_MODE","errno":1251,"sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client","sqlState":"08004","fatal":true},"code":"ER_NOT_SUPPORTED_AUTH_MODE","sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client"}
    at main (/home/yellowtent/box/box.js:48:41)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

So this leads me to suspect that MySQL should be the point of focus first, as I do not see that other log referenced in the troubleshoot output

What are your thoughts here?

[nebulon]

Are you running Cloudron on a desktop ubuntu? This is not supported and will cause various side-effects.
You might be right that there are other issues then. For the mysql issue, have you updated or changed the system mysql? Note that Cloudron needs specific versions of some dependencies and just updating ubuntu will also break often.

If the cloudron-support script cannot open that nginx config file, can you check permissions of that?

[mpeterson0418]

@nebulon

I have not updated anything from the MySQL side of things, so it should be running on whatever platform was installed when setting up the docker. This only recently started happening. Things have been working fine on this box for the latter of a year now without any other issues

Also where exactly are you getting info that Ubuntu 20.04 is not supported? It's outlined in your docs that it is = https://docs.cloudron.io/guides/upgrade-ubuntu-20/

Below are the permissions I see on the file -

michael@pcomp-webadmin:~$ stat /home/yellowtent/platformdata/nginx/applications/dashboard/my.petersoncomputers.net.conf
  File: /home/yellowtent/platformdata/nginx/applications/dashboard/my.petersoncomputers.net.conf
  Size: 8973      	Blocks: 24         IO Block: 4096   regular file
Device: 805h/2053d	Inode: 1090691     Links: 1
Access: (0644/-rw-r--r--)  Uid: (  998/yellowtent)   Gid: (  998/yellowtent)
Access: 2024-09-04 08:26:16.248544678 -0400
Modify: 2024-08-02 03:02:40.451157726 -0400
Change: 2024-08-02 03:02:40.451157726 -0400
 Birth: -

[nebulon]

Ubuntu 20.04 is supported, I was more guessing from the screenshot of a desktop app, that the desktop flavor of Ubuntu 20.04 was installed and not the server edition. Things may work, but we do not test and support non-server versions.

So the error above kinda makes no sense then:

sed: can't read /home/yellowtent/platformdata/nginx/applications/dashboard/my..conf: No such file or directory

Just to be sure, those two dots in my..conf is how it is printed or did you remove the actual domain between the dots?

I guess it cannot read the domain as the mysql server connection does not work.
Can you run the following command via SSH successfully (not that root and password are literal like this, this is on purpose)?

mysql -uroot -ppassword box -e "select * from domains"

[mpeterson0418]

@nebulon

Yes that's just how the error is printed out. Full file name is my.petersoncomputers.net.conf and can be accessed without any issues

I left the installs as is, and there was a point months back where the InnoDB was corrupted so I had to basically rebuilt it. The documentation prompted me to use root and password to finish the recovery so I left that as is

This is also where the problem arises. When I try running with those creds it fails stating that access is denied -

michael@pcomp-webadmin:~$ mysql -uroot -ppassword -e "select * from domains"
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

What I'm wondering here is.... how do I possibly repair MySQL again so that I can gain access to it using the root account?

[nebulon]

Can you run /home/yellowtent/box/setup/start.sh via SSH on the server? This should reset the root password on the mysql instance.

[mpeterson0418]

@nebulon

Sorry for the delay here. Still getting access denied errors -

michael@pcomp-webadmin:~$ sudo /home/yellowtent/box/setup/start.sh
[sudo] password for michael:
2024-09-05T08:41:42 ==> start: Cloudron Start
media:x:500:
2024-09-05T08:41:42 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable apparmor
2024-09-05T08:41:43 ==> start: Ensuring directories
2024-09-05T08:41:43 ==> start: Configuring journald
2024-09-05T08:41:43 ==> start: Setting up unbound
/var/lib/unbound/root.key has content
success: the anchor is ok
2024-09-05T08:41:43 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable cron
Synchronizing state of rpcbind.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable rpcbind
2024-09-05T08:41:48 ==> start: Configuring sudoers
2024-09-05T08:41:48 ==> start: Configuring collectd
2024-09-05T08:41:48 ==> start: Configuring logrotate
2024-09-05T08:41:48 ==> start: Adding motd message for admins
2024-09-05T08:41:48 ==> start: Configuring nginx
2024-09-05T08:41:48 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
michael@pcomp-webadmin:~$

[nebulon]

Looks like the mysql server may not be listening correctly on localhost as expected. Basically you have to get mysqladmin -u root -ppassword password password to work to reset the password. If you have updated or changed the mysql server configs, please make sure password login for root is enabled.

[mpeterson0418]

@nebulon

Any recommendations for that..... aside from blowing away the MySQL config?

[joseph]

@mpeterson0418 mysqladmin -u root -ppassword password password is the command to reset back the root password to password (which is what Cloudron expects).

Did you upgrade ubuntu or change some mysql password? There is no code on Cloudron side to change the mysql password afaik.

[mpeterson0418]

@joseph

I guess that's what I was hoping perhaps you could help figure out for me. Working on this I think maybe I missed a step following the details on this doc where it left a temp password when trying to repair the instance = https://cloudron.io/documentation/troubleshooting/#recover-mysql

So hence why I'm curious about how I can recover things here if I'm not aware of what the root password is for the instance. I did collect copies of the DB before doing this so maybe if I uncomment out the .old extension I could get things going here

Thoughts?

[nebulon]

So that command reset the password to password you do not need any password currently set. Something else is wrong with your mysql server as such.

[mpeterson0418]

@nebulon

So look I'm completely aware of that. It's clearly a lockout issue of the root password. I'm just kind of at a loss on how to proceed at this point. If I figure something out.... I'll update you

Also no offense but we aren't doing anything productive here if all you are going to keep telling me is that it's a MySQL issue which I've stated numerous times that I'm focusing on right now. So unless you want to jump on my box and have a look, or outline some other steps I can follow to recover the root password, then let's just refrain from throwing more input here on this for now

What about just rebuilding the server from scratch? Is that a possibility without having access to the dashboard?

[nebulon]

Ah, maybe the confusion comes from the fact that the mysql database, where you hit the issue, is not the one for the apps (addons/services) but the mysql on the ubuntu host system. So generic mysql on ubuntu solutions like https://stackoverflow.com/questions/16556497/mysql-how-to-reset-or-change-the-mysql-root-password would work.

Cloudron just needs a mysql system where the root user has the password password once you reset the root user's password via mysql tooling, things should work if the database data as such is not corrupted.

[mpeterson0418]

@nebulon

That's solid thank you for that. Reviewed the same doc and got access restored to the MySQL app. Was able to rebuild the DB and get that working with the default creds

However I'm now hung again on the cert, and for the life of me can't figure out where to go on here to address that. Any thoughts? All the certs on my box are showing that it expires on 09/27/2024 so I'm totally lost on this one

I should also mention that the directory that it points me to for assessing the situation also has no files listed for troubleshooting this problem:

root@pcomp-webadmin:/home/yellowtent/platformdata/logs/tasks# ls | wc -l
0

[nebulon]

Actually on that domain, the cert looks fine to me from my browser. nginx appears fine also, however the box backend service is not working. Check the logs at /home/yellowtent/platformdata/logs/box.log

[mpeterson0418]

@nebulon

This is what I'm getting and it keeps repeating -

2024-09-06T15:04:36.771Z box:server ==========================================
2024-09-06T15:04:36.772Z box:server            Cloudron 8.0.3
2024-09-06T15:04:36.772Z box:server ==========================================
2024-09-06T15:04:36.772Z box:platform initialize: start platform
2024-09-06T15:04:36.792Z box:tasks stopAllTasks: stopping all tasks
2024-09-06T15:04:36.792Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
All tasks stopped
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client","nestedError":{"code":"ER_NOT_SUPPORTED_AUTH_MODE","errno":1251,"sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client","sqlState":"08004","fatal":true},"code":"ER_NOT_SUPPORTED_AUTH_MODE","sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client"}
    at main (/home/yellowtent/box/box.js:48:41)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Circled back to your previous update and ran this command -

/home/yellowtent/box/setup/start.sh

Now I have a screen asking me to setup cloudron again. I'm guessing I should be able to restore my instance with this interface right?

If so.... what options would you recommend for me here?

[joseph]

@mpeterson0418 You should not see the setup screen. It's almost like the database is entirely gone. I recommend contacting us at support@cloudron.io , so we can check what has happened.

[mpeterson0418]

@joseph

Thanks I just sent an email to support@cloudron.io

If that's the case.... is there something on the backend here for my account which would contain a backup I can recover from? I'm a bit confused on what's going on here

Aside from a MySQL DB recovery, this is not what I expected to see. Is my website completely hosed at this point?