Cloudron dash is stuck in reconnecting loop
-
@mpeterson0418 Can you run
cloudron-support --troubleshooton the server? That should give us a better idea. It looks like some networking issue. From what I can make out you are using Cloudflare. Have you tried disabling Cloudflare? Maybe cert renewal is failing? Did you change the API key of Cloudflare?Also enabled SSH for cloudron-support if anyone wants to connect and check things out
Please don't do this automatically unless contacted you are asked by the support staff.
Thanks for the feedback. Here is the output to the troubleshoot command -
michael@pcomp-webadmin:~$ sudo cloudron-support --troubleshoot [sudo] password for michael: Vendor: VMware, Inc. Product: VMware Virtual Platform Linux: 5.15.0-58-generic Ubuntu: focal 20.04 Processor: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz x 4 RAM: 3983200KB Disk: /dev/sda5 19G [OK] node version is correct [OK] docker is running [OK] MySQL is running [OK] nginx is running sed: can't read /home/yellowtent/platformdata/nginx/applications/dashboard/my..conf: No such file or directory Can't open for reading, No such file or directory 140280590427456:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('','r') 140280590427456:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76: unable to load certificate [FAIL] Certificate has expired. Certificate expired at Please check /home/yellowtent/platformdata/logs/tasks/.log for last cert renewal logs Common issues include expiry of domain's API key OR incoming http port 80 not being openFor the record, I am able to locate the file on the OS -
Although I do see the cert error, I am not entirely sure that's the only problem here. Granted yes I got the below error to check the yellowtent directory for logs.... but when doing so I am seeing issues with MySQL:
2024-09-04T04:00:13.051Z box:server ========================================== 2024-09-04T04:00:13.052Z box:server Cloudron 8.0.3 2024-09-04T04:00:13.052Z box:server ========================================== 2024-09-04T04:00:13.052Z box:platform initialize: start platform 2024-09-04T04:00:13.073Z box:tasks stopAllTasks: stopping all tasks 2024-09-04T04:00:13.073Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all All tasks stopped Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client","nestedError":{"code":"ER_NOT_SUPPORTED_AUTH_MODE","errno":1251,"sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client","sqlState":"08004","fatal":true},"code":"ER_NOT_SUPPORTED_AUTH_MODE","sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client"} at main (/home/yellowtent/box/box.js:48:41) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)So this leads me to suspect that MySQL should be the point of focus first, as I do not see that other log referenced in the troubleshoot output
What are your thoughts here?
-
Are you running Cloudron on a desktop ubuntu? This is not supported and will cause various side-effects.
You might be right that there are other issues then. For the mysql issue, have you updated or changed the system mysql? Note that Cloudron needs specific versions of some dependencies and just updating ubuntu will also break often.If the cloudron-support script cannot open that nginx config file, can you check permissions of that?
-
Are you running Cloudron on a desktop ubuntu? This is not supported and will cause various side-effects.
You might be right that there are other issues then. For the mysql issue, have you updated or changed the system mysql? Note that Cloudron needs specific versions of some dependencies and just updating ubuntu will also break often.If the cloudron-support script cannot open that nginx config file, can you check permissions of that?
I have not updated anything from the MySQL side of things, so it should be running on whatever platform was installed when setting up the docker. This only recently started happening. Things have been working fine on this box for the latter of a year now without any other issues
Also where exactly are you getting info that Ubuntu 20.04 is not supported? It's outlined in your docs that it is = https://docs.cloudron.io/guides/upgrade-ubuntu-20/
Below are the permissions I see on the file -
michael@pcomp-webadmin:~$ stat /home/yellowtent/platformdata/nginx/applications/dashboard/my.petersoncomputers.net.conf File: /home/yellowtent/platformdata/nginx/applications/dashboard/my.petersoncomputers.net.conf Size: 8973 Blocks: 24 IO Block: 4096 regular file Device: 805h/2053d Inode: 1090691 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 998/yellowtent) Gid: ( 998/yellowtent) Access: 2024-09-04 08:26:16.248544678 -0400 Modify: 2024-08-02 03:02:40.451157726 -0400 Change: 2024-08-02 03:02:40.451157726 -0400 Birth: - -
Ubuntu 20.04 is supported, I was more guessing from the screenshot of a desktop app, that the desktop flavor of Ubuntu 20.04 was installed and not the server edition. Things may work, but we do not test and support non-server versions.
So the error above kinda makes no sense then:
sed: can't read /home/yellowtent/platformdata/nginx/applications/dashboard/my..conf: No such file or directoryJust to be sure, those two dots in
my..confis how it is printed or did you remove the actual domain between the dots?I guess it cannot read the domain as the mysql server connection does not work.
Can you run the following command via SSH successfully (not thatrootandpasswordare literal like this, this is on purpose)?mysql -uroot -ppassword box -e "select * from domains" -
Ubuntu 20.04 is supported, I was more guessing from the screenshot of a desktop app, that the desktop flavor of Ubuntu 20.04 was installed and not the server edition. Things may work, but we do not test and support non-server versions.
So the error above kinda makes no sense then:
sed: can't read /home/yellowtent/platformdata/nginx/applications/dashboard/my..conf: No such file or directoryJust to be sure, those two dots in
my..confis how it is printed or did you remove the actual domain between the dots?I guess it cannot read the domain as the mysql server connection does not work.
Can you run the following command via SSH successfully (not thatrootandpasswordare literal like this, this is on purpose)?mysql -uroot -ppassword box -e "select * from domains"Yes that's just how the error is printed out. Full file name is my.petersoncomputers.net.conf and can be accessed without any issues
I left the installs as is, and there was a point months back where the InnoDB was corrupted so I had to basically rebuilt it. The documentation prompted me to use root and password to finish the recovery so I left that as is
This is also where the problem arises. When I try running with those creds it fails stating that access is denied -
michael@pcomp-webadmin:~$ mysql -uroot -ppassword -e "select * from domains" mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)What I'm wondering here is.... how do I possibly repair MySQL again so that I can gain access to it using the root account?
-
Can you run
/home/yellowtent/box/setup/start.shvia SSH on the server? This should reset the root password on the mysql instance.Sorry for the delay here. Still getting access denied errors -
michael@pcomp-webadmin:~$ sudo /home/yellowtent/box/setup/start.sh [sudo] password for michael: 2024-09-05T08:41:42 ==> start: Cloudron Start media:x:500: 2024-09-05T08:41:42 ==> start: Configuring docker Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable apparmor 2024-09-05T08:41:43 ==> start: Ensuring directories 2024-09-05T08:41:43 ==> start: Configuring journald 2024-09-05T08:41:43 ==> start: Setting up unbound /var/lib/unbound/root.key has content success: the anchor is ok 2024-09-05T08:41:43 ==> start: Adding systemd services Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable unbound Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable cron Synchronizing state of rpcbind.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable rpcbind 2024-09-05T08:41:48 ==> start: Configuring sudoers 2024-09-05T08:41:48 ==> start: Configuring collectd 2024-09-05T08:41:48 ==> start: Configuring logrotate 2024-09-05T08:41:48 ==> start: Adding motd message for admins 2024-09-05T08:41:48 ==> start: Configuring nginx 2024-09-05T08:41:48 ==> start: Starting mysql mysqladmin: [Warning] Using a password on the command line interface can be insecure. mysqladmin: connect to server at 'localhost' failed error: 'Access denied for user 'root'@'localhost' (using password: YES)' michael@pcomp-webadmin:~$ -
Looks like the mysql server may not be listening correctly on localhost as expected. Basically you have to get
mysqladmin -u root -ppassword password passwordto work to reset the password. If you have updated or changed the mysql server configs, please make sure password login for root is enabled. -
Looks like the mysql server may not be listening correctly on localhost as expected. Basically you have to get
mysqladmin -u root -ppassword password passwordto work to reset the password. If you have updated or changed the mysql server configs, please make sure password login for root is enabled.Any recommendations for that..... aside from blowing away the MySQL config?
-
@mpeterson0418
mysqladmin -u root -ppassword password passwordis the command to reset back the root password topassword(which is what Cloudron expects).Did you upgrade ubuntu or change some mysql password? There is no code on Cloudron side to change the mysql password afaik.
-
@mpeterson0418
mysqladmin -u root -ppassword password passwordis the command to reset back the root password topassword(which is what Cloudron expects).Did you upgrade ubuntu or change some mysql password? There is no code on Cloudron side to change the mysql password afaik.
I guess that's what I was hoping perhaps you could help figure out for me. Working on this I think maybe I missed a step following the details on this doc where it left a temp password when trying to repair the instance = https://cloudron.io/documentation/troubleshooting/#recover-mysql
So hence why I'm curious about how I can recover things here if I'm not aware of what the root password is for the instance. I did collect copies of the DB before doing this so maybe if I uncomment out the .old extension I could get things going here
Thoughts?
-
So that command reset the password to
passwordyou do not need any password currently set. Something else is wrong with your mysql server as such.So look I'm completely aware of that. It's clearly a lockout issue of the root password. I'm just kind of at a loss on how to proceed at this point. If I figure something out.... I'll update you
Also no offense but we aren't doing anything productive here if all you are going to keep telling me is that it's a MySQL issue which I've stated numerous times that I'm focusing on right now. So unless you want to jump on my box and have a look, or outline some other steps I can follow to recover the root password, then let's just refrain from throwing more input here on this for now
What about just rebuilding the server from scratch? Is that a possibility without having access to the dashboard?
-
Ah, maybe the confusion comes from the fact that the mysql database, where you hit the issue, is not the one for the apps (addons/services) but the mysql on the ubuntu host system. So generic mysql on ubuntu solutions like https://stackoverflow.com/questions/16556497/mysql-how-to-reset-or-change-the-mysql-root-password would work.
Cloudron just needs a mysql system where the
rootuser has the passwordpasswordonce you reset the root user's password via mysql tooling, things should work if the database data as such is not corrupted. -
Ah, maybe the confusion comes from the fact that the mysql database, where you hit the issue, is not the one for the apps (addons/services) but the mysql on the ubuntu host system. So generic mysql on ubuntu solutions like https://stackoverflow.com/questions/16556497/mysql-how-to-reset-or-change-the-mysql-root-password would work.
Cloudron just needs a mysql system where the
rootuser has the passwordpasswordonce you reset the root user's password via mysql tooling, things should work if the database data as such is not corrupted.That's solid thank you for that. Reviewed the same doc and got access restored to the MySQL app. Was able to rebuild the DB and get that working with the default creds
However I'm now hung again on the cert, and for the life of me can't figure out where to go on here to address that. Any thoughts? All the certs on my box are showing that it expires on 09/27/2024 so I'm totally lost on this one
I should also mention that the directory that it points me to for assessing the situation also has no files listed for troubleshooting this problem:
root@pcomp-webadmin:/home/yellowtent/platformdata/logs/tasks# ls | wc -l 0 -
Actually on that domain, the cert looks fine to me from my browser. nginx appears fine also, however the box backend service is not working. Check the logs at
/home/yellowtent/platformdata/logs/box.logThis is what I'm getting and it keeps repeating -
2024-09-06T15:04:36.771Z box:server ========================================== 2024-09-06T15:04:36.772Z box:server Cloudron 8.0.3 2024-09-06T15:04:36.772Z box:server ========================================== 2024-09-06T15:04:36.772Z box:platform initialize: start platform 2024-09-06T15:04:36.792Z box:tasks stopAllTasks: stopping all tasks 2024-09-06T15:04:36.792Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all All tasks stopped Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client","nestedError":{"code":"ER_NOT_SUPPORTED_AUTH_MODE","errno":1251,"sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client","sqlState":"08004","fatal":true},"code":"ER_NOT_SUPPORTED_AUTH_MODE","sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client"} at main (/home/yellowtent/box/box.js:48:41) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)Circled back to your previous update and ran this command -
/home/yellowtent/box/setup/start.shNow I have a screen asking me to setup cloudron again. I'm guessing I should be able to restore my instance with this interface right?
If so.... what options would you recommend for me here?
-
@mpeterson0418 You should not see the setup screen. It's almost like the database is entirely gone. I recommend contacting us at support@cloudron.io , so we can check what has happened.
-
@mpeterson0418 You should not see the setup screen. It's almost like the database is entirely gone. I recommend contacting us at support@cloudron.io , so we can check what has happened.
Thanks I just sent an email to support@cloudron.io
If that's the case.... is there something on the backend here for my account which would contain a backup I can recover from? I'm a bit confused on what's going on here
Aside from a MySQL DB recovery, this is not what I expected to see. Is my website completely hosed at this point?
-
@mpeterson0418 the app database is totally separate from cloudron's own database. But we have to see what is the state of the server to assess the situation.
