Cloudron dash is stuck in reconnecting loop
-
Sorry for the delay here. Still getting access denied errors -
michael@pcomp-webadmin:~$ sudo /home/yellowtent/box/setup/start.sh [sudo] password for michael: 2024-09-05T08:41:42 ==> start: Cloudron Start media:x:500: 2024-09-05T08:41:42 ==> start: Configuring docker Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable apparmor 2024-09-05T08:41:43 ==> start: Ensuring directories 2024-09-05T08:41:43 ==> start: Configuring journald 2024-09-05T08:41:43 ==> start: Setting up unbound /var/lib/unbound/root.key has content success: the anchor is ok 2024-09-05T08:41:43 ==> start: Adding systemd services Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable unbound Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable cron Synchronizing state of rpcbind.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install disable rpcbind 2024-09-05T08:41:48 ==> start: Configuring sudoers 2024-09-05T08:41:48 ==> start: Configuring collectd 2024-09-05T08:41:48 ==> start: Configuring logrotate 2024-09-05T08:41:48 ==> start: Adding motd message for admins 2024-09-05T08:41:48 ==> start: Configuring nginx 2024-09-05T08:41:48 ==> start: Starting mysql mysqladmin: [Warning] Using a password on the command line interface can be insecure. mysqladmin: connect to server at 'localhost' failed error: 'Access denied for user 'root'@'localhost' (using password: YES)' michael@pcomp-webadmin:~$ -
Looks like the mysql server may not be listening correctly on localhost as expected. Basically you have to get
mysqladmin -u root -ppassword password passwordto work to reset the password. If you have updated or changed the mysql server configs, please make sure password login for root is enabled. -
@mpeterson0418
mysqladmin -u root -ppassword password passwordis the command to reset back the root password topassword(which is what Cloudron expects).Did you upgrade ubuntu or change some mysql password? There is no code on Cloudron side to change the mysql password afaik.
-
I guess that's what I was hoping perhaps you could help figure out for me. Working on this I think maybe I missed a step following the details on this doc where it left a temp password when trying to repair the instance = https://cloudron.io/documentation/troubleshooting/#recover-mysql
So hence why I'm curious about how I can recover things here if I'm not aware of what the root password is for the instance. I did collect copies of the DB before doing this so maybe if I uncomment out the .old extension I could get things going here
Thoughts?
-
So look I'm completely aware of that. It's clearly a lockout issue of the root password. I'm just kind of at a loss on how to proceed at this point. If I figure something out.... I'll update you
Also no offense but we aren't doing anything productive here if all you are going to keep telling me is that it's a MySQL issue which I've stated numerous times that I'm focusing on right now. So unless you want to jump on my box and have a look, or outline some other steps I can follow to recover the root password, then let's just refrain from throwing more input here on this for now
What about just rebuilding the server from scratch? Is that a possibility without having access to the dashboard?
-
Ah, maybe the confusion comes from the fact that the mysql database, where you hit the issue, is not the one for the apps (addons/services) but the mysql on the ubuntu host system. So generic mysql on ubuntu solutions like https://stackoverflow.com/questions/16556497/mysql-how-to-reset-or-change-the-mysql-root-password would work.
Cloudron just needs a mysql system where the
rootuser has the passwordpasswordonce you reset the root user's password via mysql tooling, things should work if the database data as such is not corrupted. -
That's solid thank you for that. Reviewed the same doc and got access restored to the MySQL app. Was able to rebuild the DB and get that working with the default creds
However I'm now hung again on the cert, and for the life of me can't figure out where to go on here to address that. Any thoughts? All the certs on my box are showing that it expires on 09/27/2024 so I'm totally lost on this one
I should also mention that the directory that it points me to for assessing the situation also has no files listed for troubleshooting this problem:
root@pcomp-webadmin:/home/yellowtent/platformdata/logs/tasks# ls | wc -l 0 -
This is what I'm getting and it keeps repeating -
2024-09-06T15:04:36.771Z box:server ========================================== 2024-09-06T15:04:36.772Z box:server Cloudron 8.0.3 2024-09-06T15:04:36.772Z box:server ========================================== 2024-09-06T15:04:36.772Z box:platform initialize: start platform 2024-09-06T15:04:36.792Z box:tasks stopAllTasks: stopping all tasks 2024-09-06T15:04:36.792Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all All tasks stopped Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client","nestedError":{"code":"ER_NOT_SUPPORTED_AUTH_MODE","errno":1251,"sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client","sqlState":"08004","fatal":true},"code":"ER_NOT_SUPPORTED_AUTH_MODE","sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client"} at main (/home/yellowtent/box/box.js:48:41) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)Circled back to your previous update and ran this command -
/home/yellowtent/box/setup/start.shNow I have a screen asking me to setup cloudron again. I'm guessing I should be able to restore my instance with this interface right?
If so.... what options would you recommend for me here?
-
@mpeterson0418 You should not see the setup screen. It's almost like the database is entirely gone. I recommend contacting us at support@cloudron.io , so we can check what has happened.
-
Thanks I just sent an email to support@cloudron.io
If that's the case.... is there something on the backend here for my account which would contain a backup I can recover from? I'm a bit confused on what's going on here
Aside from a MySQL DB recovery, this is not what I expected to see. Is my website completely hosed at this point?
-
@mpeterson0418 the app database is totally separate from cloudron's own database. But we have to see what is the state of the server to assess the situation.
-
Ok well that makes more sense.... I guess lol
To be honest from a website perspective I didn't touch a thing on here. So whatever is going on at this point I'd be very curious to know. Honestly only other thing I could imagine here would be some potential disk corruption of some sort..... but hopefully you can give some better insight on that after looking things over
-
@mpeterson0418 so the issue was that mysql, docker, containerd were all corrupt. I am not sure what happened to the disk or the server. I restore mysql from a previous backup. container/docker (i.e /var/lib/docker and /var/lib/containerd) had to be deleted entirely and re-created .
-
J joseph has marked this topic as solved on
-
Thank you for all your assistance. Everything looks great now!
Send you a couple other questions in the support case I raised. If you don't mind following up with me on it.... and then I believe we can close this one out
-
@mpeterson0418 I found that the root file system was either corrupt or "reset" . I don't know and unable to guess what happened to the disk (maybe power outage?), but I can explain what I did to recover.
-
I found host MySQL to be empty. The
boxdatabase had tables but was empty. The fact that it had tables meant that migration scripts were run on this . Maybe/home/yellowtent/box/setup/start.shwas run at some point on an empty (i.e no tables even) database.- Usually, the last well known mysql dump is located
/home/yellowtent/boxdata/box/box.mysqldump. One can also find the same inside the latest backup (inside the box.tar.gz) . - To "restore",
mysql -uroot -ppassword < /home/yellowtent/boxdata/box/box.mysqldump
- Usually, the last well known mysql dump is located
-
Then I found, docker won't start. In Cloudron, containers are throw away infrastructure since data is stored outside the containers. I tried
cloudron-support --recreate-dockerbut this did not work. On investigating further, it seems/var/lib/containerdwas also corrupt . Corrupt here means no docker commands work. So, I removed that directory as well. After that,cloudron-support --recreate-dockerworked.
The best way to migrate in Cloudron is not to do the above but instead just restore from backups. See https://docs.cloudron.io/backups/#move-cloudron-to-another-server .
-
