Cloudron dash is stuck in reconnecting loop

[nebulon]

Can you run /home/yellowtent/box/setup/start.sh via SSH on the server? This should reset the root password on the mysql instance.

[mpeterson0418]

@nebulon

Sorry for the delay here. Still getting access denied errors -

michael@pcomp-webadmin:~$ sudo /home/yellowtent/box/setup/start.sh
[sudo] password for michael:
2024-09-05T08:41:42 ==> start: Cloudron Start
media:x:500:
2024-09-05T08:41:42 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable apparmor
2024-09-05T08:41:43 ==> start: Ensuring directories
2024-09-05T08:41:43 ==> start: Configuring journald
2024-09-05T08:41:43 ==> start: Setting up unbound
/var/lib/unbound/root.key has content
success: the anchor is ok
2024-09-05T08:41:43 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable cron
Synchronizing state of rpcbind.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable rpcbind
2024-09-05T08:41:48 ==> start: Configuring sudoers
2024-09-05T08:41:48 ==> start: Configuring collectd
2024-09-05T08:41:48 ==> start: Configuring logrotate
2024-09-05T08:41:48 ==> start: Adding motd message for admins
2024-09-05T08:41:48 ==> start: Configuring nginx
2024-09-05T08:41:48 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
michael@pcomp-webadmin:~$

[nebulon]

Looks like the mysql server may not be listening correctly on localhost as expected. Basically you have to get mysqladmin -u root -ppassword password password to work to reset the password. If you have updated or changed the mysql server configs, please make sure password login for root is enabled.

[mpeterson0418]

@nebulon

Any recommendations for that..... aside from blowing away the MySQL config?

[joseph]

@mpeterson0418 mysqladmin -u root -ppassword password password is the command to reset back the root password to password (which is what Cloudron expects).

Did you upgrade ubuntu or change some mysql password? There is no code on Cloudron side to change the mysql password afaik.

[mpeterson0418]

@joseph

I guess that's what I was hoping perhaps you could help figure out for me. Working on this I think maybe I missed a step following the details on this doc where it left a temp password when trying to repair the instance = https://cloudron.io/documentation/troubleshooting/#recover-mysql

So hence why I'm curious about how I can recover things here if I'm not aware of what the root password is for the instance. I did collect copies of the DB before doing this so maybe if I uncomment out the .old extension I could get things going here

Thoughts?

[nebulon]

So that command reset the password to password you do not need any password currently set. Something else is wrong with your mysql server as such.

[mpeterson0418]

@nebulon

So look I'm completely aware of that. It's clearly a lockout issue of the root password. I'm just kind of at a loss on how to proceed at this point. If I figure something out.... I'll update you

Also no offense but we aren't doing anything productive here if all you are going to keep telling me is that it's a MySQL issue which I've stated numerous times that I'm focusing on right now. So unless you want to jump on my box and have a look, or outline some other steps I can follow to recover the root password, then let's just refrain from throwing more input here on this for now

What about just rebuilding the server from scratch? Is that a possibility without having access to the dashboard?

[nebulon]

Ah, maybe the confusion comes from the fact that the mysql database, where you hit the issue, is not the one for the apps (addons/services) but the mysql on the ubuntu host system. So generic mysql on ubuntu solutions like https://stackoverflow.com/questions/16556497/mysql-how-to-reset-or-change-the-mysql-root-password would work.

Cloudron just needs a mysql system where the root user has the password password once you reset the root user's password via mysql tooling, things should work if the database data as such is not corrupted.

[mpeterson0418]

@nebulon

That's solid thank you for that. Reviewed the same doc and got access restored to the MySQL app. Was able to rebuild the DB and get that working with the default creds

However I'm now hung again on the cert, and for the life of me can't figure out where to go on here to address that. Any thoughts? All the certs on my box are showing that it expires on 09/27/2024 so I'm totally lost on this one

I should also mention that the directory that it points me to for assessing the situation also has no files listed for troubleshooting this problem:

root@pcomp-webadmin:/home/yellowtent/platformdata/logs/tasks# ls | wc -l
0

[nebulon]

Actually on that domain, the cert looks fine to me from my browser. nginx appears fine also, however the box backend service is not working. Check the logs at /home/yellowtent/platformdata/logs/box.log

[mpeterson0418]

@nebulon

This is what I'm getting and it keeps repeating -

2024-09-06T15:04:36.771Z box:server ==========================================
2024-09-06T15:04:36.772Z box:server            Cloudron 8.0.3
2024-09-06T15:04:36.772Z box:server ==========================================
2024-09-06T15:04:36.772Z box:platform initialize: start platform
2024-09-06T15:04:36.792Z box:tasks stopAllTasks: stopping all tasks
2024-09-06T15:04:36.792Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
All tasks stopped
Error: Error starting server: {"name":"BoxError","reason":"Database Error","details":{},"message":"ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client","nestedError":{"code":"ER_NOT_SUPPORTED_AUTH_MODE","errno":1251,"sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client","sqlState":"08004","fatal":true},"code":"ER_NOT_SUPPORTED_AUTH_MODE","sqlMessage":"Client does not support authentication protocol requested by server; consider upgrading MySQL client"}
    at main (/home/yellowtent/box/box.js:48:41)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Circled back to your previous update and ran this command -

/home/yellowtent/box/setup/start.sh

Now I have a screen asking me to setup cloudron again. I'm guessing I should be able to restore my instance with this interface right?

If so.... what options would you recommend for me here?

[joseph]

@mpeterson0418 You should not see the setup screen. It's almost like the database is entirely gone. I recommend contacting us at support@cloudron.io , so we can check what has happened.

[mpeterson0418]

@joseph

Thanks I just sent an email to support@cloudron.io

If that's the case.... is there something on the backend here for my account which would contain a backup I can recover from? I'm a bit confused on what's going on here

Aside from a MySQL DB recovery, this is not what I expected to see. Is my website completely hosed at this point?

[joseph]

@mpeterson0418 the app database is totally separate from cloudron's own database. But we have to see what is the state of the server to assess the situation.

[mpeterson0418]

@joseph

Ok well that makes more sense.... I guess lol

To be honest from a website perspective I didn't touch a thing on here. So whatever is going on at this point I'd be very curious to know. Honestly only other thing I could imagine here would be some potential disk corruption of some sort..... but hopefully you can give some better insight on that after looking things over

[mpeterson0418]

@joseph

FYI the SSH access has been fixed. You should be able to remotely connect to the server for review

[joseph]

@mpeterson0418 so the issue was that mysql, docker, containerd were all corrupt. I am not sure what happened to the disk or the server. I restore mysql from a previous backup. container/docker (i.e /var/lib/docker and /var/lib/containerd) had to be deleted entirely and re-created .

[mpeterson0418]

@joseph

Thank you for all your assistance. Everything looks great now!

Send you a couple other questions in the support case I raised. If you don't mind following up with me on it.... and then I believe we can close this one out

[joseph]

@mpeterson0418 I found that the root file system was either corrupt or "reset" . I don't know and unable to guess what happened to the disk (maybe power outage?), but I can explain what I did to recover.

• I found host MySQL to be empty. The box database had tables but was empty. The fact that it had tables meant that migration scripts were run on this . Maybe /home/yellowtent/box/setup/start.sh was run at some point on an empty (i.e no tables even) database.

  • Usually, the last well known mysql dump is located /home/yellowtent/boxdata/box/box.mysqldump . One can also find the same inside the latest backup (inside the box.tar.gz) .
  • To "restore", mysql -uroot -ppassword < /home/yellowtent/boxdata/box/box.mysqldump

• Then I found, docker won't start. In Cloudron, containers are throw away infrastructure since data is stored outside the containers. I tried cloudron-support --recreate-docker but this did not work. On investigating further, it seems /var/lib/containerd was also corrupt . Corrupt here means no docker commands work. So, I removed that directory as well. After that, cloudron-support --recreate-docker worked.

The best way to migrate in Cloudron is not to do the above but instead just restore from backups. See https://docs.cloudron.io/backups/#move-cloudron-to-another-server .