SOLVED Cannot Access Cloudron
I have recently cloned my VPS to a more powerful server due to one or more apps running out of memory. After the migration, I am no longer able to access my Cloudron admin dashboard at my.<domain>.
I am almost certain that the problem is the memory limits imposed on the app(s), but I cannot access the admin dashboard to increase the memory limits. The Cloudron CLI interface also fails to log in, throwing error 502.
My apps are showing "This app is currently not responding. Please try refreshing the page in a few minutes."
Is there a way to increase the memory size of Cloudron/Cloudron admin dashboard, so I can regain access to my server? I have critical data in my apps and the backups from my old server are no longer accessible. I have backups of my apps before the memory limit was reached, but I cannot access those, either, due to lack of access to admin dashboard.
@inference If you migrated to a new server, did you change the DNS entry of my.<domain> to point to the new location manually? That's most likely the issue. Otherwise, can you give check
systemctl status box?
systemctl status boxshows that the process/service is running.
Yes, I manually changed the DNS record to point to my new server. I changed both the my.<domain> and <app>.<domain> (my apps run on their own subdomains).
When I try to connect to my.<domain>, it states that Cloudron is offline and is reconnecting; this remains on screen, permanently.
@inference Can you also check
systemctl status nginx? You can also check
/home/yellowtent/platformdata/logs/box.log. The last few lines should say the apps are alive / dead.
systemctl status nginxshows that the process/service is also running.
The log file you mentioned shows:
2021-10-08T01:56:37.344Z box:server ========================================== 2021-10-08T01:56:37.345Z box:server Cloudron 6.3.6 2021-10-08T01:56:37.345Z box:server ========================================== 2021-10-08T01:56:37.397Z box:settings initCache: pre-load settings 2021-10-08T01:56:37.429Z box:tasks stopTask: stopping all tasks 2021-10-08T01:56:37.430Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all 2021-10-08T01:56:37.462Z box:dockerproxy startDockerProxy: started proxy on port 3003
@inference it is supposed to "start up" after that. It seems to be stuck for some reason. You can check if
systemctl restart boxmakes it go ahead. Otherwise, do you see lots of containers running when you do
How did you migrate btw? With a backup or using VM image / snapshot & restore ?
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json: dial unix /var/run/docker.sock: connect: permission denied.
When I run the command as root, it hangs without any output.
I migrated by cloning my previous VPS (2 shared CPU cores, 4 GiB RAM). My previous VPS is now unavailable. New VPS has 4 shared CPU cores, 8 GiB RAM.
@inference said in Cannot Access Cloudron:
When I run the command as root, it hangs without any output.
Ah ok, this is the root cause then. Cloudron is trying to check status on services on start up (the services are docker containers) and looks like docker is not responding. Does
systemctl restart dockerhelp ? You can also try
jounalctl -u dockerto see docker logs.
systemctl restart dockeroutputs
A dependency job for docker.service failed. See 'journalctl -xe' for details.
journalctl -u dockeroutputs
Oct 08 02:04:30 localhost systemd: docker.service: Job docker.service/start failed with result 'dependency'. Oct 08 02:04:35 localhost systemd: Dependency failed for Docker Application Container Engine.
@inference @girish After searching the internet, I tried the
/usr/bin/dockerd -H unix://command; it outputs (IDs are container ID and network ID, which I have removed due to privacy/security concerns)
ERRO[2021-10-08T03:30:06.719778335+01:00] failed to start container container=<container ID> error="network <network ID> not found
@inference mm, not sure what that message means. you shouldn't have to run dockerd manually. maybe you can try a server reboot for good measure to see if that helps run docker again. Is there anything in
@girish I restarted the server, then restarted both
docker-socket, but no change,
-- The unit UNIT has successfully entered the 'dead' state. Oct 08 16:32:09 localhost systemd: var-lib-docker-overlay2-383347feb2d94ef4018a64c1ec85641766201d0e624b311fa83d82c652355b05-merged.mount: Succeeded. -- Subject: Unit succeeded -- Defined-By: systemd -- Support: http://www.ubuntu.com/support
and other similar messages.
I noticed that the Ubuntu MOTD has a connection error, too, but my IP address and domain is reachable.
@inference looks like something more fundamental is wrong with this instance, since you mentioned initially that you had previous Cloudron installation on a small instance, maybe you can do a fresh clone of this and see if this error is the same. Also which VPS provider is this? Regarding the motd message, are there any other outbound connection issues you have seen?
@nebulon My VPS provider is Linode. I think the issue is 1 of 2 things; either a network issue due to the cloned VPS having a different IP address, or a memory issue of the Cloudron apps, which ran out of memory and I have not been able to increase the memory limits due to lack of access to the Cloudron admin dashboard.
I have not noticed any other outbound issues.
Is there a way to recover my backups from Cloudron, so I could create a new VPS, then restore from an earlier backup before the issues occurred?
@inference the issue is docker itself is not running (for some reason). app memory is not an issue because apps are run as containers using docker.
What I can suggest is:
- spin up another new VPS using the VM image you have. check if docker runs then
- for backups, do you remember how you configured backups in Cloudron ? By default, the backups are in
/var/backupsof the server. Do you see many timestamped directories in that folder?