Uhoh - Disk space is full and cloudron dashboard/other apps not working

[ZeZaung]

Hello

I received an email that cloudron was not able to perform a backup. I did not pay too much heed until a few hours later, upon realizing I was unable to see the dashboard or make use of any of the apps I have downloaded.

We are personally quite puzzled how we've managed to fill 150GB with a few applications and are trying to figure it out. It's possible something was misconfigured, but not sure where the problem went wrong.

Where is the best way to start in diagnosing this fix? How should it best be approached so as not to interfere with cloudron while also balancing the need to SSH into the server and conduct some administrative tasks to get to the bottom of this? We can still SSH into server and static sites seem to be ok.

[girish]

@zezaung the usual problem here is that you are backing up to the same disk. is this the case? it could also be that you change the backup format/location and forgot to delete the old backups (cloudron currently does not delete old backups if you change the format or location).

I think you have to SSH to get things back up. Can you look into where all the space is getting used? I would start with the backup directory (default is /var/backups)

[girish]

Another popular disk issue is nextcloud. nextcloud does not delete files that you delete by default. So, you might have to clean out the nextcloud trash. See https://docs.cloudron.io/apps/nextcloud/#managing-deleted-files

[ZeZaung]

@girish We're running a VPS and do not recall having specified backups and.. well look at that.. after checking disk usage I've found a whopping 83GB backup directory that we're dealing with here.

[girish]

@zezaung the default backup location is /var/backups. You should ideally change this to some other place (ideally some external service or external disk). But for the moment, you can simply delete directories (save the latest 3 or so) which have a timestamp in it.

After that, https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full (specifically, make sure unbound is running and Service view shows all green)

[ZeZaung]

@girish Fantastic, I'll get right on this tomorrow! Cheers and thank you for the help, I will recontact in the event I struggle more. Very appreciated

[ZeZaung]

@girish
Hi girish,
For the last few months we have been unable to perform typical sysadmin tools such as cronjobs and rsync to wipe and replicate the /var/backups storage. We do not want a major cloud provider storage solution for this and the result is that we often have to perform the instructions to get cloudron back up and running after it runs out of space.

Right now we are not able to get access to the dashboard. I've followed all the instructions that I typically follow, regarding unbound, restarting all the other services, the anchor, etc.

I'm really hoping there might be a way to troubleshoot how to get the server up and running again. I've done the typical route of getting the dashboard back online, but sadly to no avail.

[ZeZaung]

UPDATE:
I re-ran the "/home/yellowtent/box/setup/start.sh" script and a wall of migrations were outputted after the command.

This then led to the dashboard changing to DNS configuration, which required access to my namecheap account. I re-inputted everything and was prompted to create a new user.

Now, after re-registering, thre are no apps running in the dashboard, however all the previousy used applications are in fact running, just it can not be seen by the dashboard. It's possibly due to it being a different user?

[girish]

@zezaung is the user you logged in, a Cloudron admin? You can make figure this out by checking if the menu items have Domains/Event Log/Email etc. A Cloudron admin always has access to all apps.

[girish]

Debugged this with @ZeZaung a bit. Somehow, it was possible to re-run cloudron-setup on the existing install and this reset the database even though the app containers and data were still there. I have made a fix which should hopefully mitigate the problem.