Full disk + failed update -> no more accessible dashboard

[vwvw]

Dear Community,

I launched an upgrade to my Cloudron today and realized afterwards my disk was full (5G of log of a failed upgrade, 26G of Overlay2). I was not able to clean up the space taken by docker (despite running only two low usage app). By removing the 5G of logs I was able to re-run the upgrade and get my apps back up and running.

Nowhoever I can't access my Cloudron dashboard anymore. The URL at which I was getting my dashboard now redirect to one of my app with an invalid cert.

I wanted to disable the app in question but can't login via Cloudron CLI.

What can I do get back access to the dashboard? The DNS didn't change but it seems the routing in Cloudron was affected by the update.

[nebulon]

Have you seen https://docs.cloudron.io/troubleshooting/ and tried to walk though that?

[vwvw]

@nebulon I had followed the troubleshooting there to no avail. Deleting the root.key as mentionned in other posts was successfull but I still lost data in the mean time. Please improve how Cloudron deals with a disk getting full. That was subpar.

[jdaviescoates]

@vwvw sounds like we should soon at least get better notifications:

@girish said in Expose disk usage / free space to API (preferably with a readonly token):

@msbt ah ok, so you just want better notifications . Can do that.

In 7.3.4 atleast, we started collecting du information in a cron job (every 24 hours).

[timconsidine]

@jdaviescoates said in Full disk + failed update -> no more accessible dashboard:

sounds like we should soon at least get better notifications:

notifications from Cloudron are great, but users are responsible for their own systems. I recommend STRONGLY that system health and disk usage are monitored independently. Yet another shout out for ntfy.sh either in their hosted version or your own self-hosted version.

Accidents will always happen, but there is no excuse for not proactively monitoring, e.g.

11/01/2023, 06:05

TC-MSI
Wed 11 Jan 06:05:01 UTC 2023
*** System restart required ***
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 109G 20G 84G 19% /

[jdaviescoates]

@timconsidine yeah, I don't (yet) have any such monitoring in place, but having previously suffered the woes of a full disk I now:

1. have a massive drive with loads of free space, and
2. very often manually check on that

[timconsidine]

@jdaviescoates I didn't mean to direct my post at you !
Just replied to and didn't remove the @
Hope no offence caused
It was a general comment
Do automate the checking ... saves oodles of time and brainwaves.

[jdaviescoates]

@timconsidine no offense caused, I realised what had happened

I need to work out best way to get those notifications alerts set-up...

[robi]

@jdaviescoates you could set up a cron that gets usage info, and prints out a webpage you can monitor with Kuma for a keyword. When a threshold is met, you add/remove the keyword and fire off an alert via TG.

Similar for monitoring something with change detection app or n8n workflow.