Solved Full disk + failed update -> no more accessible dashboard
I launched an upgrade to my Cloudron today and realized afterwards my disk was full (5G of log of a failed upgrade, 26G of Overlay2). I was not able to clean up the space taken by docker (despite running only two low usage app). By removing the 5G of logs I was able to re-run the upgrade and get my apps back up and running.
Nowhoever I can't access my Cloudron dashboard anymore. The URL at which I was getting my dashboard now redirect to one of my app with an invalid cert.
I wanted to disable the app in question but can't login via Cloudron CLI.
What can I do get back access to the dashboard? The DNS didn't change but it seems the routing in Cloudron was affected by the update.
Have you seen https://docs.cloudron.io/troubleshooting/ and tried to walk though that?
@nebulon I had followed the troubleshooting there to no avail. Deleting the root.key as mentionned in other posts was successfull but I still lost data in the mean time. Please improve how Cloudron deals with a disk getting full. That was subpar.
@vwvw sounds like we should soon at least get better notifications:
@msbt ah ok, so you just want better notifications . Can do that.
In 7.3.4 atleast, we started collecting du information in a cron job (every 24 hours).
timconsidine last edited by timconsidine
sounds like we should soon at least get better notifications:
notifications from Cloudron are great, but users are responsible for their own systems. I recommend STRONGLY that system health and disk usage are monitored independently. Yet another shout out for
ntfy.sheither in their hosted version or your own self-hosted version.
Accidents will always happen, but there is
no excusefor not proactively monitoring, e.g.
11/01/2023, 06:05 TC-MSI Wed 11 Jan 06:05:01 UTC 2023 *** System restart required *** Filesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-ubuntu--lv 109G 20G 84G 19% /
@timconsidine yeah, I don't (yet) have any such monitoring in place, but having previously suffered the woes of a full disk I now:
- have a massive drive with loads of free space, and
- very often manually check on that
timconsidine last edited by
@jdaviescoates I didn't mean to direct my post at you !
Just replied to and didn't remove the
Hope no offence caused
It was a general comment
Do automate the checking ... saves oodles of time and brainwaves.
@timconsidine no offense caused, I realised what had happened
I need to work out best way to get those notifications alerts set-up...
robi last edited by robi
@jdaviescoates you could set up a cron that gets usage info, and prints out a webpage you can monitor with Kuma for a keyword. When a threshold is met, you add/remove the keyword and fire off an alert via TG.
Similar for monitoring something with change detection app or n8n workflow.