Cloudron Admin Panel Down

[saint]

Hi there,

I just ran into an issue where I tried to install a new app (Joplin server), and the installation failed. I don't think it is due to the package, but likely more due to disk space?

The error message I got was:

Task Error
If a configuration, update, restore or backup action resulted in an error, you can retry the task.
An error occurred during the install operation: FileSystem Error: Error creating directory: ENOENT: no such file or directory, mkdir '/home/yellowtent/appsdata/e2b71a70-2d9b-48ed-98c8-aaf99534527c'

Some observations and steps I've attempted:

• Tried all of the below troubleshooting steps

- systemctl status and restart nginx
- system systemctl status and restart docker
- systemctl status and restart box
- Ran /home/yellowtent/box/setup/start.sh

- Ran apt-get remove `dpkg --list 'linux-image*' |grep ^ii | awk '{print $2}'\ | grep -v \`uname -r\``

The above were fine, and showed all were active or didn't give errors.

• I have noticed that I am running out of space on the server and/or;
• The MySQL database seems to be down when i did a systemctl status mysql (see below)

• I have also noticed Unbound seems to be down as well when i did a systemctl status unbound, and there is an error that says "Could not fflush" ... "No space left on device."

A df -h shows the below, and it looks like /dev/vda2 is extremely full, but I'm not sure what to do about that.

root@server:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           394M  2.0M  392M   1% /run
/dev/vda2        99G   96G     0 100% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/loop0       56M   56M     0 100% /snap/core18/2714
/dev/loop1       56M   56M     0 100% /snap/core18/2721
/dev/loop3       64M   64M     0 100% /snap/core20/1852
/dev/loop2      143M  143M     0 100% /snap/lxd/24483
/dev/loop4       73M   73M     0 100% /snap/core22/583
/dev/loop5       64M   64M     0 100% /snap/core20/1828
/dev/loop6       73M   73M     0 100% /snap/core22/607
/dev/loop7      163M  163M     0 100% /snap/lxd/24643
/dev/loop9       50M   50M     0 100% /snap/snapd/18357
/dev/loop8       50M   50M     0 100% /snap/snapd/18596
tmpfs           394M     0  394M   0% /run/user/0
overlay          99G   96G     0 100% /var/lib/docker/overlay2/5622e5e4a0509f7435f120de894243aa84fd7f0f217972c8a66897183c10fa60/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/9806e31f0d815938028a3871909ea8350c735bc62922c69ec08b540f4de4de61/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/842c348622671b853ec1900f1cbfca9077fc0d582823b895f9ed6f9b88542808/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/a2bd4fadd972a4cb79a70cd024a3df13a9786a0729d0a238b352aa187cdd4f9d/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/bb6073e44eeaf106990e1c1b1076170c2f8dfe6445c43b20df8ce4d61a530540/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/8272b642f023dc33a2466c81f2d478af445628138921d9928fe706d425d50c51/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/d1964be7093c93ff6933c2833078d7bab481eda8eb3778a5079b88bbc8591227/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/f38dbd9d38069c35bc86c352e4e28ae76114b4162dd1cc4174bc53a72ce7c3d6/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/91c649919a97a56c4c7b639fe957187d1b7c9f6142cde887b7ed8679cd1aec00/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/5dee1219784bd35166469b7909d04369b99a2b6e08bd54b2afa2900bdb14ac78/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/7c25c359f68d5f4a8fce3fdf566ce0f444a2afd10f4873795e2c1d14176c01ea/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/61552af60fe61456f7e0a7a937300b300aaac22fd8ec0b0cbbd92b6d5e01194e/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/d02b8cfdb5e8a9ff1517d44773b3f3a8eaa1add5f62437a9e2c0f8f34bee5f38/merged
overlay          99G   96G     0 100% /var/lib/docker/overlay2/5faa2b5cd2fb958e5966e181d09bde2b2c0e3517e50a026eb9e6ed94fcc97162/merged

My questions/Asks:

• Is this a MySQL issue and should I attempt to recover MySQL? I'm hoping not to mess with the DB unless I absolutely have to.

What about deleting previous backups?

• Delete Older Backups - Is there a way to delete some of the older version backups to free up some space? And if so, where would I go to do that since I can't access the Cloudron Admin Panel.
• Install/Update Apps on Staging - Could we have an update path where the upgrade happens on a STAGING instance for the apps, and when there's no issue, can be pushed out to PROD? This could help minimize workflow issues in the future since right now, I'm not able to use any of the apps since Cloudron AP is down.

Appreciate any assistance that could help bring Cloudron AP back online. Thank you.

[girish]

@saint Where are your backups? Are they on the same server as Cloudron ? If so, I would start with deleting some backups in /var/backups. These are timestamped directories, you can delete the old ones, I guess. But do keep some around, in case, you are not able to recover!

https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full has the steps for recovery.

[saint]

@girish Yes, they are on the same server as Cloudron.

Based off what I can see below, everything in backup totals up to only 2.0MB? That seems strange... I know my Nextcloud instance has a ton of photos in it, so not sure if I'm just reading it incorrectly.

Sounds like I can delete between *.3 - *6 as *6.gz is the furthest one away. Is there any file below that I should not remove?

root@server:/var/backups# ls -alhS
total 2.0M
-rw-r--r--  1 root root 705K Apr  4 06:08 dpkg.status.0
-rw-r--r--  1 root root 175K Apr  4 06:08 dpkg.status.1.gz
-rw-r--r--  1 root root 175K Apr  4 06:08 dpkg.status.2.gz
-rw-r--r--  1 root root 175K Mar 28 06:11 dpkg.status.3.gz
-rw-r--r--  1 root root 175K Mar 28 06:11 dpkg.status.4.gz
-rw-r--r--  1 root root 175K Mar 28 06:11 dpkg.status.5.gz
-rw-r--r--  1 root root 175K Mar 28 06:11 dpkg.status.6.gz
-rw-r--r--  1 root root  50K Mar 22 06:25 alternatives.tar.0
-rw-r--r--  1 root root  38K Mar 28 06:11 apt.extended_states.0
-rw-r--r--  1 root root 4.2K Mar  3 06:28 apt.extended_states.1.gz
-rw-r--r--  1 root root 4.2K Feb 10 06:11 apt.extended_states.2.gz
-rw-r--r--  1 root root 4.2K Jan 14 06:22 apt.extended_states.3.gz
-rw-r--r--  1 root root 4.2K Jan  8 06:04 apt.extended_states.4.gz
-rw-r--r--  1 root root 4.2K Dec  3 06:03 apt.extended_states.5.gz
-rw-r--r--  1 root root 4.2K Nov 16 06:05 apt.extended_states.6.gz
drwxrwxrwx  2 root root 4.0K Apr  6 06:25 .
drwxr-xr-x 14 root root 4.0K Sep  4  2022 ..
-rw-r--r--  1 root root 2.6K Mar  1 06:25 alternatives.tar.1.gz
-rw-r--r--  1 root root 2.6K Jan 31 06:25 alternatives.tar.2.gz
-rw-r--r--  1 root root 2.6K Oct 27 06:25 alternatives.tar.5.gz
-rw-r--r--  1 root root 2.6K Jan 14 06:25 alternatives.tar.4.gz
-rw-r--r--  1 root root 2.6K Jan 27 06:25 alternatives.tar.3.gz
-rw-r--r--  1 root root 2.6K Sep 20  2022 alternatives.tar.6.gz
-rw-r--r--  1 root root  268 Jan 11  2021 dpkg.diversions.0
-rw-r--r--  1 root root  139 Jan 11  2021 dpkg.diversions.1.gz
-rw-r--r--  1 root root  139 Jan 11  2021 dpkg.diversions.2.gz
-rw-r--r--  1 root root  139 Jan 11  2021 dpkg.diversions.3.gz
-rw-r--r--  1 root root  139 Jan 11  2021 dpkg.diversions.4.gz
-rw-r--r--  1 root root  139 Jan 11  2021 dpkg.diversions.5.gz
-rw-r--r--  1 root root  139 Jan 11  2021 dpkg.diversions.6.gz
-rw-r--r--  1 root root  120 Apr 23  2020 dpkg.statoverride.1.gz
-rw-r--r--  1 root root  120 Apr 23  2020 dpkg.statoverride.2.gz
-rw-r--r--  1 root root  120 Apr 23  2020 dpkg.statoverride.3.gz
-rw-r--r--  1 root root  120 Apr 23  2020 dpkg.statoverride.4.gz
-rw-r--r--  1 root root  120 Apr 23  2020 dpkg.statoverride.5.gz
-rw-r--r--  1 root root  120 Apr 23  2020 dpkg.statoverride.6.gz
-rw-r--r--  1 root root  100 Apr 23  2020 dpkg.statoverride.0

[girish]

@saint it doesn't looks like the backups are in /var/backups. It will contain a snapshot directory among other things. Can you check in Backups view where it's backing up?

[girish]

@saint the backups will look like this:

root@my:/var/backups# ls -l
total 2200
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  8 07:00 2023-01-08-070000-649
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  9 07:00 2023-01-09-070000-722
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  9 09:29 2023-01-09-092923-525
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  9 09:41 2023-01-09-094131-393
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  9 13:38 2023-01-09-133856-696
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  9 15:12 2023-01-09-151231-272
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan  9 15:42 2023-01-09-154249-626
....
drwxr-xr-x 2 yellowtent yellowtent   4096 Apr  6 08:04 snapshot

[saint]

@girish I can't get into Backups view because Cloudron Admin Panel is down. Ha ha, I think this is a chicken & egg question. Is there any other places you could think of where backups might be defaulting to?

[girish]

@saint Ah lol, mmm . Try mysql -uroot -ppassword -e "SELECT value from box.settings where name='backup_config'" . That should give the backup location.

Er, not really. Since mysql is down too!

Well, maybe it's best you send us a mail to support@cloudron.io and we can try to fix things up?

[saint]

@girish Sure thing, will do - thank you so much!

[nebulon]

Turns out the disk got full with docker images related to jupiter hub notebooks.