Cloudron Admin Panel Down
Solved
Support
-
Hi there,
I just ran into an issue where I tried to install a new app (Joplin server), and the installation failed. I don't think it is due to the package, but likely more due to disk space?
The error message I got was:
Task Error If a configuration, update, restore or backup action resulted in an error, you can retry the task. An error occurred during the install operation: FileSystem Error: Error creating directory: ENOENT: no such file or directory, mkdir '/home/yellowtent/appsdata/e2b71a70-2d9b-48ed-98c8-aaf99534527c'Some observations and steps I've attempted:
- Tried all of the below troubleshooting steps
- systemctl status and restart nginx - system systemctl status and restart docker - systemctl status and restart box - Ran /home/yellowtent/box/setup/start.sh - Ran apt-get remove `dpkg --list 'linux-image*' |grep ^ii | awk '{print $2}'\ | grep -v \`uname -r\``The above were fine, and showed all were active or didn't give errors.
- I have noticed that I am running out of space on the server and/or;
- The MySQL database seems to be down when i did a
systemctl status mysql(see below)
- I have also noticed Unbound seems to be down as well when i did a
systemctl status unbound, and there is an error that says "Could not fflush" ... "No space left on device."
A
df -hshows the below, and it looks like/dev/vda2is extremely full, but I'm not sure what to do about that.root@server:/# df -h Filesystem Size Used Avail Use% Mounted on udev 1.9G 0 1.9G 0% /dev tmpfs 394M 2.0M 392M 1% /run /dev/vda2 99G 96G 0 100% / tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/loop0 56M 56M 0 100% /snap/core18/2714 /dev/loop1 56M 56M 0 100% /snap/core18/2721 /dev/loop3 64M 64M 0 100% /snap/core20/1852 /dev/loop2 143M 143M 0 100% /snap/lxd/24483 /dev/loop4 73M 73M 0 100% /snap/core22/583 /dev/loop5 64M 64M 0 100% /snap/core20/1828 /dev/loop6 73M 73M 0 100% /snap/core22/607 /dev/loop7 163M 163M 0 100% /snap/lxd/24643 /dev/loop9 50M 50M 0 100% /snap/snapd/18357 /dev/loop8 50M 50M 0 100% /snap/snapd/18596 tmpfs 394M 0 394M 0% /run/user/0 overlay 99G 96G 0 100% /var/lib/docker/overlay2/5622e5e4a0509f7435f120de894243aa84fd7f0f217972c8a66897183c10fa60/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/9806e31f0d815938028a3871909ea8350c735bc62922c69ec08b540f4de4de61/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/842c348622671b853ec1900f1cbfca9077fc0d582823b895f9ed6f9b88542808/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/a2bd4fadd972a4cb79a70cd024a3df13a9786a0729d0a238b352aa187cdd4f9d/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/bb6073e44eeaf106990e1c1b1076170c2f8dfe6445c43b20df8ce4d61a530540/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/8272b642f023dc33a2466c81f2d478af445628138921d9928fe706d425d50c51/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/d1964be7093c93ff6933c2833078d7bab481eda8eb3778a5079b88bbc8591227/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/f38dbd9d38069c35bc86c352e4e28ae76114b4162dd1cc4174bc53a72ce7c3d6/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/91c649919a97a56c4c7b639fe957187d1b7c9f6142cde887b7ed8679cd1aec00/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/5dee1219784bd35166469b7909d04369b99a2b6e08bd54b2afa2900bdb14ac78/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/7c25c359f68d5f4a8fce3fdf566ce0f444a2afd10f4873795e2c1d14176c01ea/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/61552af60fe61456f7e0a7a937300b300aaac22fd8ec0b0cbbd92b6d5e01194e/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/d02b8cfdb5e8a9ff1517d44773b3f3a8eaa1add5f62437a9e2c0f8f34bee5f38/merged overlay 99G 96G 0 100% /var/lib/docker/overlay2/5faa2b5cd2fb958e5966e181d09bde2b2c0e3517e50a026eb9e6ed94fcc97162/mergedMy questions/Asks:
- Is this a MySQL issue and should I attempt to recover MySQL? I'm hoping not to mess with the DB unless I absolutely have to.
What about deleting previous backups?
- Delete Older Backups - Is there a way to delete some of the older version backups to free up some space? And if so, where would I go to do that since I can't access the Cloudron Admin Panel.
- Install/Update Apps on Staging - Could we have an update path where the upgrade happens on a STAGING instance for the apps, and when there's no issue, can be pushed out to PROD? This could help minimize workflow issues in the future since right now, I'm not able to use any of the apps since Cloudron AP is down.
Appreciate any assistance that could help bring Cloudron AP back online. Thank you.
-
@saint Where are your backups? Are they on the same server as Cloudron ? If so, I would start with deleting some backups in
/var/backups. These are timestamped directories, you can delete the old ones, I guess. But do keep some around, in case, you are not able to recover!https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full has the steps for recovery.
-
@girish Yes, they are on the same server as Cloudron.
Based off what I can see below, everything in backup totals up to only 2.0MB? That seems strange... I know my Nextcloud instance has a ton of photos in it, so not sure if I'm just reading it incorrectly.
Sounds like I can delete between *.3 - *6 as *6.gz is the furthest one away. Is there any file below that I should not remove?
root@server:/var/backups# ls -alhS total 2.0M -rw-r--r-- 1 root root 705K Apr 4 06:08 dpkg.status.0 -rw-r--r-- 1 root root 175K Apr 4 06:08 dpkg.status.1.gz -rw-r--r-- 1 root root 175K Apr 4 06:08 dpkg.status.2.gz -rw-r--r-- 1 root root 175K Mar 28 06:11 dpkg.status.3.gz -rw-r--r-- 1 root root 175K Mar 28 06:11 dpkg.status.4.gz -rw-r--r-- 1 root root 175K Mar 28 06:11 dpkg.status.5.gz -rw-r--r-- 1 root root 175K Mar 28 06:11 dpkg.status.6.gz -rw-r--r-- 1 root root 50K Mar 22 06:25 alternatives.tar.0 -rw-r--r-- 1 root root 38K Mar 28 06:11 apt.extended_states.0 -rw-r--r-- 1 root root 4.2K Mar 3 06:28 apt.extended_states.1.gz -rw-r--r-- 1 root root 4.2K Feb 10 06:11 apt.extended_states.2.gz -rw-r--r-- 1 root root 4.2K Jan 14 06:22 apt.extended_states.3.gz -rw-r--r-- 1 root root 4.2K Jan 8 06:04 apt.extended_states.4.gz -rw-r--r-- 1 root root 4.2K Dec 3 06:03 apt.extended_states.5.gz -rw-r--r-- 1 root root 4.2K Nov 16 06:05 apt.extended_states.6.gz drwxrwxrwx 2 root root 4.0K Apr 6 06:25 . drwxr-xr-x 14 root root 4.0K Sep 4 2022 .. -rw-r--r-- 1 root root 2.6K Mar 1 06:25 alternatives.tar.1.gz -rw-r--r-- 1 root root 2.6K Jan 31 06:25 alternatives.tar.2.gz -rw-r--r-- 1 root root 2.6K Oct 27 06:25 alternatives.tar.5.gz -rw-r--r-- 1 root root 2.6K Jan 14 06:25 alternatives.tar.4.gz -rw-r--r-- 1 root root 2.6K Jan 27 06:25 alternatives.tar.3.gz -rw-r--r-- 1 root root 2.6K Sep 20 2022 alternatives.tar.6.gz -rw-r--r-- 1 root root 268 Jan 11 2021 dpkg.diversions.0 -rw-r--r-- 1 root root 139 Jan 11 2021 dpkg.diversions.1.gz -rw-r--r-- 1 root root 139 Jan 11 2021 dpkg.diversions.2.gz -rw-r--r-- 1 root root 139 Jan 11 2021 dpkg.diversions.3.gz -rw-r--r-- 1 root root 139 Jan 11 2021 dpkg.diversions.4.gz -rw-r--r-- 1 root root 139 Jan 11 2021 dpkg.diversions.5.gz -rw-r--r-- 1 root root 139 Jan 11 2021 dpkg.diversions.6.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.1.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.2.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.3.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.4.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.5.gz -rw-r--r-- 1 root root 120 Apr 23 2020 dpkg.statoverride.6.gz -rw-r--r-- 1 root root 100 Apr 23 2020 dpkg.statoverride.0 -
@saint the backups will look like this:
root@my:/var/backups# ls -l total 2200 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 8 07:00 2023-01-08-070000-649 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 9 07:00 2023-01-09-070000-722 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 9 09:29 2023-01-09-092923-525 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 9 09:41 2023-01-09-094131-393 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 9 13:38 2023-01-09-133856-696 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 9 15:12 2023-01-09-151231-272 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 9 15:42 2023-01-09-154249-626 .... drwxr-xr-x 2 yellowtent yellowtent 4096 Apr 6 08:04 snapshot -
-
@saint Ah lol, mmm . Try
mysql -uroot -ppassword -e "SELECT value from box.settings where name='backup_config'". That should give the backup location.Er, not really. Since mysql is down too!
Well, maybe it's best you send us a mail to support@cloudron.io and we can try to fix things up?
-
N nebulon marked this topic as a question on
-
N nebulon has marked this topic as solved on
