Disk often full, cannot be extended, what remediations exist?

[james]

Hello @SansGuidon
A full disk is never good.
Trying to prevent this can only be done with monitoring and alerting.
Maybe @BrutalBirdie could help with that since he stated the following here:

@BrutalBirdie said in [Freelance] Need help Disk full:

Would be happy to help and can even offer a full monitoring solution that would notify you if something is happening or even before it gets critical.
Including dashboard access to full statistics like Docker System, file system, Networking, System performance, website overview (selenium smoke tests etc.) and to some extend Cloudron API monitoring.
This could also be extended if something else is needed.

@SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

I'm only interested in the storage. And I'm not sure adding a Hetzner volume would work with Cloudron. Is this documented somewhere? As this already been done? I would appreciate guidance on this.

Yes, using Hetzner Cloud Volumes for easy extendability of Hetzner Cloud Servers storage capacity is possible and not that complicated.
There is a full guide here https://docs.cloudron.io/storage/#storage for moving Docker Images and Default Data Directory.
This guide was very recently tested and verified by @BrutalBirdie

If I understood @BrutalBirdie correctly in our exchange he is always running Hetzner Cloud Servers in a minimal configuration starting with CX22 and only ever scaling up with only CPU and RAM and never the disk size, since you can't scale down when scaling with the storage.
And he is using Hetzner Cloud Volumes with the linked doc/guide to have a scalable storage.

If I misrepresent anything here @BrutalBirdie please let me know.

[BrutalBirdie]

@james said in Disk often full, cannot be extended, what remediations exist?:

moving Docker Images and Default Data Directory.

Moving the Default Data Directory also means moving all app data storage to this Cloud Volume.
And yes, this is doable with no issues.

So your worries @SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

The app with more data are Immich and Syncthing, I'm thinking of moving the Immich data on external volumes, but I'm not sure Immich allows to migrate existing data or specify which folder I want to use for storage.

Are understandable, but should be no issue at all.

@james said in Disk often full, cannot be extended, what remediations exist?:

If I understood @BrutalBirdie correctly in our exchange he is always running Hetzner Cloud Servers in a minimal configuration starting with CX22 and only ever scaling up with only CPU and RAM and never the disk size, since you can't scale down when scaling with the storage.
And he is using Hetzner Cloud Volumes with the linked doc/guide to have a scalable storage.

You hit the nail on the head.
This is what I do.

I am planning automating these setups with Ansible if demand increases.
And within this plan, I also think about auto-scaling down and up in the "unproductive" times.
Example, why have a server use the CX42 plan at night when nothing is happening? With my monitoring setup I could automate a downscale at night and upscale at morning before work start to reduce costs.
(But that is a dream of mine which I want to tackle if I got the time and there is much to consider, like RAM limits of apps and what happens if e.g. an app has a limit of 8GB RAM and I downscale at night to a CX22 with just 4GB RAM, will the system just crash or just be super slow?)

Sorry for drifting off
Yes, you can use this guide and if you follow it step by step there should be no issues.
If you need some help with that or want to use some of my services like monitoring @SansGuidon feel free to contact me in a forum DM or on what ever platform suits you.
How to contact me? See: https://keyoxide.org/3EFABCB708412AEE165EDC6C7B17D07087F79F9E

[jdaviescoates]

I'd also add that if you're using Nextcloud check my recent guide about how to reclaim lots of space from that https://forum.cloudron.io/post/110118

[SansGuidon]

Thanks folks
I've read the guide at https://docs.cloudron.io/storage/#storage and I'm not sure how to proceed then. The guide mentions we can move the data directory but not to a Cloudron volume, so this should be an external volume, is it where Hetzner extra volumes comes into play? I guess then I might be a tutorial away from this first part of the solution.

But these are still dragons to me:

1. The guide advises to not use symlinks but the example commands in the same guide at https://docs.cloudron.io/storage/#default-data-directory
   contain symlink afaik, right?

2. As far as I understand it, the guide seems to cover the case for a full migration of docker images or apps data but it is unclear to me how to tackle this only for specific apps, and if that's a supported scenario. I mean maybe the guide's purpose is to migrate all /app/data volume to an external volume, that could work but then this renders my VPS main partition quite useless, right?

3. I also read several times it was advised not to touch the Cloudron server configuration except via the UI, to ensure future Cloudron upgrades, backups, restore etc would work.
   I guess this is a specific case that requires anyway to touch the terminal, but is it well supported by Cloudron (future upgrades etc)?

I've checked the disk usage to decide what to migrate in my case, I believe migrating the appsdata makes more sense in this case. But the docker images take as much space.

root@ubuntu-cloudron-16gb-nbg1-3:/home/yellowtent# du -hs  appsdata/ boxdata/ platformdata/
28G	appsdata/
49M	boxdata/
4.1G	platformdata/
root@ubuntu-cloudron-16gb-nbg1-3:/home/yellowtent# du -hs /var/lib/docker/
156G	/var/lib/docker/
root@ubuntu-cloudron-16gb-nbg1-3:/home/yellowtent# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          36        36        27.24GB   2.511GB (9%)
Containers      42        42        0B        0B
Local Volumes   86        86        11.6GB    0B (0%)
Build Cache     0         0         0B        0B

Thanks in advance for the clarifications

[BrutalBirdie]

There is a difference between Cloudron Volumes and Hetzner Volumes.
In the guide is written:

Do not use a Cloudron Volume as storage location for the directories below. Set up fstab or systemd mount manually.

@SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

The guide advises to not use symlinks

It states:

While symlinking various directories will work, this is not supported.

It works, but is not officially supported.
Meaning, yes you can do that, but when you have an issue, you will get less or no support.

@SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

As far as I understand it, the guide seems to cover the case for a full migration of docker images or apps data but it is unclear to me how to tackle this only for specific apps, and if that's a supported scenario.

That what Cloudron Volumes are for.
You can add a Hetzner Volume and mount it to e.g /mnt/hetzner-volume/ and then in the Cloudron Dashboard at volumes do the following:

Then, for each app you can move the data directory.
Like it is described in the doc https://docs.cloudron.io/apps/#data-directory and https://docs.cloudron.io/volumes

@SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

I also read several times it was advised not to touch the Cloudron server configuration except via the UI, to ensure future Cloudron upgrades, backups, restore etc would work.
I guess this is a specific case that requires anyway to touch the terminal, but is it well supported by Cloudron (future upgrades etc)?

Let me quote the doc again:

While symlinking various directories will work, this is not supported.

Officially, this is not supported to reduce support.
But it does work.
And yes, you should not touch the underlaying Server. But it is still your server. If you know what you do and can handle certain issues yourself you can do tasks on the root level.

---

I am doing this for almost all my servers and can tell you, I rarely had issues.
The only issues that ever appeared was when Hetzner Cloud Volumes had some outages (which rarely happens).

[BrutalBirdie]

And here is my terminal output from my personal Cloudron:

root@my-hackradt-com:~# ls -lah /home/yellowtent/
total 32K
drwxr-xr-x 5 yellowtent yellowtent 4.0K Jul  2 18:49 .
drwxr-xr-x 4 root       root       4.0K Oct  3  2024 ..
-rw-r--r-- 1 yellowtent yellowtent  220 Mar 31  2024 .bash_logout
-rw-r--r-- 1 yellowtent yellowtent 3.7K Mar 31  2024 .bashrc
-rw-r--r-- 1 yellowtent yellowtent    0 Jul 16  2024 .cloud-locale-test.skip
drwx------ 2 yellowtent yellowtent 4.0K Nov  9  2024 .gnupg
-rw-r--r-- 1 yellowtent yellowtent  807 Mar 31  2024 .profile
drwx------ 2 yellowtent yellowtent 4.0K Oct  3  2024 .ssh
lrwxrwxrwx 1 root       root         54 Jul  2 18:48 appsdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/appsdata
drwxr-xr-x 8 yellowtent yellowtent 4.0K Jul 21 12:02 box
lrwxrwxrwx 1 root       root         53 Jul  2 18:49 boxdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/boxdata
lrwxrwxrwx 1 root       root         58 Jul  2 18:49 platformdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/platformdata

I did not move the Docker Images Location since I did not need that for this server.
And here is a screenshot of my Hetzner UI.
I've used this setup for 10 months with little to no issues.

[SansGuidon]

Thank you, this clarifies things
Now the pros and cons of opting for either hetzner volumes vs using cloudron volumes is unclear to me, you mention the data directory can also be moved app-per-app via Cloudron UI. I've opted for moving the appsdata folder manually for all apps to a new location, following the guide at https://docs.cloudron.io/storage/#default-data-directory but this seems a different strategy than switching data directory from Cloudron Admin UI and mounting Cloudron volumes.

[BrutalBirdie]

@SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

Now the pros and cons of opting for either hetzner volumes vs using cloudron volumes

You can use a Hetzner Volume with Cloudron Volumes.
What you should not do is use a Hetzner Volume as a Cloudron Volume for a full move which is described in https://docs.cloudron.io/storage/#storage.
Example, you get a 100GB Hetzner Volume, add this in the Cloudron Dashboard as a volume and then move /home/yellowtent/$DIR into this location.
This, you should not do!

---

In my opinion, the pro of using a Cloudron Volume and moving each app to this volume with the GUI is, you stick to the way it is meant to be done and what is officially supported.

On the other hand, I am lazy, I don't want to move every app, I have 30x+ apps.
I just move the whole thing and also include:

lrwxrwxrwx 1 root       root         53 Jul  2 18:49 boxdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/boxdata
lrwxrwxrwx 1 root       root         58 Jul  2 18:49 platformdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/platformdata

This way, all appsdata, all boxdata and all platformdata is on a separate volume.
And when I need more storage, I just increase the Hetzner Volume, resize it in the terminal, and I am done.

[SansGuidon]

Thanks for the advise. Currently I'm in a weird situation, as I've followed the guide step by step and Cloudron starts but my apps do not show any data anymore.

[BrutalBirdie]

I will send you a DM.

[jdaviescoates]

@BrutalBirdie said in Disk often full, cannot be extended, what remediations exist?:

I really had issues.

s/really/rarely

[BrutalBirdie]

@jdaviescoates Good Catch

[SansGuidon]

I solved the problem, not sure what had the most impact:

Change permissions on my mount points/symlinks:

chown yellowtent:yellowtent /mnt/appsdata
chown yellowtent:yellowtent /mnt/appsdata/appsdata

Reboot: not mentioned in the guides but after changing permissions and rebooting, the problem was solved compared to several attempts before this, without reboot and with improper permissions changes.
The permissions and reboot parts are not mentioned in the guide for doing those operations if I'm right, so I'm not sure this was really needed, anyway I prefer to share

Thanks to @BrutalBirdie for all the feedback above and the proposed support (in DM), this community is awesome!