Disk often full, cannot be extended, what remediations exist?

[BrutalBirdie]

And here is my terminal output from my personal Cloudron:

root@my-hackradt-com:~# ls -lah /home/yellowtent/
total 32K
drwxr-xr-x 5 yellowtent yellowtent 4.0K Jul  2 18:49 .
drwxr-xr-x 4 root       root       4.0K Oct  3  2024 ..
-rw-r--r-- 1 yellowtent yellowtent  220 Mar 31  2024 .bash_logout
-rw-r--r-- 1 yellowtent yellowtent 3.7K Mar 31  2024 .bashrc
-rw-r--r-- 1 yellowtent yellowtent    0 Jul 16  2024 .cloud-locale-test.skip
drwx------ 2 yellowtent yellowtent 4.0K Nov  9  2024 .gnupg
-rw-r--r-- 1 yellowtent yellowtent  807 Mar 31  2024 .profile
drwx------ 2 yellowtent yellowtent 4.0K Oct  3  2024 .ssh
lrwxrwxrwx 1 root       root         54 Jul  2 18:48 appsdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/appsdata
drwxr-xr-x 8 yellowtent yellowtent 4.0K Jul 21 12:02 box
lrwxrwxrwx 1 root       root         53 Jul  2 18:49 boxdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/boxdata
lrwxrwxrwx 1 root       root         58 Jul  2 18:49 platformdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/platformdata

I did not move the Docker Images Location since I did not need that for this server.
And here is a screenshot of my Hetzner UI.
I've used this setup for 10 months with little to no issues.

[SansGuidon]

Thank you, this clarifies things
Now the pros and cons of opting for either hetzner volumes vs using cloudron volumes is unclear to me, you mention the data directory can also be moved app-per-app via Cloudron UI. I've opted for moving the appsdata folder manually for all apps to a new location, following the guide at https://docs.cloudron.io/storage/#default-data-directory but this seems a different strategy than switching data directory from Cloudron Admin UI and mounting Cloudron volumes.

[BrutalBirdie]

@SansGuidon said in Disk often full, cannot be extended, what remediations exist?:

Now the pros and cons of opting for either hetzner volumes vs using cloudron volumes

You can use a Hetzner Volume with Cloudron Volumes.
What you should not do is use a Hetzner Volume as a Cloudron Volume for a full move which is described in https://docs.cloudron.io/storage/#storage.
Example, you get a 100GB Hetzner Volume, add this in the Cloudron Dashboard as a volume and then move /home/yellowtent/$DIR into this location.
This, you should not do!

---

In my opinion, the pro of using a Cloudron Volume and moving each app to this volume with the GUI is, you stick to the way it is meant to be done and what is officially supported.

On the other hand, I am lazy, I don't want to move every app, I have 30x+ apps.
I just move the whole thing and also include:

lrwxrwxrwx 1 root       root         53 Jul  2 18:49 boxdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/boxdata
lrwxrwxrwx 1 root       root         58 Jul  2 18:49 platformdata -> /mnt/my-hackradt-com-Cloudron-Data/yellowtent/platformdata

This way, all appsdata, all boxdata and all platformdata is on a separate volume.
And when I need more storage, I just increase the Hetzner Volume, resize it in the terminal, and I am done.

[SansGuidon]

Thanks for the advise. Currently I'm in a weird situation, as I've followed the guide step by step and Cloudron starts but my apps do not show any data anymore.

[BrutalBirdie]

I will send you a DM.

[jdaviescoates]

@BrutalBirdie said in Disk often full, cannot be extended, what remediations exist?:

I really had issues.

s/really/rarely

[BrutalBirdie]

@jdaviescoates Good Catch

[SansGuidon]

I solved the problem, not sure what had the most impact:

Change permissions on my mount points/symlinks:

chown yellowtent:yellowtent /mnt/appsdata
chown yellowtent:yellowtent /mnt/appsdata/appsdata

Reboot: not mentioned in the guides but after changing permissions and rebooting, the problem was solved compared to several attempts before this, without reboot and with improper permissions changes.
The permissions and reboot parts are not mentioned in the guide for doing those operations if I'm right, so I'm not sure this was really needed, anyway I prefer to share

Thanks to @BrutalBirdie for all the feedback above and the proposed support (in DM), this community is awesome!

[SansGuidon]

To add to the story, I've noticed that Cloudron system will often fill the disk with /var/log/syslog, this is patched for Cloudron 9 but for Cloudron 8.x however users are forced to patch manually , see https://forum.cloudron.io/topic/13361/after-ubuntu-22-24-upgrade-syslog-getting-spammed-and-grows-way-to-much-clogging-up-the-diskspace/6?_=1756455716990

I'm a bit confused that such a problem remains to patch manually for half a year anyway I've applied it.

systemctl stop rsyslog syslog.socket
truncate -s 0 /var/log/syslog.*
rsystemctl restart cloudron-syslog.service
systemctl start rsyslog syslog.socket

[scooke]

I was in a similar situation as you. However, rather than bashing my head (figuratively) against the same wall, i just moved to a vps with more space. Granted, this was before using external volumes was as easy as it is now. But reading that your two main apps are Immich and Sync thing... there's two walls.

Listen, there are amazing workarounds and solutions offered by Cloudron and from others on this forum. But, I've never viewed any of this as "recommended and official"; it's more, "enough people refuse to/can't do the obvious, so how can we help them?" The obvious being - get a server or vps or whatever that has all the space you need. It's one reason I moved to ssdnodes (but just enough users have not had a good experience with them so this isn't an ad, I've just had excellent results with them and the 1.2tb server they offer.)

[SansGuidon]

@scooke Thanks for the suggestion,
while I admit this is a way to move forward, moving to a bigger vps/storage with my provider (Hetzner) would cost me money. I do not want to waste any money to create room for wasted storage. But I understand the advice which is pragmatic. However I find also satisfaction in understanding the root cause of a problem, and sharing the findings with the community rather than throwing dust under the carpet. For now I've been able to mitigate the issue and develop some scripts to monitor logs usage per container. Syncthing is indeed an issue but the main problem seems that there are too many logs generated/kept by application in Cloudron and it seems it's an issue with log management in Cloudron on Ubuntu. I could fix this with some extra logrotate configs on the server to prevent the log files to grow up to many GB in size.
Of course some apps might generate lot of activity, but I do not believe this is the main problem, and for now I do not want to waste TB of storage for logs when I could use this for data. So I've tuned the config to reduce proliferation of logs. I've learned some things in the process, about Cloudron & its logs management, about why there were so many issues reported in this forum about disk being full, and I'm happy to see it's partially solved in Cloudron 9.
The obvious thing for me is always to understand things and try to find the root cause. however I understand also why for lot of people, the most obvious thing is to throw more Resources (CPU/GPU/Storage) at a problem to not waste time on optimization/maintenance. But for my own education I like to dig deeper when my gut feeling tells me there is a shitty story behind a bug

[SansGuidon]

From a deeper investigation, it seems that is has to do with backups and not with Syncthing or Photos app.
4 times a day, Uptime Kuma is backed up, and each time the heartbeat table is dumped into syslog, growing the syslog by 1-2GBs each time. More details in https://forum.cloudron.io/post/112261