Oversized Disk Usage

[murgero]

@Scardinavia I'd go into the backup tab in the cloudron admin and clean it.

Change retention to 1 day, then hit cleanup backups button. Change retention back to whatever you need it to after.

[Scardinavia]

@murgero thanks - I’ve tried this, but it hasn’t worked. I think the issue is that there are a couple of backups left on the original disk that are not being recognized now that I’ve set the backup location to a remote bucket…

I’ve tried re-setting the location of the backups to the Filesystem and doing a cleanup there too, but nothing has changed… I also ran

ls -sh /var/backups

and there’s nothing in there.

I’m quite new to all this, so sorry if I’m overlooking something obvious.

Cheers

[nebulon]

I think you have to look around the disk with the du command to find the location then. Beware that docker related folders are not reporting the real disk size with du so you can ignore those folders.

[murgero]

@nebulon As an addon here, you can install ncdu (du command using ncurses) for a more "gui" like interface for it.

[pbischoff]

Hello everyone,

I have a very similar Problem.

I am wondering what makes "Everything else (Ubuntu, etc)" 495.02 GB big.
For Backups I used S3 bucket from the beginning.

I also already decreased the retention policy and cleand up backups, but nothing changed in disk usage.

Thanks for your help and best regards
Philipp

[girish]

@pbischoff Cloudron does not keep track of the old backups. So, if things were previously on disk (before you moved to S3), you have to remove them manually. Can you check if /var/backups is taking up lots of space?

[pbischoff]

@girish Thanks for your answer.

As I said "For Backups I used S3 bucket from the beginning."
So backups NEVER were on disk.
The disk usage of /var/backups is 2,1 MB.

[girish]

@pbischoff Ah ok. Clearly something is wrong somewhere. It's hard to tell, you have to run manually du manually on the server to find out what is taking up so much space.

[pbischoff]

@girish okay, can you give me a hint which directories a the ones that cloudron uses?

I already tested all directories in /var directory. Here only lib/ has 135 GB and the rest under 100 MB.

[girish]

@pbischoff Can you check if the graphs match the df output ? Is it reflecting that ~930GB is used?

[pbischoff]

@girish Yes it looks like that!

[pbischoff]

@girish and the yellowtent/ directory uses 858 GB

[girish]

@pbischoff OK, so maybe du -hcs /home/yellowtent/appsdata/* will tell you which app is taking lots of space . If you put the appid that you get there into the search box of the Dashboard, you will see which app it is.

/home/yellowtent/box is just source code. I don't expect this to take much space

/home/yellowtent/platformdata is run time data (logs, databases). In the graph, this is around 180GB or so.

[pbischoff]

@girish okay thank you, that helped very much.

I identified a Directus instance that uses almost 500 GB. I think this is database data because of a cron process that fetches data periodically and writes logs for every fetch. How can I identify if it is really the database data or some files?

[girish]

@pbischoff /home/yellowtent/appsdata/<appid> only has the files. The database data is under /home/yellowtent/platformdata/postgresql .

One thing, you might want to check is the size of /home/yellowtent/appsdata/<appid>/postgresqldump . This is the size of the postgres dump of the app. Other than that, it's all file data.

You can open the web terminal for Directus app. And then, inspect /app/data there. (this is the same as /home/yellowtent/appsdata/<directus-app-id>/data on the host).

[pbischoff]

@girish okay this is really curious. When I look under "Storage" of that app in the Cloudron Dashboard it tells me The app is currently using 9.5 MB of storage (as of 1 hours ago).

[girish]

@pbischoff Sorry, I edited my post Maybe it's in the database after all. Can you check the size of the dump file?

[pbischoff]

@girish Yes it's this file! It has 447G!

Do I have to manually modify this file or will just deleting the data within Postgres do the trick?

[girish]

@pbischoff Yes, if you delete data from postgres and create a backup that file will shrink accordingly (that file is part of an app's "backup")

[pbischoff]

@girish I just wanted to say thank you for your help two weeks ago!

I didn't get Postgres running because of too little storage on the disk, so I had no idea in this short time how to delete the things out of Postgres.
For us this issue was very critical and I had to fix it very fast and so I just deleted the dump file which means the loss of the directus instance data, but that was okay in this case.
Again: thank you very much and best regards!