Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?

[robi]

it's from all the crashing

[nebulon]

@necrevistonnezr you can delete those files safely, of course you will lose all logs history, but if you don't have any immediate issue on your server besides log size, you can just remove them via ssh.

[d19dotca]

The logs are still being poorly maintained it seems, they account for several GBs of disk space which seems crazy high for just logs:

813M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499/app.log
788M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6/app.log
712M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d/app.log
656M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d/app.log
631M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a/app.log
629M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719/app.log
613M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e/app.log
596M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20/app.log
578M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244/app.log
514M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991/app.log

In my case, it seems the same issue I reported back February remains... there are missing logrotate.conf files for the area where there's far too high of log file sizes.

Example: Note that logs for 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 was the largest... when I run the following command, there's nothing returned, suggesting the logrotate.conf is missing for this one, and likely all the rest too: ls -alh /home/yellowtent/platformdata/logrotate.d/ | grep 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 returns no files.

When I do an ls -alh listing for the directory, here is all that exists which seems far too few:

ls -alh /home/yellowtent/platformdata/logrotate.d
total 80K
drwxr-xr-x  2 root       root       4.0K May 29 05:00 .
drwxr-xr-x 15 yellowtent yellowtent 4.0K May 15 07:27 ..
-rw-r--r--  1 root       root        946 May 15 20:21 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
-rw-r--r--  1 root       root        946 May 17 22:43 02b2f762-afa4-4278-afaa-5c07ad29715e
-rw-r--r--  1 root       root        946 May 20 16:28 200d36ca-8109-409a-9b56-07f46c2f09bd
-rw-r--r--  1 root       root        946 May 21 17:35 29354a9c-7e1d-4986-b78e-3510fe73b370
-rw-r--r--  1 root       root        946 May 27 05:00 3167ef7d-e4c2-4072-8e11-8de49c5db970
-rw-r--r--  1 root       root        946 May 29 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
-rw-r--r--  1 root       root        946 May 20 16:23 633f2c77-9b33-457a-bc3c-919c87d25ca2
-rw-r--r--  1 root       root        946 May 17 22:41 64098702-00d1-4d2e-8d27-4597e100d1c9
-rw-r--r--  1 root       root        946 May 23 06:28 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
-rw-r--r--  1 root       root        946 May 16 06:37 9f9aba4a-f7b1-44d8-859c-c6953027c39e
-rw-r--r--  1 root       root        230 May 15 09:00 box
-rw-r--r--  1 root       root        946 May 17 22:38 c3ddc268-deef-49e0-8e3a-97e174cf0594
-rw-r--r--  1 root       root        946 May 27 20:55 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
-rw-r--r--  1 root       root        946 May 27 21:00 ca7827f8-4686-4818-a821-f92fe4f5726d
-rw-r--r--  1 root       root        946 May 18 05:00 e18fa18b-b176-4b25-bf78-4d82c022b0f5
-rw-r--r--  1 root       root        946 May 26 15:44 e50f78c7-da6e-4036-a1a3-0cc934448cf1
-rw-r--r--  1 root       root        946 May 24 00:24 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
-rw-r--r--  1 root       root       1.1K May 15 09:00 platform

[nebulon]

@d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

[d19dotca]

@nebulon said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

@d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

No, unfortunately not. That has never been the case either, wish it was that easy. haha. I reported earlier in this thread that re-saving an app with from Location tab will regenerate the logrotate configs for the app. They just go missing randomly, no idea why they disappear, but I at least know how to get them back. Unfortunately I am not aware of any of this until I randomly check sizing on my server and find incredibly large log files which then leads me to discover the apps are missing logrotate configs. Saving the app again so it sort of rebuilds itself will regenerate it. I just looked now and there are a bunch more, for example: I didn't do anything beyond restoring the Cloudron backup to a new Cloudron instance in the Vultr Marketplace last night:

ls -alh /home/yellowtent/platformdata/logrotate.d
total 156K
drwxr-xr-x  2 root       root       4.0K May 31 03:54 .
drwxr-xr-x 15 yellowtent yellowtent 4.0K May 30 06:27 ..
-rw-r--r--  1 root       root        946 May 30 06:28 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
-rw-r--r--  1 root       root        946 May 30 06:28 02b2f762-afa4-4278-afaa-5c07ad29715e
-rw-r--r--  1 root       root        946 May 30 06:28 05db518f-2c45-411b-865f-813dbf85c244
-rw-r--r--  1 root       root        946 May 30 06:28 1247a348-c8da-4f30-8a4c-3fa014f092a1
-rw-r--r--  1 root       root        946 May 30 06:28 134ef7f1-46f8-403e-8ac5-a3297b37893f
-rw-r--r--  1 root       root        946 May 30 06:28 200d36ca-8109-409a-9b56-07f46c2f09bd
-rw-r--r--  1 root       root        946 May 30 06:28 21aecd6a-398a-433a-a954-5bcd5f861e20
-rw-r--r--  1 root       root        946 May 30 06:28 29354a9c-7e1d-4986-b78e-3510fe73b370
-rw-r--r--  1 root       root        946 May 30 06:28 3167ef7d-e4c2-4072-8e11-8de49c5db970
-rw-r--r--  1 root       root        946 May 30 06:29 38f883dc-4047-4237-a6a1-3ecd77efcde9
-rw-r--r--  1 root       root        946 May 30 06:28 40ae3d28-d67f-453e-895d-0f22b539372c
-rw-r--r--  1 root       root        946 May 30 06:29 4929b73d-cfa4-4246-82a7-4bc8127982f6
-rw-r--r--  1 root       root        946 May 30 06:29 492b4264-fea8-4d52-bbe6-1f15ce72c53a
-rw-r--r--  1 root       root        946 May 30 06:29 493c4f88-5d4d-40e5-8361-fd7d4541441d
-rw-r--r--  1 root       root        946 May 30 06:29 4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
-rw-r--r--  1 root       root        946 May 30 06:29 6120f84a-0953-4982-9167-ef872c0ee49a
-rw-r--r--  1 root       root        946 May 30 06:29 633f2c77-9b33-457a-bc3c-919c87d25ca2
-rw-r--r--  1 root       root        946 May 30 06:29 64098702-00d1-4d2e-8d27-4597e100d1c9
-rw-r--r--  1 root       root        946 May 31 03:54 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
-rw-r--r--  1 root       root        946 May 30 06:29 6e21d87b-ea85-420b-8eed-8b14d601438d
-rw-r--r--  1 root       root        946 May 30 06:30 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
-rw-r--r--  1 root       root        946 May 30 06:29 8315f008-87e9-4f62-9ae8-1938d7b3f9b2
-rw-r--r--  1 root       root        946 May 30 06:29 8732aed0-273e-49f5-837f-cd9c600bf7b9
-rw-r--r--  1 root       root        946 May 30 06:30 8dba8d47-7890-49d3-8dfa-970e1ec52991
-rw-r--r--  1 root       root        946 May 30 06:30 95c45c33-bd8a-4fae-9718-509c69456abe
-rw-r--r--  1 root       root        946 May 30 06:30 9f9aba4a-f7b1-44d8-859c-c6953027c39e
-rw-r--r--  1 root       root        946 May 30 06:30 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
-rw-r--r--  1 root       root        230 May 28 17:54 box
-rw-r--r--  1 root       root        946 May 30 06:30 c3ddc268-deef-49e0-8e3a-97e174cf0594
-rw-r--r--  1 root       root        946 May 30 06:31 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
-rw-r--r--  1 root       root        946 May 30 06:32 d3121e48-4196-48fe-907d-ee831e11ce5c
-rw-r--r--  1 root       root        946 May 30 06:32 e18fa18b-b176-4b25-bf78-4d82c022b0f5
-rw-r--r--  1 root       root        946 May 30 06:33 e50f78c7-da6e-4036-a1a3-0cc934448cf1
-rw-r--r--  1 root       root        946 May 30 06:31 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
-rw-r--r--  1 root       root        946 May 30 06:31 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
-rw-r--r--  1 root       root        946 May 30 06:31 f6b21e27-6d25-4c69-a547-0466e622d719
-rw-r--r--  1 root       root       1.1K May 28 17:54 platform

[d19dotca]

This is still an issue by the way. Noticed my disk space accumulating quickly. There's practically no logrotate's configured. Is this due to 6.3 though perhaps?

ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d
total 20K
drwxr-xr-x  2 root       root       4.0K Jun 30 15:28 .
drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 ..
-rw-r--r--  1 root       root        946 Jun 30 15:28 38f883dc-4047-4237-a6a1-3ecd77efcde9
-rw-r--r--  1 root       root        230 Jun 30 05:05 box
-rw-r--r--  1 root       root       1.1K Jun 30 05:05 platform

ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
4.1G	total
4.1G	/home/yellowtent/platformdata/
2.2G	/home/yellowtent/platformdata/mysql
891M	/home/yellowtent/platformdata/logs
470M	/home/yellowtent/platformdata/graphite
421M	/home/yellowtent/platformdata/mongodb
97M	/home/yellowtent/platformdata/postgresql
54M	/home/yellowtent/platformdata/backup
15M	/home/yellowtent/platformdata/redis
4.0M	/home/yellowtent/platformdata/nginx
192K	/home/yellowtent/platformdata/update
176K	/home/yellowtent/platformdata/collectd
36K	/home/yellowtent/platformdata/addons
16K	/home/yellowtent/platformdata/sftp
16K	/home/yellowtent/platformdata/logrotate.d
8.0K	/home/yellowtent/platformdata/firewall
4.0K	/home/yellowtent/platformdata/acme

To re-iterate, the logrotate files get generated using the workaround of hitting "Save" on the "Locations" tab of each app, but this is a manual process and I now have to do this for about 30 apps, causing a bit of downtime with each one. Then I need to run the logrotate function manually to save the disk space again.

I'm actually a little surprised this would be an issue when all the containers already had to be regenerated as part of the 6.3 upgrade.

@staff - I'm concerned that this defect hasn't been fixed since it was reported quite a while back. Any possibility of getting to the bottom of this one sooner than later?

[d19dotca]

I took the time to manually re-save each app Location tab to recreate the logrotate file as per the workaround, which resulted in a bit of downtime for each app.

I then ran the sudo logrotate -f /home/yellowtent/platformdata/logrotate.d/* command and saved roughly half a GB in logs directory alone (I see the mongodb and graphite folders among others went much smaller too, presumably due to restarting of the apps?). So basically 500 MB in extra logs and that's only on a server that's been running for only about a week, presumably would have grown much further in just another week or two too if the logrotate files weren't regenerated.

I really hope this will be fixed in the next bug-fix release.

ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
3.2G	total
3.2G	/home/yellowtent/platformdata/
2.2G	/home/yellowtent/platformdata/mysql
422M	/home/yellowtent/platformdata/mongodb
229M	/home/yellowtent/platformdata/graphite
171M	/home/yellowtent/platformdata/logs
97M	/home/yellowtent/platformdata/postgresql
54M	/home/yellowtent/platformdata/backup
17M	/home/yellowtent/platformdata/redis
4.0M	/home/yellowtent/platformdata/nginx
192K	/home/yellowtent/platformdata/update
176K	/home/yellowtent/platformdata/collectd
172K	/home/yellowtent/platformdata/logrotate.d
36K	/home/yellowtent/platformdata/addons
16K	/home/yellowtent/platformdata/sftp
8.0K	/home/yellowtent/platformdata/firewall
4.0K	/home/yellowtent/platformdata/acme

[d19dotca]

@girish - I was wanting to follow-up on this. The defect still seems to exist. Right now, despite me running about 41 apps, I only have several in the logrotate.d folder:

ubuntu@my:/home/yellowtent/platformdata$ ls -alh logrotate.d/
total 60K
drwxr-xr-x  2 root       root       4.0K Sep  7 05:00 .
drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 ..
-rw-r--r--  1 root       root        946 Aug 22 03:43 3167ef7d-e4c2-4072-8e11-8de49c5db970
-rw-r--r--  1 root       root        946 Aug 23 16:40 4929b73d-cfa4-4246-82a7-4bc8127982f6
-rw-r--r--  1 root       root        946 Sep  7 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
-rw-r--r--  1 root       root        946 Sep  4 03:34 684cf8eb-f80c-4304-b97d-ab915a18cfe7
-rw-r--r--  1 root       root        946 Sep  4 23:07 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
-rw-r--r--  1 root       root        946 Aug 19 23:11 7384a604-7a42-4dc7-86eb-d3f6dd2d546b
-rw-r--r--  1 root       root        946 Aug 19 03:25 8dba8d47-7890-49d3-8dfa-970e1ec52991
-rw-r--r--  1 root       root        946 Aug 24 05:48 9c70787a-3529-4b66-be2e-7b1b5aa92b02
-rw-r--r--  1 root       root        946 Aug 19 23:29 b4888539-e835-4f21-97f5-2fc856be6ecf
-rw-r--r--  1 root       root        230 Aug 13 06:39 box
-rw-r--r--  1 root       root        946 Aug 22 03:05 c93cef28-3cc4-42d8-88d5-0f581dff74a2
-rw-r--r--  1 root       root        946 Sep  4 05:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
-rw-r--r--  1 root       root       1.1K Aug 13 06:39 platform

[nebulon]

@d19dotca if you reconfigure the location of one of the missing apps, does the logrotate config file show up afterwards? I am wondering if something is purging them after they are created. Looking at the code I cannot spot why it wouldn't create that file.

[d19dotca]

@nebulon Yes, I think I've mentioned that before too. Re-saving from the Location tab on each app regenerates the logrotate file and finally the logrotates will proceed successfully.

I know @girish confirmed that he saw the same behaviour on one of your (or his) servers too. So this definitely seems to be a Cloudron bug that needs fixing.

This issue has been sitting a long time (confirmed/reproduced in February 2021), so hoping this gets resolved soon because this keeps causing issues on my server when it comes to disk space consumption alerts.

[d19dotca]

@nebulon @girish - This is still an issue by the way. Hoping this fix comes soon! I just checked my server after migrating it over the weekend, and I only have 2 apps in the logrotate.d directory. Having to go through 35 apps now to reinitialize that for proper log maintenance.

[girish]

@d19dotca I found the issue finally! After a cloudron update, all the old logrotate configs are removed. This was added many releases ago to regenerate new logrotate configs and we forgot to remove the line after that release. Fixed it now.

[d19dotca]

@girish yay! Haha, thank you so much!!! That one was really annoying even though minor. Lol. It happens after a Cloudron migration (not just upgrade) and stuff too FYI but I’m assuming you already knew that.

Edit: actually maybe not… I also had upgraded Cloudron the very next day so I likely just saw the result of that instead. Ignore me. glad it’s fixed though!!