Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?
-
@girish Okay... I think I found a possible root cause of the issue (though it shouldn't be a big deal long-term, would only impact why I saw only 5 apps yesterday)... I believe the root cause is when an app has a pending update (the green up arrow next to it) then the logrotate mysteriously disappears. Here's why I feel this way and my detailed testing and investigation here...
- I forced the logrotate again today and it somehow cleared up another large batch of space, taking it from the leftover nearly 2 GB yesterday to just under 600 MB today.
ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr 3.3G total 3.3G /home/yellowtent/platformdata/ 1.8G /home/yellowtent/platformdata/mysql 592M /home/yellowtent/platformdata/logs 522M /home/yellowtent/platformdata/mongodb 358M /home/yellowtent/platformdata/graphite 89M /home/yellowtent/platformdata/postgresql 7.6M /home/yellowtent/platformdata/redis 724K /home/yellowtent/platformdata/nginx 164K /home/yellowtent/platformdata/update 128K /home/yellowtent/platformdata/collectd 116K /home/yellowtent/platformdata/backup 104K /home/yellowtent/platformdata/logrotate.d 32K /home/yellowtent/platformdata/addons 8.0K /home/yellowtent/platformdata/acme- Realized that when I listed out the /home/yellowtent/platformdata/logrotate.d/ directory, I suddenly had many more app configs listed in there than there was yesterday. Yesterday there was 5 apps only, today there were many (if not all of them at quick glance):
ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d/ total 108K drwxr-xr-x 2 root root 4.0K Feb 9 04:22 . drwxr-xr-x 15 yellowtent yellowtent 4.0K Dec 9 04:03 .. -rw-r--r-- 1 root root 946 Feb 9 04:13 0e55f920-98bd-4ff6-b591-7254c28451c0 -rw-r--r-- 1 root root 946 Feb 9 04:22 11c6fe46-82ff-4ea4-b816-3b78a962d048 -rw-r--r-- 1 root root 946 Feb 9 04:14 1c4fd6e4-09ed-4628-a2f4-0938312536f5 -rw-r--r-- 1 root root 946 Feb 9 04:13 24bff5cf-736c-49a2-a0aa-87e59131c4b6 -rw-r--r-- 1 root root 946 Feb 9 04:12 28f810a3-2d17-4e45-b0f8-e581528ac4ba -rw-r--r-- 1 root root 946 Feb 9 04:10 3167ef7d-e4c2-4072-8e11-8de49c5db970 -rw-r--r-- 1 root root 946 Feb 9 04:08 38f883dc-4047-4237-a6a1-3ecd77efcde9 -rw-r--r-- 1 root root 946 Feb 9 04:09 47164e82-f1d2-4fae-9959-507062614906 -rw-r--r-- 1 root root 946 Feb 9 04:14 83d329e1-6571-46e5-b14f-07477b8f2aac -rw-r--r-- 1 root root 946 Feb 3 04:00 8732aed0-273e-49f5-837f-cd9c600bf7b9 -rw-r--r-- 1 root root 946 Feb 3 04:00 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd -rw-r--r-- 1 root root 946 Feb 9 04:12 b6fa7dcd-1d93-4786-82e0-90a634f8c3a7 -rw-r--r-- 1 root root 946 Feb 9 04:09 be3840f0-458e-4bd8-8d18-34120ebdfce4 -rw-r--r-- 1 root root 230 Feb 2 16:21 box -rw-r--r-- 1 root root 946 Feb 9 04:15 c6a63a87-aa45-4281-b4ad-e65c85817f5d -rw-r--r-- 1 root root 946 Feb 9 04:15 cf1d5c7c-d4f0-4c32-ada8-e25ab025bd3d -rw-r--r-- 1 root root 946 Feb 9 04:09 d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d -rw-r--r-- 1 root root 946 Feb 9 04:14 d2df514f-18f6-4e4e-84a1-69bd6360765c -rw-r--r-- 1 root root 946 Feb 9 04:14 d2e46149-3287-4944-8e86-1c9a03e12641 -rw-r--r-- 1 root root 946 Feb 9 04:14 dc527d19-aced-44fa-bae1-a0b22c358a44 -rw-r--r-- 1 root root 946 Feb 9 04:12 de175741-ee83-4085-b04f-a0048cbdf493 -rw-r--r-- 1 root root 946 Feb 3 04:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7 -rw-r--r-- 1 root root 946 Feb 9 04:13 f1fe6d67-0897-4063-bc76-0ef7256bac7a -rw-r--r-- 1 root root 946 Feb 9 04:14 f2dbde27-0159-4e28-803e-7a3fc21450fb -rw-r--r-- 1 root root 1.1K Feb 2 16:21 platform- Digged further into the logs usage and found that one app in particular was using significantly more than the rest:
ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr 592M total 592M /home/yellowtent/platformdata/logs 204M /home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c 76M /home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1 72M /home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5 37M /home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d 22M /home/yellowtent/platformdata/logs/turn 11M /home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a 11M /home/yellowtent/platformdata/logs/11c6fe46-82ff-4ea4-b816-3b78a962d048 11M /home/yellowtent/platformdata/logs/0e55f920-98bd-4ff6-b591-7254c28451c0 9.8M /home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5 9.6M /home/yellowtent/platformdata/logs/47164e82-f1d2-4fae-9959-507062614906 9.4M /home/yellowtent/platformdata/logs/f2dbde27-0159-4e28-803e-7a3fc21450fb 9.1M /home/yellowtent/platformdata/logs/c6a63a87-aa45-4281-b4ad-e65c85817f5d 8.6M /home/yellowtent/platformdata/logs/dc527d19-aced-44fa-bae1-a0b22c358a44 8.5M /home/yellowtent/platformdata/logs/d2e46149-3287-4944-8e86-1c9a03e12641 8.4M /home/yellowtent/platformdata/logs/d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d 8.1M /home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a 7.2M /home/yellowtent/platformdata/logs/mail- I noticed that app d3121e48-4196-48fe-907d-ee831e11ce5c was the Radicale card/calDAV app. I then checked the list of logrotate app configs and sure enough that one was missing from the list as you can see above in #2.
- I followed the steps suggested by @girish and that was then successful in creating the new logrotate for Radicale app, and then I ran logrotate again and it of course cleared even more space now. This lead me to see the following usage afterwards (took the Radicale log usage from 204 MB to 53 MB now):
ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr 272M total 272M /home/yellowtent/platformdata/logs 76M /home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1 72M /home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5 53M /home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c 37M /home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d 8.1M /home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a 368K /home/yellowtent/platformdata/logs/turn 136K /home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a 88K /home/yellowtent/platformdata/logs/24bff5cf-736c-49a2-a0aa-87e59131c4b6 76K /home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5 64K /home/yellowtent/platformdata/logs/mail- I then looked at the new #1 consumer app ID e50f78c7-da6e-4036-a1a3-0cc934448cf1 which was associated to Matomo, and it too was missing a logrotate config file.
I realized at this point that Matomo had an app update associated with it currently in my Dashboard. Yesterday when I only saw 5 apps, the vast majority of the apps at that time also had updates pending (WordPress makes up around 12-15 of my app instances, then Roundcube was one, plus Bitwarden was another, so had roughly 14-17 or so apps with pending updates at that time which I then updated last night). This makes me suspicious that possibly an app having a pending app update somehow removes it's logrotate configuration file. It kind of explains why I only saw a few apps listed yesterday, as well as why it recovered so much more disk space today when all those apps were updated last night and today their logrotates suddenly existed again. To be fair though, it doesn't explain why Radicale didn't have one today as Radicale hasn't been updated in a while and no pending app updates for it, so that's the one app that goes against my theory.
Either way... I see issues here with missing logrotate configs. And that's of course on top of the disk space concern, however I suppose if everything was running properly with logrotate my main concern may be a moot point by then. So the root cause of all of all of this may simply be missing logrotate configurations for apps.
I should add... I haven't updated Matomo yet, in case the Cloudron team wanted to connect to my server to do that update and see if logrotate suddenly exists afterwards, in case that helps at all with troubleshooting.
--
Dustin Dauncey
www.d19.ca -
@d19dotca Thanks for the details analysis! So, after you report I decided to look into some of the Cloudrons I have access to and indeed logrotate configs for some apps are missing. Let me debug further, but there is definitely some bug.
-
I just wanted to say I noticed an excessive use of logs again on my server. I guess this wasn't fixed yet, eh?
ubuntu@cloudron:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr 12G total 12G /home/yellowtent/platformdata/logs 1.6G /home/yellowtent/platformdata/logs/tasks 919M /home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499 916M /home/yellowtent/platformdata/logs/76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b 804M /home/yellowtent/platformdata/logs/9f9aba4a-f7b1-44d8-859c-c6953027c39e 803M /home/yellowtent/platformdata/logs/00895422-a1ff-4196-8bb8-cb4ff8d6eeaa 801M /home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d 755M /home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719 754M /home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d 735M /home/yellowtent/platformdata/logs/633f2c77-9b33-457a-bc3c-919c87d25ca2 726M /home/yellowtent/platformdata/logs/29354a9c-7e1d-4986-b78e-3510fe73b370 699M /home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a 685M /home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244 561M /home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991 294M /home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5 271M /home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c 85M /home/yellowtent/platformdata/logs/200d36ca-8109-409a-9b56-07f46c2f09bd 83M /home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e 76M /home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20 67M /home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6 57M /home/yellowtent/platformdata/logs/93c19968-5b36-4412-adfe-3a8f338143ec 46M /home/yellowtent/platformdata/logs/95c45c33-bd8a-4fae-9718-509c69456abe 43M /home/yellowtent/platformdata/logs/134ef7f1-46f8-403e-8ac5-a3297b37893f 43M /home/yellowtent/platformdata/logs/1247a348-c8da-4f30-8a4c-3fa014f092a1 36M /home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d 33M /home/yellowtent/platformdata/logs/8315f008-87e9-4f62-9ae8-1938d7b3f9b2 26M /home/yellowtent/platformdata/logs/graphite 25M /home/yellowtent/platformdata/logs/mail 23M /home/yellowtent/platformdata/logs/turn 20M /home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1 19M /home/yellowtent/platformdata/logs/3167ef7d-e4c2-4072-8e11-8de49c5db970 16M /home/yellowtent/platformdata/logs/sftp 14M /home/yellowtent/platformdata/logs/ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7 13M /home/yellowtent/platformdata/logs/38f883dc-4047-4237-a6a1-3ecd77efcde9 9.3M /home/yellowtent/platformdata/logs/collectd 8.9M /home/yellowtent/platformdata/logs/redis-e18fa18b-b176-4b25-bf78-4d82c022b0f5 4.5M /home/yellowtent/platformdata/logs/9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd 3.9M /home/yellowtent/platformdata/logs/0db23e8d-9515-45ad-8739-6cf2a2b1078f 3.7M /home/yellowtent/platformdata/logs/mysql 3.4M /home/yellowtent/platformdata/logs/8732aed0-273e-49f5-837f-cd9c600bf7b9 2.9M /home/yellowtent/platformdata/logs/redis-1247a348-c8da-4f30-8a4c-3fa014f092a1 2.8M /home/yellowtent/platformdata/logs/mongodb 2.7M /home/yellowtent/platformdata/logs/redis-f6b21e27-6d25-4c69-a547-0466e622d719 2.7M /home/yellowtent/platformdata/logs/redis-9f9aba4a-f7b1-44d8-859c-c6953027c39e 2.7M /home/yellowtent/platformdata/logs/redis-95c45c33-bd8a-4fae-9718-509c69456abe 2.7M /home/yellowtent/platformdata/logs/redis-8dba8d47-7890-49d3-8dfa-970e1ec52991 2.7M /home/yellowtent/platformdata/logs/redis-76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b 2.7M /home/yellowtent/platformdata/logs/redis-6e21d87b-ea85-420b-8eed-8b14d601438d 2.7M /home/yellowtent/platformdata/logs/redis-6b3ebc4f-9708-4243-bc21-4c9a01bc3499 2.7M /home/yellowtent/platformdata/logs/redis-6120f84a-0953-4982-9167-ef872c0ee49a 2.7M /home/yellowtent/platformdata/logs/redis-493c4f88-5d4d-40e5-8361-fd7d4541441d 2.7M /home/yellowtent/platformdata/logs/redis-29354a9c-7e1d-4986-b78e-3510fe73b370 2.7M /home/yellowtent/platformdata/logs/redis-200d36ca-8109-409a-9b56-07f46c2f09bd 2.7M /home/yellowtent/platformdata/logs/redis-05db518f-2c45-411b-865f-813dbf85c244 2.7M /home/yellowtent/platformdata/logs/redis-00895422-a1ff-4196-8bb8-cb4ff8d6eeaa 2.6M /home/yellowtent/platformdata/logs/redis-134ef7f1-46f8-403e-8ac5-a3297b37893f 2.6M /home/yellowtent/platformdata/logs/40ae3d28-d67f-453e-895d-0f22b539372c 2.0M /home/yellowtent/platformdata/logs/redis-4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e 1.5M /home/yellowtent/platformdata/logs/redis-4929b73d-cfa4-4246-82a7-4bc8127982f6 1.1M /home/yellowtent/platformdata/logs/redis-21aecd6a-398a-433a-a954-5bcd5f861e20 924K /home/yellowtent/platformdata/logs/redis-93c19968-5b36-4412-adfe-3a8f338143ec 416K /home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a 288K /home/yellowtent/platformdata/logs/updater 168K /home/yellowtent/platformdata/logs/redis-633f2c77-9b33-457a-bc3c-919c87d25ca2 152K /home/yellowtent/platformdata/logs/postgresql 112K /home/yellowtent/platformdata/logs/redis-0db23e8d-9515-45ad-8739-6cf2a2b1078f 4.0K /home/yellowtent/platformdata/logs/crash 4.0K /home/yellowtent/platformdata/logs/backup--
Dustin Dauncey
www.d19.ca -
@d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:
sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
for me,
/home/yellowtent/platformdata/logs/graphitehas more than 580 MB of logs, as well. Is that normal? -
@necrevistonnezr you can delete those files safely, of course you will lose all logs history, but if you don't have any immediate issue on your server besides log size, you can just remove them via ssh.
-
The logs are still being poorly maintained it seems, they account for several GBs of disk space which seems crazy high for just logs:
813M /home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499/app.log 788M /home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6/app.log 712M /home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d/app.log 656M /home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d/app.log 631M /home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a/app.log 629M /home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719/app.log 613M /home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e/app.log 596M /home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20/app.log 578M /home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244/app.log 514M /home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991/app.logIn my case, it seems the same issue I reported back February remains... there are missing logrotate.conf files for the area where there's far too high of log file sizes.
Example: Note that logs for
6b3ebc4f-9708-4243-bc21-4c9a01bc3499was the largest... when I run the following command, there's nothing returned, suggesting the logrotate.conf is missing for this one, and likely all the rest too:ls -alh /home/yellowtent/platformdata/logrotate.d/ | grep 6b3ebc4f-9708-4243-bc21-4c9a01bc3499returns no files.When I do an
ls -alhlisting for the directory, here is all that exists which seems far too few:ls -alh /home/yellowtent/platformdata/logrotate.d total 80K drwxr-xr-x 2 root root 4.0K May 29 05:00 . drwxr-xr-x 15 yellowtent yellowtent 4.0K May 15 07:27 .. -rw-r--r-- 1 root root 946 May 15 20:21 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa -rw-r--r-- 1 root root 946 May 17 22:43 02b2f762-afa4-4278-afaa-5c07ad29715e -rw-r--r-- 1 root root 946 May 20 16:28 200d36ca-8109-409a-9b56-07f46c2f09bd -rw-r--r-- 1 root root 946 May 21 17:35 29354a9c-7e1d-4986-b78e-3510fe73b370 -rw-r--r-- 1 root root 946 May 27 05:00 3167ef7d-e4c2-4072-8e11-8de49c5db970 -rw-r--r-- 1 root root 946 May 29 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a -rw-r--r-- 1 root root 946 May 20 16:23 633f2c77-9b33-457a-bc3c-919c87d25ca2 -rw-r--r-- 1 root root 946 May 17 22:41 64098702-00d1-4d2e-8d27-4597e100d1c9 -rw-r--r-- 1 root root 946 May 23 06:28 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b -rw-r--r-- 1 root root 946 May 16 06:37 9f9aba4a-f7b1-44d8-859c-c6953027c39e -rw-r--r-- 1 root root 230 May 15 09:00 box -rw-r--r-- 1 root root 946 May 17 22:38 c3ddc268-deef-49e0-8e3a-97e174cf0594 -rw-r--r-- 1 root root 946 May 27 20:55 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d -rw-r--r-- 1 root root 946 May 27 21:00 ca7827f8-4686-4818-a821-f92fe4f5726d -rw-r--r-- 1 root root 946 May 18 05:00 e18fa18b-b176-4b25-bf78-4d82c022b0f5 -rw-r--r-- 1 root root 946 May 26 15:44 e50f78c7-da6e-4036-a1a3-0cc934448cf1 -rw-r--r-- 1 root root 946 May 24 00:24 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9 -rw-r--r-- 1 root root 1.1K May 15 09:00 platform--
Dustin Dauncey
www.d19.ca -
@nebulon said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:
@d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?
No, unfortunately not. That has never been the case either, wish it was that easy.
haha. I reported earlier in this thread that re-saving an app with from Location tab will regenerate the logrotate configs for the app. They just go missing randomly, no idea why they disappear, but I at least know how to get them back. Unfortunately I am not aware of any of this until I randomly check sizing on my server and find incredibly large log files which then leads me to discover the apps are missing logrotate configs. Saving the app again so it sort of rebuilds itself will regenerate it. I just looked now and there are a bunch more, for example: I didn't do anything beyond restoring the Cloudron backup to a new Cloudron instance in the Vultr Marketplace last night:ls -alh /home/yellowtent/platformdata/logrotate.d total 156K drwxr-xr-x 2 root root 4.0K May 31 03:54 . drwxr-xr-x 15 yellowtent yellowtent 4.0K May 30 06:27 .. -rw-r--r-- 1 root root 946 May 30 06:28 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa -rw-r--r-- 1 root root 946 May 30 06:28 02b2f762-afa4-4278-afaa-5c07ad29715e -rw-r--r-- 1 root root 946 May 30 06:28 05db518f-2c45-411b-865f-813dbf85c244 -rw-r--r-- 1 root root 946 May 30 06:28 1247a348-c8da-4f30-8a4c-3fa014f092a1 -rw-r--r-- 1 root root 946 May 30 06:28 134ef7f1-46f8-403e-8ac5-a3297b37893f -rw-r--r-- 1 root root 946 May 30 06:28 200d36ca-8109-409a-9b56-07f46c2f09bd -rw-r--r-- 1 root root 946 May 30 06:28 21aecd6a-398a-433a-a954-5bcd5f861e20 -rw-r--r-- 1 root root 946 May 30 06:28 29354a9c-7e1d-4986-b78e-3510fe73b370 -rw-r--r-- 1 root root 946 May 30 06:28 3167ef7d-e4c2-4072-8e11-8de49c5db970 -rw-r--r-- 1 root root 946 May 30 06:29 38f883dc-4047-4237-a6a1-3ecd77efcde9 -rw-r--r-- 1 root root 946 May 30 06:28 40ae3d28-d67f-453e-895d-0f22b539372c -rw-r--r-- 1 root root 946 May 30 06:29 4929b73d-cfa4-4246-82a7-4bc8127982f6 -rw-r--r-- 1 root root 946 May 30 06:29 492b4264-fea8-4d52-bbe6-1f15ce72c53a -rw-r--r-- 1 root root 946 May 30 06:29 493c4f88-5d4d-40e5-8361-fd7d4541441d -rw-r--r-- 1 root root 946 May 30 06:29 4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e -rw-r--r-- 1 root root 946 May 30 06:29 6120f84a-0953-4982-9167-ef872c0ee49a -rw-r--r-- 1 root root 946 May 30 06:29 633f2c77-9b33-457a-bc3c-919c87d25ca2 -rw-r--r-- 1 root root 946 May 30 06:29 64098702-00d1-4d2e-8d27-4597e100d1c9 -rw-r--r-- 1 root root 946 May 31 03:54 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 -rw-r--r-- 1 root root 946 May 30 06:29 6e21d87b-ea85-420b-8eed-8b14d601438d -rw-r--r-- 1 root root 946 May 30 06:30 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b -rw-r--r-- 1 root root 946 May 30 06:29 8315f008-87e9-4f62-9ae8-1938d7b3f9b2 -rw-r--r-- 1 root root 946 May 30 06:29 8732aed0-273e-49f5-837f-cd9c600bf7b9 -rw-r--r-- 1 root root 946 May 30 06:30 8dba8d47-7890-49d3-8dfa-970e1ec52991 -rw-r--r-- 1 root root 946 May 30 06:30 95c45c33-bd8a-4fae-9718-509c69456abe -rw-r--r-- 1 root root 946 May 30 06:30 9f9aba4a-f7b1-44d8-859c-c6953027c39e -rw-r--r-- 1 root root 946 May 30 06:30 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd -rw-r--r-- 1 root root 230 May 28 17:54 box -rw-r--r-- 1 root root 946 May 30 06:30 c3ddc268-deef-49e0-8e3a-97e174cf0594 -rw-r--r-- 1 root root 946 May 30 06:31 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d -rw-r--r-- 1 root root 946 May 30 06:32 d3121e48-4196-48fe-907d-ee831e11ce5c -rw-r--r-- 1 root root 946 May 30 06:32 e18fa18b-b176-4b25-bf78-4d82c022b0f5 -rw-r--r-- 1 root root 946 May 30 06:33 e50f78c7-da6e-4036-a1a3-0cc934448cf1 -rw-r--r-- 1 root root 946 May 30 06:31 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7 -rw-r--r-- 1 root root 946 May 30 06:31 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9 -rw-r--r-- 1 root root 946 May 30 06:31 f6b21e27-6d25-4c69-a547-0466e622d719 -rw-r--r-- 1 root root 1.1K May 28 17:54 platform -
This is still an issue by the way. Noticed my disk space accumulating quickly. There's practically no logrotate's configured. Is this due to 6.3 though perhaps?
ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d total 20K drwxr-xr-x 2 root root 4.0K Jun 30 15:28 . drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 .. -rw-r--r-- 1 root root 946 Jun 30 15:28 38f883dc-4047-4237-a6a1-3ecd77efcde9 -rw-r--r-- 1 root root 230 Jun 30 05:05 box -rw-r--r-- 1 root root 1.1K Jun 30 05:05 platformubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr 4.1G total 4.1G /home/yellowtent/platformdata/ 2.2G /home/yellowtent/platformdata/mysql 891M /home/yellowtent/platformdata/logs 470M /home/yellowtent/platformdata/graphite 421M /home/yellowtent/platformdata/mongodb 97M /home/yellowtent/platformdata/postgresql 54M /home/yellowtent/platformdata/backup 15M /home/yellowtent/platformdata/redis 4.0M /home/yellowtent/platformdata/nginx 192K /home/yellowtent/platformdata/update 176K /home/yellowtent/platformdata/collectd 36K /home/yellowtent/platformdata/addons 16K /home/yellowtent/platformdata/sftp 16K /home/yellowtent/platformdata/logrotate.d 8.0K /home/yellowtent/platformdata/firewall 4.0K /home/yellowtent/platformdata/acmeTo re-iterate, the logrotate files get generated using the workaround of hitting "Save" on the "Locations" tab of each app, but this is a manual process and I now have to do this for about 30 apps, causing a bit of downtime with each one.
Then I need to run the logrotate function manually to save the disk space again.I'm actually a little surprised this would be an issue when all the containers already had to be regenerated as part of the 6.3 upgrade.
@staff - I'm concerned that this defect hasn't been fixed since it was reported quite a while back. Any possibility of getting to the bottom of this one sooner than later?
-
I took the time to manually re-save each app Location tab to recreate the logrotate file as per the workaround, which resulted in a bit of downtime for each app.
I then ran the
sudo logrotate -f /home/yellowtent/platformdata/logrotate.d/*command and saved roughly half a GB in logs directory alone (I see the mongodb and graphite folders among others went much smaller too, presumably due to restarting of the apps?). So basically 500 MB in extra logs and that's only on a server that's been running for only about a week, presumably would have grown much further in just another week or two too if the logrotate files weren't regenerated.I really hope this will be fixed in the next bug-fix release.
ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr 3.2G total 3.2G /home/yellowtent/platformdata/ 2.2G /home/yellowtent/platformdata/mysql 422M /home/yellowtent/platformdata/mongodb 229M /home/yellowtent/platformdata/graphite 171M /home/yellowtent/platformdata/logs 97M /home/yellowtent/platformdata/postgresql 54M /home/yellowtent/platformdata/backup 17M /home/yellowtent/platformdata/redis 4.0M /home/yellowtent/platformdata/nginx 192K /home/yellowtent/platformdata/update 176K /home/yellowtent/platformdata/collectd 172K /home/yellowtent/platformdata/logrotate.d 36K /home/yellowtent/platformdata/addons 16K /home/yellowtent/platformdata/sftp 8.0K /home/yellowtent/platformdata/firewall 4.0K /home/yellowtent/platformdata/acme -
@girish - I was wanting to follow-up on this. The defect still seems to exist. Right now, despite me running about 41 apps, I only have several in the logrotate.d folder:
ubuntu@my:/home/yellowtent/platformdata$ ls -alh logrotate.d/ total 60K drwxr-xr-x 2 root root 4.0K Sep 7 05:00 . drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 .. -rw-r--r-- 1 root root 946 Aug 22 03:43 3167ef7d-e4c2-4072-8e11-8de49c5db970 -rw-r--r-- 1 root root 946 Aug 23 16:40 4929b73d-cfa4-4246-82a7-4bc8127982f6 -rw-r--r-- 1 root root 946 Sep 7 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a -rw-r--r-- 1 root root 946 Sep 4 03:34 684cf8eb-f80c-4304-b97d-ab915a18cfe7 -rw-r--r-- 1 root root 946 Sep 4 23:07 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 -rw-r--r-- 1 root root 946 Aug 19 23:11 7384a604-7a42-4dc7-86eb-d3f6dd2d546b -rw-r--r-- 1 root root 946 Aug 19 03:25 8dba8d47-7890-49d3-8dfa-970e1ec52991 -rw-r--r-- 1 root root 946 Aug 24 05:48 9c70787a-3529-4b66-be2e-7b1b5aa92b02 -rw-r--r-- 1 root root 946 Aug 19 23:29 b4888539-e835-4f21-97f5-2fc856be6ecf -rw-r--r-- 1 root root 230 Aug 13 06:39 box -rw-r--r-- 1 root root 946 Aug 22 03:05 c93cef28-3cc4-42d8-88d5-0f581dff74a2 -rw-r--r-- 1 root root 946 Sep 4 05:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7 -rw-r--r-- 1 root root 1.1K Aug 13 06:39 platform--
Dustin Dauncey
www.d19.ca -
@d19dotca if you reconfigure the location of one of the missing apps, does the logrotate config file show up afterwards? I am wondering if something is purging them after they are created. Looking at the code I cannot spot why it wouldn't create that file.
-
@nebulon Yes, I think I've mentioned that before too. Re-saving from the Location tab on each app regenerates the logrotate file and finally the logrotates will proceed successfully.
I know @girish confirmed that he saw the same behaviour on one of your (or his) servers too. So this definitely seems to be a Cloudron bug that needs fixing.
This issue has been sitting a long time (confirmed/reproduced in February 2021), so hoping this gets resolved soon because this keeps causing issues on my server when it comes to disk space consumption alerts.
-
@nebulon @girish - This is still an issue by the way. Hoping this fix comes soon! I just checked my server after migrating it over the weekend, and I only have 2 apps in the
logrotate.ddirectory.
Having to go through 35 apps now to reinitialize that for proper log maintenance.--
Dustin Dauncey
www.d19.ca -
@d19dotca I found the issue finally! After a cloudron update, all the old logrotate configs are removed. This was added many releases ago to regenerate new logrotate configs and we forgot to remove the line after that release. Fixed it now.
-
@girish yay! Haha, thank you so much!!! That one was really annoying even though minor. Lol. It happens after a Cloudron migration (not just upgrade) and stuff too FYI but I’m assuming you already knew that.
Edit: actually maybe not… I also had upgraded Cloudron the very next day so I likely just saw the result of that instead. Ignore me.
glad it’s fixed though!!
