Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?

Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?

Scheduled Pinned Locked Moved Solved Support
logging
26 Posts 6 Posters 2.1k Views 7 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • girishG girish

    I have to admin I haven't thought about how much logs to retain. It really depends on the app and the incident to determine how much longs one needs. The current settings are simply values that seemed reasonable.

    @d19dotca It is not correct that only some of the apps have logrotate configs. Can you pick some missing app and then go to Location view and click save? This will re-create the logrotate config for them. BTW, is this after a full server restore? Maybe there is some bug here.

    d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by
    #7

    @girish Yeah I definitely think there's room for improvement there. Personally I think I'd set the default to be whatever you most often see being reported (or more specifically how quickly you see it being reported). I.e. do you see issues being reported from a month or two ago? Or do you see them usually being reported within hours to a week of the incident noted in the logs? I'd assume the latter, meaning we likely don't need to be keeping so many logs. And then when there's a rare issue that comes up where we need more logs, the person reporting it can simply make that quick change in the logrotate.conf file to help you guys out to debug more. That's my two cents anyways. Personally I don't think there's any use in keeping logs for more than a few days to a week in most cases.

    Glad to know that there should be a folder for each app. I definitely only have 5 in there right now which isn't even close to the ~20 I have running, so I will try that out on a few and get back to you then if that helps at all.

    The last time I would have restored this particular server would have been last summer sometime, so I would hope that'd have been fixed by now over time if there was a bug in there since almost every app has been updated since then too. Maybe app package updates need to also trigger that process to ensure it has a custom logrotate directory for itself?

    I'll update once I've tested that more. Thanks for the help, Girish. πŸ™‚

    --
    Dustin Dauncey
    www.d19.ca

    girishG 1 Reply Last reply
    0
    • d19dotcaD d19dotca

      @girish Yeah I definitely think there's room for improvement there. Personally I think I'd set the default to be whatever you most often see being reported (or more specifically how quickly you see it being reported). I.e. do you see issues being reported from a month or two ago? Or do you see them usually being reported within hours to a week of the incident noted in the logs? I'd assume the latter, meaning we likely don't need to be keeping so many logs. And then when there's a rare issue that comes up where we need more logs, the person reporting it can simply make that quick change in the logrotate.conf file to help you guys out to debug more. That's my two cents anyways. Personally I don't think there's any use in keeping logs for more than a few days to a week in most cases.

      Glad to know that there should be a folder for each app. I definitely only have 5 in there right now which isn't even close to the ~20 I have running, so I will try that out on a few and get back to you then if that helps at all.

      The last time I would have restored this particular server would have been last summer sometime, so I would hope that'd have been fixed by now over time if there was a bug in there since almost every app has been updated since then too. Maybe app package updates need to also trigger that process to ensure it has a custom logrotate directory for itself?

      I'll update once I've tested that more. Thanks for the help, Girish. πŸ™‚

      girishG Offline
      girishG Offline
      girish
      Staff
      wrote on last edited by
      #8

      @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

      Glad to know that there should be a folder for each app

      There's a config file for each app btw and not a folder. It will like <appid>.conf

      d19dotcaD 2 Replies Last reply
      0
      • girishG girish

        @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

        Glad to know that there should be a folder for each app

        There's a config file for each app btw and not a folder. It will like <appid>.conf

        d19dotcaD Offline
        d19dotcaD Offline
        d19dotca
        wrote on last edited by
        #9

        @girish Ah sorry, wrong terminology on my part. Thanks for clarifying.

        --
        Dustin Dauncey
        www.d19.ca

        1 Reply Last reply
        0
        • girishG girish

          @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

          Glad to know that there should be a folder for each app

          There's a config file for each app btw and not a folder. It will like <appid>.conf

          d19dotcaD Offline
          d19dotcaD Offline
          d19dotca
          wrote on last edited by d19dotca
          #10

          @girish Okay... I think I found a possible root cause of the issue (though it shouldn't be a big deal long-term, would only impact why I saw only 5 apps yesterday)... I believe the root cause is when an app has a pending update (the green up arrow next to it) then the logrotate mysteriously disappears. Here's why I feel this way and my detailed testing and investigation here...

          1. I forced the logrotate again today and it somehow cleared up another large batch of space, taking it from the leftover nearly 2 GB yesterday to just under 600 MB today.
          ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
          3.3G	total
          3.3G	/home/yellowtent/platformdata/
          1.8G	/home/yellowtent/platformdata/mysql
          592M	/home/yellowtent/platformdata/logs
          522M	/home/yellowtent/platformdata/mongodb
          358M	/home/yellowtent/platformdata/graphite
          89M	/home/yellowtent/platformdata/postgresql
          7.6M	/home/yellowtent/platformdata/redis
          724K	/home/yellowtent/platformdata/nginx
          164K	/home/yellowtent/platformdata/update
          128K	/home/yellowtent/platformdata/collectd
          116K	/home/yellowtent/platformdata/backup
          104K	/home/yellowtent/platformdata/logrotate.d
          32K	/home/yellowtent/platformdata/addons
          8.0K	/home/yellowtent/platformdata/acme
          
          1. Realized that when I listed out the /home/yellowtent/platformdata/logrotate.d/ directory, I suddenly had many more app configs listed in there than there was yesterday. Yesterday there was 5 apps only, today there were many (if not all of them at quick glance):
          ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d/
          total 108K
          drwxr-xr-x  2 root       root       4.0K Feb  9 04:22 .
          drwxr-xr-x 15 yellowtent yellowtent 4.0K Dec  9 04:03 ..
          -rw-r--r--  1 root       root        946 Feb  9 04:13 0e55f920-98bd-4ff6-b591-7254c28451c0
          -rw-r--r--  1 root       root        946 Feb  9 04:22 11c6fe46-82ff-4ea4-b816-3b78a962d048
          -rw-r--r--  1 root       root        946 Feb  9 04:14 1c4fd6e4-09ed-4628-a2f4-0938312536f5
          -rw-r--r--  1 root       root        946 Feb  9 04:13 24bff5cf-736c-49a2-a0aa-87e59131c4b6
          -rw-r--r--  1 root       root        946 Feb  9 04:12 28f810a3-2d17-4e45-b0f8-e581528ac4ba
          -rw-r--r--  1 root       root        946 Feb  9 04:10 3167ef7d-e4c2-4072-8e11-8de49c5db970
          -rw-r--r--  1 root       root        946 Feb  9 04:08 38f883dc-4047-4237-a6a1-3ecd77efcde9
          -rw-r--r--  1 root       root        946 Feb  9 04:09 47164e82-f1d2-4fae-9959-507062614906
          -rw-r--r--  1 root       root        946 Feb  9 04:14 83d329e1-6571-46e5-b14f-07477b8f2aac
          -rw-r--r--  1 root       root        946 Feb  3 04:00 8732aed0-273e-49f5-837f-cd9c600bf7b9
          -rw-r--r--  1 root       root        946 Feb  3 04:00 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
          -rw-r--r--  1 root       root        946 Feb  9 04:12 b6fa7dcd-1d93-4786-82e0-90a634f8c3a7
          -rw-r--r--  1 root       root        946 Feb  9 04:09 be3840f0-458e-4bd8-8d18-34120ebdfce4
          -rw-r--r--  1 root       root        230 Feb  2 16:21 box
          -rw-r--r--  1 root       root        946 Feb  9 04:15 c6a63a87-aa45-4281-b4ad-e65c85817f5d
          -rw-r--r--  1 root       root        946 Feb  9 04:15 cf1d5c7c-d4f0-4c32-ada8-e25ab025bd3d
          -rw-r--r--  1 root       root        946 Feb  9 04:09 d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d
          -rw-r--r--  1 root       root        946 Feb  9 04:14 d2df514f-18f6-4e4e-84a1-69bd6360765c
          -rw-r--r--  1 root       root        946 Feb  9 04:14 d2e46149-3287-4944-8e86-1c9a03e12641
          -rw-r--r--  1 root       root        946 Feb  9 04:14 dc527d19-aced-44fa-bae1-a0b22c358a44
          -rw-r--r--  1 root       root        946 Feb  9 04:12 de175741-ee83-4085-b04f-a0048cbdf493
          -rw-r--r--  1 root       root        946 Feb  3 04:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
          -rw-r--r--  1 root       root        946 Feb  9 04:13 f1fe6d67-0897-4063-bc76-0ef7256bac7a
          -rw-r--r--  1 root       root        946 Feb  9 04:14 f2dbde27-0159-4e28-803e-7a3fc21450fb
          -rw-r--r--  1 root       root       1.1K Feb  2 16:21 platform
          
          1. Digged further into the logs usage and found that one app in particular was using significantly more than the rest:
          ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
          592M	total
          592M	/home/yellowtent/platformdata/logs
          204M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
          76M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
          72M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
          37M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
          22M	/home/yellowtent/platformdata/logs/turn
          11M	/home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a
          11M	/home/yellowtent/platformdata/logs/11c6fe46-82ff-4ea4-b816-3b78a962d048
          11M	/home/yellowtent/platformdata/logs/0e55f920-98bd-4ff6-b591-7254c28451c0
          9.8M	/home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5
          9.6M	/home/yellowtent/platformdata/logs/47164e82-f1d2-4fae-9959-507062614906
          9.4M	/home/yellowtent/platformdata/logs/f2dbde27-0159-4e28-803e-7a3fc21450fb
          9.1M	/home/yellowtent/platformdata/logs/c6a63a87-aa45-4281-b4ad-e65c85817f5d
          8.6M	/home/yellowtent/platformdata/logs/dc527d19-aced-44fa-bae1-a0b22c358a44
          8.5M	/home/yellowtent/platformdata/logs/d2e46149-3287-4944-8e86-1c9a03e12641
          8.4M	/home/yellowtent/platformdata/logs/d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d
          8.1M	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
          7.2M	/home/yellowtent/platformdata/logs/mail
          
          1. I noticed that app d3121e48-4196-48fe-907d-ee831e11ce5c was the Radicale card/calDAV app. I then checked the list of logrotate app configs and sure enough that one was missing from the list as you can see above in #2.
          2. I followed the steps suggested by @girish and that was then successful in creating the new logrotate for Radicale app, and then I ran logrotate again and it of course cleared even more space now. This lead me to see the following usage afterwards (took the Radicale log usage from 204 MB to 53 MB now):
          ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
          272M	total
          272M	/home/yellowtent/platformdata/logs
          76M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
          72M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
          53M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
          37M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
          8.1M	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
          368K	/home/yellowtent/platformdata/logs/turn
          136K	/home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a
          88K	/home/yellowtent/platformdata/logs/24bff5cf-736c-49a2-a0aa-87e59131c4b6
          76K	/home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5
          64K	/home/yellowtent/platformdata/logs/mail
          
          1. I then looked at the new #1 consumer app ID e50f78c7-da6e-4036-a1a3-0cc934448cf1 which was associated to Matomo, and it too was missing a logrotate config file.

          I realized at this point that Matomo had an app update associated with it currently in my Dashboard. Yesterday when I only saw 5 apps, the vast majority of the apps at that time also had updates pending (WordPress makes up around 12-15 of my app instances, then Roundcube was one, plus Bitwarden was another, so had roughly 14-17 or so apps with pending updates at that time which I then updated last night). This makes me suspicious that possibly an app having a pending app update somehow removes it's logrotate configuration file. It kind of explains why I only saw a few apps listed yesterday, as well as why it recovered so much more disk space today when all those apps were updated last night and today their logrotates suddenly existed again. To be fair though, it doesn't explain why Radicale didn't have one today as Radicale hasn't been updated in a while and no pending app updates for it, so that's the one app that goes against my theory.

          Either way... I see issues here with missing logrotate configs. And that's of course on top of the disk space concern, however I suppose if everything was running properly with logrotate my main concern may be a moot point by then. So the root cause of all of all of this may simply be missing logrotate configurations for apps.

          I should add... I haven't updated Matomo yet, in case the Cloudron team wanted to connect to my server to do that update and see if logrotate suddenly exists afterwards, in case that helps at all with troubleshooting.

          --
          Dustin Dauncey
          www.d19.ca

          girishG 1 Reply Last reply
          4
          • d19dotcaD d19dotca

            @girish Okay... I think I found a possible root cause of the issue (though it shouldn't be a big deal long-term, would only impact why I saw only 5 apps yesterday)... I believe the root cause is when an app has a pending update (the green up arrow next to it) then the logrotate mysteriously disappears. Here's why I feel this way and my detailed testing and investigation here...

            1. I forced the logrotate again today and it somehow cleared up another large batch of space, taking it from the leftover nearly 2 GB yesterday to just under 600 MB today.
            ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
            3.3G	total
            3.3G	/home/yellowtent/platformdata/
            1.8G	/home/yellowtent/platformdata/mysql
            592M	/home/yellowtent/platformdata/logs
            522M	/home/yellowtent/platformdata/mongodb
            358M	/home/yellowtent/platformdata/graphite
            89M	/home/yellowtent/platformdata/postgresql
            7.6M	/home/yellowtent/platformdata/redis
            724K	/home/yellowtent/platformdata/nginx
            164K	/home/yellowtent/platformdata/update
            128K	/home/yellowtent/platformdata/collectd
            116K	/home/yellowtent/platformdata/backup
            104K	/home/yellowtent/platformdata/logrotate.d
            32K	/home/yellowtent/platformdata/addons
            8.0K	/home/yellowtent/platformdata/acme
            
            1. Realized that when I listed out the /home/yellowtent/platformdata/logrotate.d/ directory, I suddenly had many more app configs listed in there than there was yesterday. Yesterday there was 5 apps only, today there were many (if not all of them at quick glance):
            ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d/
            total 108K
            drwxr-xr-x  2 root       root       4.0K Feb  9 04:22 .
            drwxr-xr-x 15 yellowtent yellowtent 4.0K Dec  9 04:03 ..
            -rw-r--r--  1 root       root        946 Feb  9 04:13 0e55f920-98bd-4ff6-b591-7254c28451c0
            -rw-r--r--  1 root       root        946 Feb  9 04:22 11c6fe46-82ff-4ea4-b816-3b78a962d048
            -rw-r--r--  1 root       root        946 Feb  9 04:14 1c4fd6e4-09ed-4628-a2f4-0938312536f5
            -rw-r--r--  1 root       root        946 Feb  9 04:13 24bff5cf-736c-49a2-a0aa-87e59131c4b6
            -rw-r--r--  1 root       root        946 Feb  9 04:12 28f810a3-2d17-4e45-b0f8-e581528ac4ba
            -rw-r--r--  1 root       root        946 Feb  9 04:10 3167ef7d-e4c2-4072-8e11-8de49c5db970
            -rw-r--r--  1 root       root        946 Feb  9 04:08 38f883dc-4047-4237-a6a1-3ecd77efcde9
            -rw-r--r--  1 root       root        946 Feb  9 04:09 47164e82-f1d2-4fae-9959-507062614906
            -rw-r--r--  1 root       root        946 Feb  9 04:14 83d329e1-6571-46e5-b14f-07477b8f2aac
            -rw-r--r--  1 root       root        946 Feb  3 04:00 8732aed0-273e-49f5-837f-cd9c600bf7b9
            -rw-r--r--  1 root       root        946 Feb  3 04:00 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
            -rw-r--r--  1 root       root        946 Feb  9 04:12 b6fa7dcd-1d93-4786-82e0-90a634f8c3a7
            -rw-r--r--  1 root       root        946 Feb  9 04:09 be3840f0-458e-4bd8-8d18-34120ebdfce4
            -rw-r--r--  1 root       root        230 Feb  2 16:21 box
            -rw-r--r--  1 root       root        946 Feb  9 04:15 c6a63a87-aa45-4281-b4ad-e65c85817f5d
            -rw-r--r--  1 root       root        946 Feb  9 04:15 cf1d5c7c-d4f0-4c32-ada8-e25ab025bd3d
            -rw-r--r--  1 root       root        946 Feb  9 04:09 d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d
            -rw-r--r--  1 root       root        946 Feb  9 04:14 d2df514f-18f6-4e4e-84a1-69bd6360765c
            -rw-r--r--  1 root       root        946 Feb  9 04:14 d2e46149-3287-4944-8e86-1c9a03e12641
            -rw-r--r--  1 root       root        946 Feb  9 04:14 dc527d19-aced-44fa-bae1-a0b22c358a44
            -rw-r--r--  1 root       root        946 Feb  9 04:12 de175741-ee83-4085-b04f-a0048cbdf493
            -rw-r--r--  1 root       root        946 Feb  3 04:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
            -rw-r--r--  1 root       root        946 Feb  9 04:13 f1fe6d67-0897-4063-bc76-0ef7256bac7a
            -rw-r--r--  1 root       root        946 Feb  9 04:14 f2dbde27-0159-4e28-803e-7a3fc21450fb
            -rw-r--r--  1 root       root       1.1K Feb  2 16:21 platform
            
            1. Digged further into the logs usage and found that one app in particular was using significantly more than the rest:
            ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
            592M	total
            592M	/home/yellowtent/platformdata/logs
            204M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
            76M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
            72M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
            37M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
            22M	/home/yellowtent/platformdata/logs/turn
            11M	/home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a
            11M	/home/yellowtent/platformdata/logs/11c6fe46-82ff-4ea4-b816-3b78a962d048
            11M	/home/yellowtent/platformdata/logs/0e55f920-98bd-4ff6-b591-7254c28451c0
            9.8M	/home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5
            9.6M	/home/yellowtent/platformdata/logs/47164e82-f1d2-4fae-9959-507062614906
            9.4M	/home/yellowtent/platformdata/logs/f2dbde27-0159-4e28-803e-7a3fc21450fb
            9.1M	/home/yellowtent/platformdata/logs/c6a63a87-aa45-4281-b4ad-e65c85817f5d
            8.6M	/home/yellowtent/platformdata/logs/dc527d19-aced-44fa-bae1-a0b22c358a44
            8.5M	/home/yellowtent/platformdata/logs/d2e46149-3287-4944-8e86-1c9a03e12641
            8.4M	/home/yellowtent/platformdata/logs/d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d
            8.1M	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
            7.2M	/home/yellowtent/platformdata/logs/mail
            
            1. I noticed that app d3121e48-4196-48fe-907d-ee831e11ce5c was the Radicale card/calDAV app. I then checked the list of logrotate app configs and sure enough that one was missing from the list as you can see above in #2.
            2. I followed the steps suggested by @girish and that was then successful in creating the new logrotate for Radicale app, and then I ran logrotate again and it of course cleared even more space now. This lead me to see the following usage afterwards (took the Radicale log usage from 204 MB to 53 MB now):
            ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
            272M	total
            272M	/home/yellowtent/platformdata/logs
            76M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
            72M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
            53M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
            37M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
            8.1M	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
            368K	/home/yellowtent/platformdata/logs/turn
            136K	/home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a
            88K	/home/yellowtent/platformdata/logs/24bff5cf-736c-49a2-a0aa-87e59131c4b6
            76K	/home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5
            64K	/home/yellowtent/platformdata/logs/mail
            
            1. I then looked at the new #1 consumer app ID e50f78c7-da6e-4036-a1a3-0cc934448cf1 which was associated to Matomo, and it too was missing a logrotate config file.

            I realized at this point that Matomo had an app update associated with it currently in my Dashboard. Yesterday when I only saw 5 apps, the vast majority of the apps at that time also had updates pending (WordPress makes up around 12-15 of my app instances, then Roundcube was one, plus Bitwarden was another, so had roughly 14-17 or so apps with pending updates at that time which I then updated last night). This makes me suspicious that possibly an app having a pending app update somehow removes it's logrotate configuration file. It kind of explains why I only saw a few apps listed yesterday, as well as why it recovered so much more disk space today when all those apps were updated last night and today their logrotates suddenly existed again. To be fair though, it doesn't explain why Radicale didn't have one today as Radicale hasn't been updated in a while and no pending app updates for it, so that's the one app that goes against my theory.

            Either way... I see issues here with missing logrotate configs. And that's of course on top of the disk space concern, however I suppose if everything was running properly with logrotate my main concern may be a moot point by then. So the root cause of all of all of this may simply be missing logrotate configurations for apps.

            I should add... I haven't updated Matomo yet, in case the Cloudron team wanted to connect to my server to do that update and see if logrotate suddenly exists afterwards, in case that helps at all with troubleshooting.

            girishG Offline
            girishG Offline
            girish
            Staff
            wrote on last edited by
            #11

            @d19dotca Thanks for the details analysis! So, after you report I decided to look into some of the Cloudrons I have access to and indeed logrotate configs for some apps are missing. Let me debug further, but there is definitely some bug.

            1 Reply Last reply
            3
            • d19dotcaD Offline
              d19dotcaD Offline
              d19dotca
              wrote on last edited by
              #12

              I just wanted to say I noticed an excessive use of logs again on my server. I guess this wasn't fixed yet, eh?

              ubuntu@cloudron:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
              12G	total
              12G	/home/yellowtent/platformdata/logs
              1.6G	/home/yellowtent/platformdata/logs/tasks
              919M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499
              916M	/home/yellowtent/platformdata/logs/76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
              804M	/home/yellowtent/platformdata/logs/9f9aba4a-f7b1-44d8-859c-c6953027c39e
              803M	/home/yellowtent/platformdata/logs/00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
              801M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d
              755M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719
              754M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d
              735M	/home/yellowtent/platformdata/logs/633f2c77-9b33-457a-bc3c-919c87d25ca2
              726M	/home/yellowtent/platformdata/logs/29354a9c-7e1d-4986-b78e-3510fe73b370
              699M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a
              685M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244
              561M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991
              294M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
              271M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
              85M	/home/yellowtent/platformdata/logs/200d36ca-8109-409a-9b56-07f46c2f09bd
              83M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
              76M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20
              67M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6
              57M	/home/yellowtent/platformdata/logs/93c19968-5b36-4412-adfe-3a8f338143ec
              46M	/home/yellowtent/platformdata/logs/95c45c33-bd8a-4fae-9718-509c69456abe
              43M	/home/yellowtent/platformdata/logs/134ef7f1-46f8-403e-8ac5-a3297b37893f
              43M	/home/yellowtent/platformdata/logs/1247a348-c8da-4f30-8a4c-3fa014f092a1
              36M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
              33M	/home/yellowtent/platformdata/logs/8315f008-87e9-4f62-9ae8-1938d7b3f9b2
              26M	/home/yellowtent/platformdata/logs/graphite
              25M	/home/yellowtent/platformdata/logs/mail
              23M	/home/yellowtent/platformdata/logs/turn
              20M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
              19M	/home/yellowtent/platformdata/logs/3167ef7d-e4c2-4072-8e11-8de49c5db970
              16M	/home/yellowtent/platformdata/logs/sftp
              14M	/home/yellowtent/platformdata/logs/ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
              13M	/home/yellowtent/platformdata/logs/38f883dc-4047-4237-a6a1-3ecd77efcde9
              9.3M	/home/yellowtent/platformdata/logs/collectd
              8.9M	/home/yellowtent/platformdata/logs/redis-e18fa18b-b176-4b25-bf78-4d82c022b0f5
              4.5M	/home/yellowtent/platformdata/logs/9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
              3.9M	/home/yellowtent/platformdata/logs/0db23e8d-9515-45ad-8739-6cf2a2b1078f
              3.7M	/home/yellowtent/platformdata/logs/mysql
              3.4M	/home/yellowtent/platformdata/logs/8732aed0-273e-49f5-837f-cd9c600bf7b9
              2.9M	/home/yellowtent/platformdata/logs/redis-1247a348-c8da-4f30-8a4c-3fa014f092a1
              2.8M	/home/yellowtent/platformdata/logs/mongodb
              2.7M	/home/yellowtent/platformdata/logs/redis-f6b21e27-6d25-4c69-a547-0466e622d719
              2.7M	/home/yellowtent/platformdata/logs/redis-9f9aba4a-f7b1-44d8-859c-c6953027c39e
              2.7M	/home/yellowtent/platformdata/logs/redis-95c45c33-bd8a-4fae-9718-509c69456abe
              2.7M	/home/yellowtent/platformdata/logs/redis-8dba8d47-7890-49d3-8dfa-970e1ec52991
              2.7M	/home/yellowtent/platformdata/logs/redis-76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
              2.7M	/home/yellowtent/platformdata/logs/redis-6e21d87b-ea85-420b-8eed-8b14d601438d
              2.7M	/home/yellowtent/platformdata/logs/redis-6b3ebc4f-9708-4243-bc21-4c9a01bc3499
              2.7M	/home/yellowtent/platformdata/logs/redis-6120f84a-0953-4982-9167-ef872c0ee49a
              2.7M	/home/yellowtent/platformdata/logs/redis-493c4f88-5d4d-40e5-8361-fd7d4541441d
              2.7M	/home/yellowtent/platformdata/logs/redis-29354a9c-7e1d-4986-b78e-3510fe73b370
              2.7M	/home/yellowtent/platformdata/logs/redis-200d36ca-8109-409a-9b56-07f46c2f09bd
              2.7M	/home/yellowtent/platformdata/logs/redis-05db518f-2c45-411b-865f-813dbf85c244
              2.7M	/home/yellowtent/platformdata/logs/redis-00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
              2.6M	/home/yellowtent/platformdata/logs/redis-134ef7f1-46f8-403e-8ac5-a3297b37893f
              2.6M	/home/yellowtent/platformdata/logs/40ae3d28-d67f-453e-895d-0f22b539372c
              2.0M	/home/yellowtent/platformdata/logs/redis-4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
              1.5M	/home/yellowtent/platformdata/logs/redis-4929b73d-cfa4-4246-82a7-4bc8127982f6
              1.1M	/home/yellowtent/platformdata/logs/redis-21aecd6a-398a-433a-a954-5bcd5f861e20
              924K	/home/yellowtent/platformdata/logs/redis-93c19968-5b36-4412-adfe-3a8f338143ec
              416K	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
              288K	/home/yellowtent/platformdata/logs/updater
              168K	/home/yellowtent/platformdata/logs/redis-633f2c77-9b33-457a-bc3c-919c87d25ca2
              152K	/home/yellowtent/platformdata/logs/postgresql
              112K	/home/yellowtent/platformdata/logs/redis-0db23e8d-9515-45ad-8739-6cf2a2b1078f
              4.0K	/home/yellowtent/platformdata/logs/crash
              4.0K	/home/yellowtent/platformdata/logs/backup
              

              --
              Dustin Dauncey
              www.d19.ca

              necrevistonnezrN 1 Reply Last reply
              0
              • d19dotcaD d19dotca

                I just wanted to say I noticed an excessive use of logs again on my server. I guess this wasn't fixed yet, eh?

                ubuntu@cloudron:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
                12G	total
                12G	/home/yellowtent/platformdata/logs
                1.6G	/home/yellowtent/platformdata/logs/tasks
                919M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499
                916M	/home/yellowtent/platformdata/logs/76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
                804M	/home/yellowtent/platformdata/logs/9f9aba4a-f7b1-44d8-859c-c6953027c39e
                803M	/home/yellowtent/platformdata/logs/00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
                801M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d
                755M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719
                754M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d
                735M	/home/yellowtent/platformdata/logs/633f2c77-9b33-457a-bc3c-919c87d25ca2
                726M	/home/yellowtent/platformdata/logs/29354a9c-7e1d-4986-b78e-3510fe73b370
                699M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a
                685M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244
                561M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991
                294M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
                271M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
                85M	/home/yellowtent/platformdata/logs/200d36ca-8109-409a-9b56-07f46c2f09bd
                83M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
                76M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20
                67M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6
                57M	/home/yellowtent/platformdata/logs/93c19968-5b36-4412-adfe-3a8f338143ec
                46M	/home/yellowtent/platformdata/logs/95c45c33-bd8a-4fae-9718-509c69456abe
                43M	/home/yellowtent/platformdata/logs/134ef7f1-46f8-403e-8ac5-a3297b37893f
                43M	/home/yellowtent/platformdata/logs/1247a348-c8da-4f30-8a4c-3fa014f092a1
                36M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
                33M	/home/yellowtent/platformdata/logs/8315f008-87e9-4f62-9ae8-1938d7b3f9b2
                26M	/home/yellowtent/platformdata/logs/graphite
                25M	/home/yellowtent/platformdata/logs/mail
                23M	/home/yellowtent/platformdata/logs/turn
                20M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
                19M	/home/yellowtent/platformdata/logs/3167ef7d-e4c2-4072-8e11-8de49c5db970
                16M	/home/yellowtent/platformdata/logs/sftp
                14M	/home/yellowtent/platformdata/logs/ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
                13M	/home/yellowtent/platformdata/logs/38f883dc-4047-4237-a6a1-3ecd77efcde9
                9.3M	/home/yellowtent/platformdata/logs/collectd
                8.9M	/home/yellowtent/platformdata/logs/redis-e18fa18b-b176-4b25-bf78-4d82c022b0f5
                4.5M	/home/yellowtent/platformdata/logs/9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
                3.9M	/home/yellowtent/platformdata/logs/0db23e8d-9515-45ad-8739-6cf2a2b1078f
                3.7M	/home/yellowtent/platformdata/logs/mysql
                3.4M	/home/yellowtent/platformdata/logs/8732aed0-273e-49f5-837f-cd9c600bf7b9
                2.9M	/home/yellowtent/platformdata/logs/redis-1247a348-c8da-4f30-8a4c-3fa014f092a1
                2.8M	/home/yellowtent/platformdata/logs/mongodb
                2.7M	/home/yellowtent/platformdata/logs/redis-f6b21e27-6d25-4c69-a547-0466e622d719
                2.7M	/home/yellowtent/platformdata/logs/redis-9f9aba4a-f7b1-44d8-859c-c6953027c39e
                2.7M	/home/yellowtent/platformdata/logs/redis-95c45c33-bd8a-4fae-9718-509c69456abe
                2.7M	/home/yellowtent/platformdata/logs/redis-8dba8d47-7890-49d3-8dfa-970e1ec52991
                2.7M	/home/yellowtent/platformdata/logs/redis-76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
                2.7M	/home/yellowtent/platformdata/logs/redis-6e21d87b-ea85-420b-8eed-8b14d601438d
                2.7M	/home/yellowtent/platformdata/logs/redis-6b3ebc4f-9708-4243-bc21-4c9a01bc3499
                2.7M	/home/yellowtent/platformdata/logs/redis-6120f84a-0953-4982-9167-ef872c0ee49a
                2.7M	/home/yellowtent/platformdata/logs/redis-493c4f88-5d4d-40e5-8361-fd7d4541441d
                2.7M	/home/yellowtent/platformdata/logs/redis-29354a9c-7e1d-4986-b78e-3510fe73b370
                2.7M	/home/yellowtent/platformdata/logs/redis-200d36ca-8109-409a-9b56-07f46c2f09bd
                2.7M	/home/yellowtent/platformdata/logs/redis-05db518f-2c45-411b-865f-813dbf85c244
                2.7M	/home/yellowtent/platformdata/logs/redis-00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
                2.6M	/home/yellowtent/platformdata/logs/redis-134ef7f1-46f8-403e-8ac5-a3297b37893f
                2.6M	/home/yellowtent/platformdata/logs/40ae3d28-d67f-453e-895d-0f22b539372c
                2.0M	/home/yellowtent/platformdata/logs/redis-4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
                1.5M	/home/yellowtent/platformdata/logs/redis-4929b73d-cfa4-4246-82a7-4bc8127982f6
                1.1M	/home/yellowtent/platformdata/logs/redis-21aecd6a-398a-433a-a954-5bcd5f861e20
                924K	/home/yellowtent/platformdata/logs/redis-93c19968-5b36-4412-adfe-3a8f338143ec
                416K	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
                288K	/home/yellowtent/platformdata/logs/updater
                168K	/home/yellowtent/platformdata/logs/redis-633f2c77-9b33-457a-bc3c-919c87d25ca2
                152K	/home/yellowtent/platformdata/logs/postgresql
                112K	/home/yellowtent/platformdata/logs/redis-0db23e8d-9515-45ad-8739-6cf2a2b1078f
                4.0K	/home/yellowtent/platformdata/logs/crash
                4.0K	/home/yellowtent/platformdata/logs/backup
                
                necrevistonnezrN Offline
                necrevistonnezrN Offline
                necrevistonnezr
                wrote on last edited by necrevistonnezr
                #13

                @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

                sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr

                for me, /home/yellowtent/platformdata/logs/graphite has more than 580 MB of logs, as well. Is that normal?

                nebulonN 1 Reply Last reply
                0
                • robiR Offline
                  robiR Offline
                  robi
                  wrote on last edited by
                  #14

                  it's from all the crashing

                  Conscious tech

                  1 Reply Last reply
                  0
                  • necrevistonnezrN necrevistonnezr

                    @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

                    sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr

                    for me, /home/yellowtent/platformdata/logs/graphite has more than 580 MB of logs, as well. Is that normal?

                    nebulonN Offline
                    nebulonN Offline
                    nebulon
                    Staff
                    wrote on last edited by
                    #15

                    @necrevistonnezr you can delete those files safely, of course you will lose all logs history, but if you don't have any immediate issue on your server besides log size, you can just remove them via ssh.

                    1 Reply Last reply
                    0
                    • d19dotcaD Offline
                      d19dotcaD Offline
                      d19dotca
                      wrote on last edited by d19dotca
                      #16

                      The logs are still being poorly maintained it seems, they account for several GBs of disk space which seems crazy high for just logs:

                      813M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499/app.log
                      788M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6/app.log
                      712M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d/app.log
                      656M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d/app.log
                      631M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a/app.log
                      629M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719/app.log
                      613M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e/app.log
                      596M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20/app.log
                      578M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244/app.log
                      514M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991/app.log
                      

                      In my case, it seems the same issue I reported back February remains... there are missing logrotate.conf files for the area where there's far too high of log file sizes.

                      Example: Note that logs for 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 was the largest... when I run the following command, there's nothing returned, suggesting the logrotate.conf is missing for this one, and likely all the rest too: ls -alh /home/yellowtent/platformdata/logrotate.d/ | grep 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 returns no files.

                      When I do an ls -alh listing for the directory, here is all that exists which seems far too few:

                      ls -alh /home/yellowtent/platformdata/logrotate.d
                      total 80K
                      drwxr-xr-x  2 root       root       4.0K May 29 05:00 .
                      drwxr-xr-x 15 yellowtent yellowtent 4.0K May 15 07:27 ..
                      -rw-r--r--  1 root       root        946 May 15 20:21 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
                      -rw-r--r--  1 root       root        946 May 17 22:43 02b2f762-afa4-4278-afaa-5c07ad29715e
                      -rw-r--r--  1 root       root        946 May 20 16:28 200d36ca-8109-409a-9b56-07f46c2f09bd
                      -rw-r--r--  1 root       root        946 May 21 17:35 29354a9c-7e1d-4986-b78e-3510fe73b370
                      -rw-r--r--  1 root       root        946 May 27 05:00 3167ef7d-e4c2-4072-8e11-8de49c5db970
                      -rw-r--r--  1 root       root        946 May 29 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
                      -rw-r--r--  1 root       root        946 May 20 16:23 633f2c77-9b33-457a-bc3c-919c87d25ca2
                      -rw-r--r--  1 root       root        946 May 17 22:41 64098702-00d1-4d2e-8d27-4597e100d1c9
                      -rw-r--r--  1 root       root        946 May 23 06:28 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
                      -rw-r--r--  1 root       root        946 May 16 06:37 9f9aba4a-f7b1-44d8-859c-c6953027c39e
                      -rw-r--r--  1 root       root        230 May 15 09:00 box
                      -rw-r--r--  1 root       root        946 May 17 22:38 c3ddc268-deef-49e0-8e3a-97e174cf0594
                      -rw-r--r--  1 root       root        946 May 27 20:55 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
                      -rw-r--r--  1 root       root        946 May 27 21:00 ca7827f8-4686-4818-a821-f92fe4f5726d
                      -rw-r--r--  1 root       root        946 May 18 05:00 e18fa18b-b176-4b25-bf78-4d82c022b0f5
                      -rw-r--r--  1 root       root        946 May 26 15:44 e50f78c7-da6e-4036-a1a3-0cc934448cf1
                      -rw-r--r--  1 root       root        946 May 24 00:24 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
                      -rw-r--r--  1 root       root       1.1K May 15 09:00 platform
                      

                      --
                      Dustin Dauncey
                      www.d19.ca

                      nebulonN 1 Reply Last reply
                      0
                      • d19dotcaD d19dotca

                        The logs are still being poorly maintained it seems, they account for several GBs of disk space which seems crazy high for just logs:

                        813M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499/app.log
                        788M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6/app.log
                        712M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d/app.log
                        656M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d/app.log
                        631M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a/app.log
                        629M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719/app.log
                        613M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e/app.log
                        596M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20/app.log
                        578M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244/app.log
                        514M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991/app.log
                        

                        In my case, it seems the same issue I reported back February remains... there are missing logrotate.conf files for the area where there's far too high of log file sizes.

                        Example: Note that logs for 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 was the largest... when I run the following command, there's nothing returned, suggesting the logrotate.conf is missing for this one, and likely all the rest too: ls -alh /home/yellowtent/platformdata/logrotate.d/ | grep 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 returns no files.

                        When I do an ls -alh listing for the directory, here is all that exists which seems far too few:

                        ls -alh /home/yellowtent/platformdata/logrotate.d
                        total 80K
                        drwxr-xr-x  2 root       root       4.0K May 29 05:00 .
                        drwxr-xr-x 15 yellowtent yellowtent 4.0K May 15 07:27 ..
                        -rw-r--r--  1 root       root        946 May 15 20:21 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
                        -rw-r--r--  1 root       root        946 May 17 22:43 02b2f762-afa4-4278-afaa-5c07ad29715e
                        -rw-r--r--  1 root       root        946 May 20 16:28 200d36ca-8109-409a-9b56-07f46c2f09bd
                        -rw-r--r--  1 root       root        946 May 21 17:35 29354a9c-7e1d-4986-b78e-3510fe73b370
                        -rw-r--r--  1 root       root        946 May 27 05:00 3167ef7d-e4c2-4072-8e11-8de49c5db970
                        -rw-r--r--  1 root       root        946 May 29 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
                        -rw-r--r--  1 root       root        946 May 20 16:23 633f2c77-9b33-457a-bc3c-919c87d25ca2
                        -rw-r--r--  1 root       root        946 May 17 22:41 64098702-00d1-4d2e-8d27-4597e100d1c9
                        -rw-r--r--  1 root       root        946 May 23 06:28 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
                        -rw-r--r--  1 root       root        946 May 16 06:37 9f9aba4a-f7b1-44d8-859c-c6953027c39e
                        -rw-r--r--  1 root       root        230 May 15 09:00 box
                        -rw-r--r--  1 root       root        946 May 17 22:38 c3ddc268-deef-49e0-8e3a-97e174cf0594
                        -rw-r--r--  1 root       root        946 May 27 20:55 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
                        -rw-r--r--  1 root       root        946 May 27 21:00 ca7827f8-4686-4818-a821-f92fe4f5726d
                        -rw-r--r--  1 root       root        946 May 18 05:00 e18fa18b-b176-4b25-bf78-4d82c022b0f5
                        -rw-r--r--  1 root       root        946 May 26 15:44 e50f78c7-da6e-4036-a1a3-0cc934448cf1
                        -rw-r--r--  1 root       root        946 May 24 00:24 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
                        -rw-r--r--  1 root       root       1.1K May 15 09:00 platform
                        
                        nebulonN Offline
                        nebulonN Offline
                        nebulon
                        Staff
                        wrote on last edited by
                        #17

                        @d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

                        d19dotcaD 1 Reply Last reply
                        0
                        • nebulonN nebulon

                          @d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

                          d19dotcaD Offline
                          d19dotcaD Offline
                          d19dotca
                          wrote on last edited by d19dotca
                          #18

                          @nebulon said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

                          @d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

                          No, unfortunately not. That has never been the case either, wish it was that easy. πŸ˜› haha. I reported earlier in this thread that re-saving an app with from Location tab will regenerate the logrotate configs for the app. They just go missing randomly, no idea why they disappear, but I at least know how to get them back. Unfortunately I am not aware of any of this until I randomly check sizing on my server and find incredibly large log files which then leads me to discover the apps are missing logrotate configs. Saving the app again so it sort of rebuilds itself will regenerate it. I just looked now and there are a bunch more, for example: I didn't do anything beyond restoring the Cloudron backup to a new Cloudron instance in the Vultr Marketplace last night:

                          ls -alh /home/yellowtent/platformdata/logrotate.d
                          total 156K
                          drwxr-xr-x  2 root       root       4.0K May 31 03:54 .
                          drwxr-xr-x 15 yellowtent yellowtent 4.0K May 30 06:27 ..
                          -rw-r--r--  1 root       root        946 May 30 06:28 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
                          -rw-r--r--  1 root       root        946 May 30 06:28 02b2f762-afa4-4278-afaa-5c07ad29715e
                          -rw-r--r--  1 root       root        946 May 30 06:28 05db518f-2c45-411b-865f-813dbf85c244
                          -rw-r--r--  1 root       root        946 May 30 06:28 1247a348-c8da-4f30-8a4c-3fa014f092a1
                          -rw-r--r--  1 root       root        946 May 30 06:28 134ef7f1-46f8-403e-8ac5-a3297b37893f
                          -rw-r--r--  1 root       root        946 May 30 06:28 200d36ca-8109-409a-9b56-07f46c2f09bd
                          -rw-r--r--  1 root       root        946 May 30 06:28 21aecd6a-398a-433a-a954-5bcd5f861e20
                          -rw-r--r--  1 root       root        946 May 30 06:28 29354a9c-7e1d-4986-b78e-3510fe73b370
                          -rw-r--r--  1 root       root        946 May 30 06:28 3167ef7d-e4c2-4072-8e11-8de49c5db970
                          -rw-r--r--  1 root       root        946 May 30 06:29 38f883dc-4047-4237-a6a1-3ecd77efcde9
                          -rw-r--r--  1 root       root        946 May 30 06:28 40ae3d28-d67f-453e-895d-0f22b539372c
                          -rw-r--r--  1 root       root        946 May 30 06:29 4929b73d-cfa4-4246-82a7-4bc8127982f6
                          -rw-r--r--  1 root       root        946 May 30 06:29 492b4264-fea8-4d52-bbe6-1f15ce72c53a
                          -rw-r--r--  1 root       root        946 May 30 06:29 493c4f88-5d4d-40e5-8361-fd7d4541441d
                          -rw-r--r--  1 root       root        946 May 30 06:29 4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
                          -rw-r--r--  1 root       root        946 May 30 06:29 6120f84a-0953-4982-9167-ef872c0ee49a
                          -rw-r--r--  1 root       root        946 May 30 06:29 633f2c77-9b33-457a-bc3c-919c87d25ca2
                          -rw-r--r--  1 root       root        946 May 30 06:29 64098702-00d1-4d2e-8d27-4597e100d1c9
                          -rw-r--r--  1 root       root        946 May 31 03:54 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
                          -rw-r--r--  1 root       root        946 May 30 06:29 6e21d87b-ea85-420b-8eed-8b14d601438d
                          -rw-r--r--  1 root       root        946 May 30 06:30 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
                          -rw-r--r--  1 root       root        946 May 30 06:29 8315f008-87e9-4f62-9ae8-1938d7b3f9b2
                          -rw-r--r--  1 root       root        946 May 30 06:29 8732aed0-273e-49f5-837f-cd9c600bf7b9
                          -rw-r--r--  1 root       root        946 May 30 06:30 8dba8d47-7890-49d3-8dfa-970e1ec52991
                          -rw-r--r--  1 root       root        946 May 30 06:30 95c45c33-bd8a-4fae-9718-509c69456abe
                          -rw-r--r--  1 root       root        946 May 30 06:30 9f9aba4a-f7b1-44d8-859c-c6953027c39e
                          -rw-r--r--  1 root       root        946 May 30 06:30 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
                          -rw-r--r--  1 root       root        230 May 28 17:54 box
                          -rw-r--r--  1 root       root        946 May 30 06:30 c3ddc268-deef-49e0-8e3a-97e174cf0594
                          -rw-r--r--  1 root       root        946 May 30 06:31 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
                          -rw-r--r--  1 root       root        946 May 30 06:32 d3121e48-4196-48fe-907d-ee831e11ce5c
                          -rw-r--r--  1 root       root        946 May 30 06:32 e18fa18b-b176-4b25-bf78-4d82c022b0f5
                          -rw-r--r--  1 root       root        946 May 30 06:33 e50f78c7-da6e-4036-a1a3-0cc934448cf1
                          -rw-r--r--  1 root       root        946 May 30 06:31 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
                          -rw-r--r--  1 root       root        946 May 30 06:31 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
                          -rw-r--r--  1 root       root        946 May 30 06:31 f6b21e27-6d25-4c69-a547-0466e622d719
                          -rw-r--r--  1 root       root       1.1K May 28 17:54 platform
                          

                          --
                          Dustin Dauncey
                          www.d19.ca

                          1 Reply Last reply
                          0
                          • d19dotcaD Offline
                            d19dotcaD Offline
                            d19dotca
                            wrote on last edited by d19dotca
                            #19

                            This is still an issue by the way. Noticed my disk space accumulating quickly. There's practically no logrotate's configured. Is this due to 6.3 though perhaps?

                            ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d
                            total 20K
                            drwxr-xr-x  2 root       root       4.0K Jun 30 15:28 .
                            drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 ..
                            -rw-r--r--  1 root       root        946 Jun 30 15:28 38f883dc-4047-4237-a6a1-3ecd77efcde9
                            -rw-r--r--  1 root       root        230 Jun 30 05:05 box
                            -rw-r--r--  1 root       root       1.1K Jun 30 05:05 platform
                            
                            ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
                            4.1G	total
                            4.1G	/home/yellowtent/platformdata/
                            2.2G	/home/yellowtent/platformdata/mysql
                            891M	/home/yellowtent/platformdata/logs
                            470M	/home/yellowtent/platformdata/graphite
                            421M	/home/yellowtent/platformdata/mongodb
                            97M	/home/yellowtent/platformdata/postgresql
                            54M	/home/yellowtent/platformdata/backup
                            15M	/home/yellowtent/platformdata/redis
                            4.0M	/home/yellowtent/platformdata/nginx
                            192K	/home/yellowtent/platformdata/update
                            176K	/home/yellowtent/platformdata/collectd
                            36K	/home/yellowtent/platformdata/addons
                            16K	/home/yellowtent/platformdata/sftp
                            16K	/home/yellowtent/platformdata/logrotate.d
                            8.0K	/home/yellowtent/platformdata/firewall
                            4.0K	/home/yellowtent/platformdata/acme
                            

                            To re-iterate, the logrotate files get generated using the workaround of hitting "Save" on the "Locations" tab of each app, but this is a manual process and I now have to do this for about 30 apps, causing a bit of downtime with each one. πŸ˜• Then I need to run the logrotate function manually to save the disk space again.

                            I'm actually a little surprised this would be an issue when all the containers already had to be regenerated as part of the 6.3 upgrade.

                            @staff - I'm concerned that this defect hasn't been fixed since it was reported quite a while back. Any possibility of getting to the bottom of this one sooner than later?

                            --
                            Dustin Dauncey
                            www.d19.ca

                            1 Reply Last reply
                            3
                            • d19dotcaD Offline
                              d19dotcaD Offline
                              d19dotca
                              wrote on last edited by
                              #20

                              I took the time to manually re-save each app Location tab to recreate the logrotate file as per the workaround, which resulted in a bit of downtime for each app. πŸ˜•

                              I then ran the sudo logrotate -f /home/yellowtent/platformdata/logrotate.d/* command and saved roughly half a GB in logs directory alone (I see the mongodb and graphite folders among others went much smaller too, presumably due to restarting of the apps?). So basically 500 MB in extra logs and that's only on a server that's been running for only about a week, presumably would have grown much further in just another week or two too if the logrotate files weren't regenerated.

                              I really hope this will be fixed in the next bug-fix release.

                              ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
                              3.2G	total
                              3.2G	/home/yellowtent/platformdata/
                              2.2G	/home/yellowtent/platformdata/mysql
                              422M	/home/yellowtent/platformdata/mongodb
                              229M	/home/yellowtent/platformdata/graphite
                              171M	/home/yellowtent/platformdata/logs
                              97M	/home/yellowtent/platformdata/postgresql
                              54M	/home/yellowtent/platformdata/backup
                              17M	/home/yellowtent/platformdata/redis
                              4.0M	/home/yellowtent/platformdata/nginx
                              192K	/home/yellowtent/platformdata/update
                              176K	/home/yellowtent/platformdata/collectd
                              172K	/home/yellowtent/platformdata/logrotate.d
                              36K	/home/yellowtent/platformdata/addons
                              16K	/home/yellowtent/platformdata/sftp
                              8.0K	/home/yellowtent/platformdata/firewall
                              4.0K	/home/yellowtent/platformdata/acme
                              

                              --
                              Dustin Dauncey
                              www.d19.ca

                              1 Reply Last reply
                              3
                              • d19dotcaD Offline
                                d19dotcaD Offline
                                d19dotca
                                wrote on last edited by
                                #21

                                @girish - I was wanting to follow-up on this. The defect still seems to exist. Right now, despite me running about 41 apps, I only have several in the logrotate.d folder:

                                ubuntu@my:/home/yellowtent/platformdata$ ls -alh logrotate.d/
                                total 60K
                                drwxr-xr-x  2 root       root       4.0K Sep  7 05:00 .
                                drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 ..
                                -rw-r--r--  1 root       root        946 Aug 22 03:43 3167ef7d-e4c2-4072-8e11-8de49c5db970
                                -rw-r--r--  1 root       root        946 Aug 23 16:40 4929b73d-cfa4-4246-82a7-4bc8127982f6
                                -rw-r--r--  1 root       root        946 Sep  7 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
                                -rw-r--r--  1 root       root        946 Sep  4 03:34 684cf8eb-f80c-4304-b97d-ab915a18cfe7
                                -rw-r--r--  1 root       root        946 Sep  4 23:07 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
                                -rw-r--r--  1 root       root        946 Aug 19 23:11 7384a604-7a42-4dc7-86eb-d3f6dd2d546b
                                -rw-r--r--  1 root       root        946 Aug 19 03:25 8dba8d47-7890-49d3-8dfa-970e1ec52991
                                -rw-r--r--  1 root       root        946 Aug 24 05:48 9c70787a-3529-4b66-be2e-7b1b5aa92b02
                                -rw-r--r--  1 root       root        946 Aug 19 23:29 b4888539-e835-4f21-97f5-2fc856be6ecf
                                -rw-r--r--  1 root       root        230 Aug 13 06:39 box
                                -rw-r--r--  1 root       root        946 Aug 22 03:05 c93cef28-3cc4-42d8-88d5-0f581dff74a2
                                -rw-r--r--  1 root       root        946 Sep  4 05:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
                                -rw-r--r--  1 root       root       1.1K Aug 13 06:39 platform
                                

                                --
                                Dustin Dauncey
                                www.d19.ca

                                nebulonN 1 Reply Last reply
                                1
                                • d19dotcaD d19dotca

                                  @girish - I was wanting to follow-up on this. The defect still seems to exist. Right now, despite me running about 41 apps, I only have several in the logrotate.d folder:

                                  ubuntu@my:/home/yellowtent/platformdata$ ls -alh logrotate.d/
                                  total 60K
                                  drwxr-xr-x  2 root       root       4.0K Sep  7 05:00 .
                                  drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 ..
                                  -rw-r--r--  1 root       root        946 Aug 22 03:43 3167ef7d-e4c2-4072-8e11-8de49c5db970
                                  -rw-r--r--  1 root       root        946 Aug 23 16:40 4929b73d-cfa4-4246-82a7-4bc8127982f6
                                  -rw-r--r--  1 root       root        946 Sep  7 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
                                  -rw-r--r--  1 root       root        946 Sep  4 03:34 684cf8eb-f80c-4304-b97d-ab915a18cfe7
                                  -rw-r--r--  1 root       root        946 Sep  4 23:07 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
                                  -rw-r--r--  1 root       root        946 Aug 19 23:11 7384a604-7a42-4dc7-86eb-d3f6dd2d546b
                                  -rw-r--r--  1 root       root        946 Aug 19 03:25 8dba8d47-7890-49d3-8dfa-970e1ec52991
                                  -rw-r--r--  1 root       root        946 Aug 24 05:48 9c70787a-3529-4b66-be2e-7b1b5aa92b02
                                  -rw-r--r--  1 root       root        946 Aug 19 23:29 b4888539-e835-4f21-97f5-2fc856be6ecf
                                  -rw-r--r--  1 root       root        230 Aug 13 06:39 box
                                  -rw-r--r--  1 root       root        946 Aug 22 03:05 c93cef28-3cc4-42d8-88d5-0f581dff74a2
                                  -rw-r--r--  1 root       root        946 Sep  4 05:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
                                  -rw-r--r--  1 root       root       1.1K Aug 13 06:39 platform
                                  
                                  nebulonN Offline
                                  nebulonN Offline
                                  nebulon
                                  Staff
                                  wrote on last edited by
                                  #22

                                  @d19dotca if you reconfigure the location of one of the missing apps, does the logrotate config file show up afterwards? I am wondering if something is purging them after they are created. Looking at the code I cannot spot why it wouldn't create that file.

                                  d19dotcaD 2 Replies Last reply
                                  0
                                  • nebulonN nebulon

                                    @d19dotca if you reconfigure the location of one of the missing apps, does the logrotate config file show up afterwards? I am wondering if something is purging them after they are created. Looking at the code I cannot spot why it wouldn't create that file.

                                    d19dotcaD Offline
                                    d19dotcaD Offline
                                    d19dotca
                                    wrote on last edited by d19dotca
                                    #23

                                    @nebulon Yes, I think I've mentioned that before too. Re-saving from the Location tab on each app regenerates the logrotate file and finally the logrotates will proceed successfully.

                                    I know @girish confirmed that he saw the same behaviour on one of your (or his) servers too. So this definitely seems to be a Cloudron bug that needs fixing. πŸ˜‰

                                    This issue has been sitting a long time (confirmed/reproduced in February 2021), so hoping this gets resolved soon because this keeps causing issues on my server when it comes to disk space consumption alerts.

                                    --
                                    Dustin Dauncey
                                    www.d19.ca

                                    1 Reply Last reply
                                    2
                                    • nebulonN nebulon

                                      @d19dotca if you reconfigure the location of one of the missing apps, does the logrotate config file show up afterwards? I am wondering if something is purging them after they are created. Looking at the code I cannot spot why it wouldn't create that file.

                                      d19dotcaD Offline
                                      d19dotcaD Offline
                                      d19dotca
                                      wrote on last edited by
                                      #24

                                      @nebulon @girish - This is still an issue by the way. Hoping this fix comes soon! I just checked my server after migrating it over the weekend, and I only have 2 apps in the logrotate.d directory. 😞 Having to go through 35 apps now to reinitialize that for proper log maintenance.

                                      --
                                      Dustin Dauncey
                                      www.d19.ca

                                      girishG 1 Reply Last reply
                                      1
                                      • d19dotcaD d19dotca

                                        @nebulon @girish - This is still an issue by the way. Hoping this fix comes soon! I just checked my server after migrating it over the weekend, and I only have 2 apps in the logrotate.d directory. 😞 Having to go through 35 apps now to reinitialize that for proper log maintenance.

                                        girishG Offline
                                        girishG Offline
                                        girish
                                        Staff
                                        wrote on last edited by
                                        #25

                                        @d19dotca I found the issue finally! After a cloudron update, all the old logrotate configs are removed. This was added many releases ago to regenerate new logrotate configs and we forgot to remove the line after that release. Fixed it now.

                                        d19dotcaD 1 Reply Last reply
                                        3
                                        • girishG girish

                                          @d19dotca I found the issue finally! After a cloudron update, all the old logrotate configs are removed. This was added many releases ago to regenerate new logrotate configs and we forgot to remove the line after that release. Fixed it now.

                                          d19dotcaD Offline
                                          d19dotcaD Offline
                                          d19dotca
                                          wrote on last edited by d19dotca
                                          #26

                                          @girish yay! Haha, thank you so much!!! That one was really annoying even though minor. Lol. It happens after a Cloudron migration (not just upgrade) and stuff too FYI but I’m assuming you already knew that.

                                          Edit: actually maybe not… I also had upgraded Cloudron the very next day so I likely just saw the result of that instead. Ignore me.πŸ˜… glad it’s fixed though!!

                                          --
                                          Dustin Dauncey
                                          www.d19.ca

                                          1 Reply Last reply
                                          1
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search