Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?

Scheduled Pinned Locked Moved Solved Support
logging
26 Posts 6 Posters 764 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by girish
    #1

    Hello,

    I noticed in doing some disk space checks on my server, that the Cloudron Logs directory is 4 GB in size, which seems excessively large to me and unnecessary. Is it safe to run something like rm -rf /home/yellowtent/platformdata/logs/* to clear it up again? Or will that cause issues with Cloudron?

    I would prefer to clear this up before I do a migration to a new server if possible. And the 4 GB size seems like low-hanging fruit to sort of "spring clean" the server a little bit.

    Here's the complete output, if that helps at all:

    <user>@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
    6.7G	total
    6.7G	/home/yellowtent/platformdata/
    4.0G	/home/yellowtent/platformdata/logs
    1.8G	/home/yellowtent/platformdata/mysql
    519M	/home/yellowtent/platformdata/mongodb
    355M	/home/yellowtent/platformdata/graphite
    89M	/home/yellowtent/platformdata/postgresql
    46M	/home/yellowtent/platformdata/backup
    7.5M	/home/yellowtent/platformdata/redis
    724K	/home/yellowtent/platformdata/nginx
    164K	/home/yellowtent/platformdata/update
    132K	/home/yellowtent/platformdata/collectd
    36K	/home/yellowtent/platformdata/logrotate.d
    32K	/home/yellowtent/platformdata/addons
    8.0K	/home/yellowtent/platformdata/acme
    

    Side note: The command shows 6.7 GB total for Platform Data, but the Cloudron GUI shows 6.4 GB, it's missing about 300 MB in its calculation, I think. Nothing big, but I assumed they'd be the same numbers shown.

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • jimcavoliJ Offline
    jimcavoliJ Offline
    jimcavoli App Dev
    wrote on last edited by
    #2

    That should all be getting logrotate-d to some extent, depending on the subdirectory. Configs are in /home/yellowtent/platformdata/logrotate.d if you want to have a peek but generally those should be getting rotated off pretty fast. A couple gigs isn't super unusual in my experience, but the bulk of that is likely app logs. AFAIK, there's not really any harm to dropping out logs other than the fact that you won't have them, but have a look at the breakdown inside the logs directory - all the logs/{UUID} folders are the app ones and likely of only marginal utility to keep, depending on your needs.

    1 Reply Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #3

    I think you can also run logrotate -f /etc/logrotate.conf to trigger instant rotation. But in general, feel free to truncate the log files manually as well.

    1 Reply Last reply
    1
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by d19dotca
    #4

    I ran the logrotate -f /etc/logrotate.conf command as suggested, and it went from 4.1 GB (this morning) for logs to 1.7 GB, so removed around 60% of it roughly. Personally I still think 1.7 GB is a lot (I mean it's just plain text files), but much more reasonable than over 4 GB, lol.

    I then looked into the logrotate.conf file contents, and I see this:

    <user>@my:~$ cat /etc/logrotate.conf
    # see "man logrotate" for details
    # rotate log files weekly
    weekly
    
    # use the syslog group by default, since this is the owning group
    # of /var/log/syslog.
    su root syslog
    
    # keep 4 weeks worth of backlogs
    rotate 4
    
    # create new (empty) log files after rotating old ones
    create
    
    # uncomment this if you want your log files compressed
    #compress
    
    # packages drop log rotation information into this directory
    include /etc/logrotate.d
    
    # no packages own wtmp, or btmp -- we'll rotate them here
    /var/log/wtmp {
        missingok
        monthly
        create 0664 root utmp
        rotate 1
    }
    
    /var/log/btmp {
        missingok
        monthly
        create 0660 root utmp
        rotate 1
    }
    
    # system-specific logs may be configured here
    
    include /home/yellowtent/platformdata/logrotate.d
    

    I then checked the /home/yellowtent/platformdata/logrotate.d/ directory and found ones for basically each app (but only 5 apps in my case, not the 20+ I'm running... is that sign of an issue? Makes me wonder if some apps are logging indefinitely if there's no logrotate set for them...), along with platform and box, etc.

    Here's an example of one of the 5 apps set in that logrotate.d directory:

    <user>@my:~$ cat /home/yellowtent/platformdata/logrotate.d/3167ef7d-e4c2-4072-8e11-8de49c5db970 
    # Generated by apptask
    
    # keep upto 7 rotated logs. rotation triggered daily or ahead of time if size is > 1M
    /var/lib/docker/volumes/1db1c5dda83ebf58f62348adcc965eca2ad82585ee912ae0dabc9a06dc3f30a2/_data/*.log /var/lib/docker/volumes/1db1c5dda83ebf58f62348adcc965eca2ad82585ee912ae0dabc9a06dc3f30a2/_data/*/*.log /var/lib/docker/volumes/1db1c5dda83ebf58f62348adcc965eca2ad82585ee912ae0dabc9a06dc3f30a2/_data/*/*/*.log {
        rotate 7
        daily
        compress
        maxsize 1M
        missingok
        delaycompress
        # this truncates the original log file and not the rotated one
        copytruncate
    }
    
    /home/yellowtent/platformdata/logs/3167ef7d-e4c2-4072-8e11-8de49c5db970/*.log {
        # only keep one rotated file, we currently do not send that over the api
        rotate 1
        size 10M
        missingok
        # we never compress so we can simply tail the files
        nocompress
        # this truncates the original log file and not the rotated one
        copytruncate
    }
    

    The above makes it sound like there should be very little disk usage of the app logs, basically seven 1 MB files and/or one 10 MB file, right? Or have I misread that?

    I guess I'm just questioning the whole logs setup... in my opinion over 4 GB is overkill for logs (4 GB of plain text is a LOT of text!), and when running logrotate it brought it to a more reasonable (but still a little high IMO) 1.7 GB in size. Then when I look at logrotate.d custom log rotates set by Cloudron, it seems to only include a few apps and most of my apps are missing from the directory which is a red flag to me... is that expected behaviour?

    PS - I realize 4 GB isn't a ton of disk space in the big picture, but I am required to run my servers in Canada and we have far fewer options here, and in many cases (like OVH) the disk space offered is quite low. So 4 GB out of a 50 GB disk offered at OVH for example is basically 1/12th of the disk consumption. Of course I could use something like DigitalOcean but they charge in USD which after conversion is almost double the price of OVH for a comparable specced system, so my choices are limited unfortunately, so several GB of stuff is a larger deal for me than it admittedly may be to most, and I certainly can apreciate that maybe it seems like I'm nitpicking on the logs in that sense. Just trying to do what I can with what I have since disk space is a bit limited in Canada pricing-wise. 🙂

    --
    Dustin Dauncey
    www.d19.ca

    robiR 1 Reply Last reply
    2
  • robiR Offline
    robiR Offline
    robi
    replied to d19dotca on last edited by
    #5

    @d19dotca you're not wrong. I asked about this in the past and we do need much better log hygiene and settings that add up.

    @staff can also reflect on how many logs they need while troubleshooting and think about a separate compressed retention setting for those who need to keep 3|6|9 months of logs.

    Life of sky tech

    1 Reply Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #6

    I have to admin I haven't thought about how much logs to retain. It really depends on the app and the incident to determine how much longs one needs. The current settings are simply values that seemed reasonable.

    @d19dotca It is not correct that only some of the apps have logrotate configs. Can you pick some missing app and then go to Location view and click save? This will re-create the logrotate config for them. BTW, is this after a full server restore? Maybe there is some bug here.

    d19dotcaD 1 Reply Last reply
    1
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by
    #7

    @girish Yeah I definitely think there's room for improvement there. Personally I think I'd set the default to be whatever you most often see being reported (or more specifically how quickly you see it being reported). I.e. do you see issues being reported from a month or two ago? Or do you see them usually being reported within hours to a week of the incident noted in the logs? I'd assume the latter, meaning we likely don't need to be keeping so many logs. And then when there's a rare issue that comes up where we need more logs, the person reporting it can simply make that quick change in the logrotate.conf file to help you guys out to debug more. That's my two cents anyways. Personally I don't think there's any use in keeping logs for more than a few days to a week in most cases.

    Glad to know that there should be a folder for each app. I definitely only have 5 in there right now which isn't even close to the ~20 I have running, so I will try that out on a few and get back to you then if that helps at all.

    The last time I would have restored this particular server would have been last summer sometime, so I would hope that'd have been fixed by now over time if there was a bug in there since almost every app has been updated since then too. Maybe app package updates need to also trigger that process to ensure it has a custom logrotate directory for itself?

    I'll update once I've tested that more. Thanks for the help, Girish. 🙂

    --
    Dustin Dauncey
    www.d19.ca

    girishG 1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    replied to d19dotca on last edited by
    #8

    @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

    Glad to know that there should be a folder for each app

    There's a config file for each app btw and not a folder. It will like <appid>.conf

    d19dotcaD 2 Replies Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by
    #9

    @girish Ah sorry, wrong terminology on my part. Thanks for clarifying.

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to girish on last edited by d19dotca
    #10

    @girish Okay... I think I found a possible root cause of the issue (though it shouldn't be a big deal long-term, would only impact why I saw only 5 apps yesterday)... I believe the root cause is when an app has a pending update (the green up arrow next to it) then the logrotate mysteriously disappears. Here's why I feel this way and my detailed testing and investigation here...

    1. I forced the logrotate again today and it somehow cleared up another large batch of space, taking it from the leftover nearly 2 GB yesterday to just under 600 MB today.
    ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
    3.3G	total
    3.3G	/home/yellowtent/platformdata/
    1.8G	/home/yellowtent/platformdata/mysql
    592M	/home/yellowtent/platformdata/logs
    522M	/home/yellowtent/platformdata/mongodb
    358M	/home/yellowtent/platformdata/graphite
    89M	/home/yellowtent/platformdata/postgresql
    7.6M	/home/yellowtent/platformdata/redis
    724K	/home/yellowtent/platformdata/nginx
    164K	/home/yellowtent/platformdata/update
    128K	/home/yellowtent/platformdata/collectd
    116K	/home/yellowtent/platformdata/backup
    104K	/home/yellowtent/platformdata/logrotate.d
    32K	/home/yellowtent/platformdata/addons
    8.0K	/home/yellowtent/platformdata/acme
    
    1. Realized that when I listed out the /home/yellowtent/platformdata/logrotate.d/ directory, I suddenly had many more app configs listed in there than there was yesterday. Yesterday there was 5 apps only, today there were many (if not all of them at quick glance):
    ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d/
    total 108K
    drwxr-xr-x  2 root       root       4.0K Feb  9 04:22 .
    drwxr-xr-x 15 yellowtent yellowtent 4.0K Dec  9 04:03 ..
    -rw-r--r--  1 root       root        946 Feb  9 04:13 0e55f920-98bd-4ff6-b591-7254c28451c0
    -rw-r--r--  1 root       root        946 Feb  9 04:22 11c6fe46-82ff-4ea4-b816-3b78a962d048
    -rw-r--r--  1 root       root        946 Feb  9 04:14 1c4fd6e4-09ed-4628-a2f4-0938312536f5
    -rw-r--r--  1 root       root        946 Feb  9 04:13 24bff5cf-736c-49a2-a0aa-87e59131c4b6
    -rw-r--r--  1 root       root        946 Feb  9 04:12 28f810a3-2d17-4e45-b0f8-e581528ac4ba
    -rw-r--r--  1 root       root        946 Feb  9 04:10 3167ef7d-e4c2-4072-8e11-8de49c5db970
    -rw-r--r--  1 root       root        946 Feb  9 04:08 38f883dc-4047-4237-a6a1-3ecd77efcde9
    -rw-r--r--  1 root       root        946 Feb  9 04:09 47164e82-f1d2-4fae-9959-507062614906
    -rw-r--r--  1 root       root        946 Feb  9 04:14 83d329e1-6571-46e5-b14f-07477b8f2aac
    -rw-r--r--  1 root       root        946 Feb  3 04:00 8732aed0-273e-49f5-837f-cd9c600bf7b9
    -rw-r--r--  1 root       root        946 Feb  3 04:00 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
    -rw-r--r--  1 root       root        946 Feb  9 04:12 b6fa7dcd-1d93-4786-82e0-90a634f8c3a7
    -rw-r--r--  1 root       root        946 Feb  9 04:09 be3840f0-458e-4bd8-8d18-34120ebdfce4
    -rw-r--r--  1 root       root        230 Feb  2 16:21 box
    -rw-r--r--  1 root       root        946 Feb  9 04:15 c6a63a87-aa45-4281-b4ad-e65c85817f5d
    -rw-r--r--  1 root       root        946 Feb  9 04:15 cf1d5c7c-d4f0-4c32-ada8-e25ab025bd3d
    -rw-r--r--  1 root       root        946 Feb  9 04:09 d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d
    -rw-r--r--  1 root       root        946 Feb  9 04:14 d2df514f-18f6-4e4e-84a1-69bd6360765c
    -rw-r--r--  1 root       root        946 Feb  9 04:14 d2e46149-3287-4944-8e86-1c9a03e12641
    -rw-r--r--  1 root       root        946 Feb  9 04:14 dc527d19-aced-44fa-bae1-a0b22c358a44
    -rw-r--r--  1 root       root        946 Feb  9 04:12 de175741-ee83-4085-b04f-a0048cbdf493
    -rw-r--r--  1 root       root        946 Feb  3 04:00 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
    -rw-r--r--  1 root       root        946 Feb  9 04:13 f1fe6d67-0897-4063-bc76-0ef7256bac7a
    -rw-r--r--  1 root       root        946 Feb  9 04:14 f2dbde27-0159-4e28-803e-7a3fc21450fb
    -rw-r--r--  1 root       root       1.1K Feb  2 16:21 platform
    
    1. Digged further into the logs usage and found that one app in particular was using significantly more than the rest:
    ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
    592M	total
    592M	/home/yellowtent/platformdata/logs
    204M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
    76M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
    72M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
    37M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
    22M	/home/yellowtent/platformdata/logs/turn
    11M	/home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a
    11M	/home/yellowtent/platformdata/logs/11c6fe46-82ff-4ea4-b816-3b78a962d048
    11M	/home/yellowtent/platformdata/logs/0e55f920-98bd-4ff6-b591-7254c28451c0
    9.8M	/home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5
    9.6M	/home/yellowtent/platformdata/logs/47164e82-f1d2-4fae-9959-507062614906
    9.4M	/home/yellowtent/platformdata/logs/f2dbde27-0159-4e28-803e-7a3fc21450fb
    9.1M	/home/yellowtent/platformdata/logs/c6a63a87-aa45-4281-b4ad-e65c85817f5d
    8.6M	/home/yellowtent/platformdata/logs/dc527d19-aced-44fa-bae1-a0b22c358a44
    8.5M	/home/yellowtent/platformdata/logs/d2e46149-3287-4944-8e86-1c9a03e12641
    8.4M	/home/yellowtent/platformdata/logs/d0d6e4ea-36e2-42df-ae47-c59a6d5bf60d
    8.1M	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
    7.2M	/home/yellowtent/platformdata/logs/mail
    
    1. I noticed that app d3121e48-4196-48fe-907d-ee831e11ce5c was the Radicale card/calDAV app. I then checked the list of logrotate app configs and sure enough that one was missing from the list as you can see above in #2.
    2. I followed the steps suggested by @girish and that was then successful in creating the new logrotate for Radicale app, and then I ran logrotate again and it of course cleared even more space now. This lead me to see the following usage afterwards (took the Radicale log usage from 204 MB to 53 MB now):
    ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
    272M	total
    272M	/home/yellowtent/platformdata/logs
    76M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
    72M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
    53M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
    37M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
    8.1M	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
    368K	/home/yellowtent/platformdata/logs/turn
    136K	/home/yellowtent/platformdata/logs/f1fe6d67-0897-4063-bc76-0ef7256bac7a
    88K	/home/yellowtent/platformdata/logs/24bff5cf-736c-49a2-a0aa-87e59131c4b6
    76K	/home/yellowtent/platformdata/logs/1c4fd6e4-09ed-4628-a2f4-0938312536f5
    64K	/home/yellowtent/platformdata/logs/mail
    
    1. I then looked at the new #1 consumer app ID e50f78c7-da6e-4036-a1a3-0cc934448cf1 which was associated to Matomo, and it too was missing a logrotate config file.

    I realized at this point that Matomo had an app update associated with it currently in my Dashboard. Yesterday when I only saw 5 apps, the vast majority of the apps at that time also had updates pending (WordPress makes up around 12-15 of my app instances, then Roundcube was one, plus Bitwarden was another, so had roughly 14-17 or so apps with pending updates at that time which I then updated last night). This makes me suspicious that possibly an app having a pending app update somehow removes it's logrotate configuration file. It kind of explains why I only saw a few apps listed yesterday, as well as why it recovered so much more disk space today when all those apps were updated last night and today their logrotates suddenly existed again. To be fair though, it doesn't explain why Radicale didn't have one today as Radicale hasn't been updated in a while and no pending app updates for it, so that's the one app that goes against my theory.

    Either way... I see issues here with missing logrotate configs. And that's of course on top of the disk space concern, however I suppose if everything was running properly with logrotate my main concern may be a moot point by then. So the root cause of all of all of this may simply be missing logrotate configurations for apps.

    I should add... I haven't updated Matomo yet, in case the Cloudron team wanted to connect to my server to do that update and see if logrotate suddenly exists afterwards, in case that helps at all with troubleshooting.

    --
    Dustin Dauncey
    www.d19.ca

    girishG 1 Reply Last reply
    4
  • girishG Offline
    girishG Offline
    girish Staff
    replied to d19dotca on last edited by
    #11

    @d19dotca Thanks for the details analysis! So, after you report I decided to look into some of the Cloudrons I have access to and indeed logrotate configs for some apps are missing. Let me debug further, but there is definitely some bug.

    1 Reply Last reply
    3
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by
    #12

    I just wanted to say I noticed an excessive use of logs again on my server. I guess this wasn't fixed yet, eh?

    ubuntu@cloudron:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr
    12G	total
    12G	/home/yellowtent/platformdata/logs
    1.6G	/home/yellowtent/platformdata/logs/tasks
    919M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499
    916M	/home/yellowtent/platformdata/logs/76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
    804M	/home/yellowtent/platformdata/logs/9f9aba4a-f7b1-44d8-859c-c6953027c39e
    803M	/home/yellowtent/platformdata/logs/00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
    801M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d
    755M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719
    754M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d
    735M	/home/yellowtent/platformdata/logs/633f2c77-9b33-457a-bc3c-919c87d25ca2
    726M	/home/yellowtent/platformdata/logs/29354a9c-7e1d-4986-b78e-3510fe73b370
    699M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a
    685M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244
    561M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991
    294M	/home/yellowtent/platformdata/logs/e18fa18b-b176-4b25-bf78-4d82c022b0f5
    271M	/home/yellowtent/platformdata/logs/d3121e48-4196-48fe-907d-ee831e11ce5c
    85M	/home/yellowtent/platformdata/logs/200d36ca-8109-409a-9b56-07f46c2f09bd
    83M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
    76M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20
    67M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6
    57M	/home/yellowtent/platformdata/logs/93c19968-5b36-4412-adfe-3a8f338143ec
    46M	/home/yellowtent/platformdata/logs/95c45c33-bd8a-4fae-9718-509c69456abe
    43M	/home/yellowtent/platformdata/logs/134ef7f1-46f8-403e-8ac5-a3297b37893f
    43M	/home/yellowtent/platformdata/logs/1247a348-c8da-4f30-8a4c-3fa014f092a1
    36M	/home/yellowtent/platformdata/logs/c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
    33M	/home/yellowtent/platformdata/logs/8315f008-87e9-4f62-9ae8-1938d7b3f9b2
    26M	/home/yellowtent/platformdata/logs/graphite
    25M	/home/yellowtent/platformdata/logs/mail
    23M	/home/yellowtent/platformdata/logs/turn
    20M	/home/yellowtent/platformdata/logs/e50f78c7-da6e-4036-a1a3-0cc934448cf1
    19M	/home/yellowtent/platformdata/logs/3167ef7d-e4c2-4072-8e11-8de49c5db970
    16M	/home/yellowtent/platformdata/logs/sftp
    14M	/home/yellowtent/platformdata/logs/ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
    13M	/home/yellowtent/platformdata/logs/38f883dc-4047-4237-a6a1-3ecd77efcde9
    9.3M	/home/yellowtent/platformdata/logs/collectd
    8.9M	/home/yellowtent/platformdata/logs/redis-e18fa18b-b176-4b25-bf78-4d82c022b0f5
    4.5M	/home/yellowtent/platformdata/logs/9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
    3.9M	/home/yellowtent/platformdata/logs/0db23e8d-9515-45ad-8739-6cf2a2b1078f
    3.7M	/home/yellowtent/platformdata/logs/mysql
    3.4M	/home/yellowtent/platformdata/logs/8732aed0-273e-49f5-837f-cd9c600bf7b9
    2.9M	/home/yellowtent/platformdata/logs/redis-1247a348-c8da-4f30-8a4c-3fa014f092a1
    2.8M	/home/yellowtent/platformdata/logs/mongodb
    2.7M	/home/yellowtent/platformdata/logs/redis-f6b21e27-6d25-4c69-a547-0466e622d719
    2.7M	/home/yellowtent/platformdata/logs/redis-9f9aba4a-f7b1-44d8-859c-c6953027c39e
    2.7M	/home/yellowtent/platformdata/logs/redis-95c45c33-bd8a-4fae-9718-509c69456abe
    2.7M	/home/yellowtent/platformdata/logs/redis-8dba8d47-7890-49d3-8dfa-970e1ec52991
    2.7M	/home/yellowtent/platformdata/logs/redis-76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
    2.7M	/home/yellowtent/platformdata/logs/redis-6e21d87b-ea85-420b-8eed-8b14d601438d
    2.7M	/home/yellowtent/platformdata/logs/redis-6b3ebc4f-9708-4243-bc21-4c9a01bc3499
    2.7M	/home/yellowtent/platformdata/logs/redis-6120f84a-0953-4982-9167-ef872c0ee49a
    2.7M	/home/yellowtent/platformdata/logs/redis-493c4f88-5d4d-40e5-8361-fd7d4541441d
    2.7M	/home/yellowtent/platformdata/logs/redis-29354a9c-7e1d-4986-b78e-3510fe73b370
    2.7M	/home/yellowtent/platformdata/logs/redis-200d36ca-8109-409a-9b56-07f46c2f09bd
    2.7M	/home/yellowtent/platformdata/logs/redis-05db518f-2c45-411b-865f-813dbf85c244
    2.7M	/home/yellowtent/platformdata/logs/redis-00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
    2.6M	/home/yellowtent/platformdata/logs/redis-134ef7f1-46f8-403e-8ac5-a3297b37893f
    2.6M	/home/yellowtent/platformdata/logs/40ae3d28-d67f-453e-895d-0f22b539372c
    2.0M	/home/yellowtent/platformdata/logs/redis-4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
    1.5M	/home/yellowtent/platformdata/logs/redis-4929b73d-cfa4-4246-82a7-4bc8127982f6
    1.1M	/home/yellowtent/platformdata/logs/redis-21aecd6a-398a-433a-a954-5bcd5f861e20
    924K	/home/yellowtent/platformdata/logs/redis-93c19968-5b36-4412-adfe-3a8f338143ec
    416K	/home/yellowtent/platformdata/logs/492b4264-fea8-4d52-bbe6-1f15ce72c53a
    288K	/home/yellowtent/platformdata/logs/updater
    168K	/home/yellowtent/platformdata/logs/redis-633f2c77-9b33-457a-bc3c-919c87d25ca2
    152K	/home/yellowtent/platformdata/logs/postgresql
    112K	/home/yellowtent/platformdata/logs/redis-0db23e8d-9515-45ad-8739-6cf2a2b1078f
    4.0K	/home/yellowtent/platformdata/logs/crash
    4.0K	/home/yellowtent/platformdata/logs/backup
    

    --
    Dustin Dauncey
    www.d19.ca

    necrevistonnezrN 1 Reply Last reply
    0
  • necrevistonnezrN Offline
    necrevistonnezrN Offline
    necrevistonnezr
    replied to d19dotca on last edited by necrevistonnezr
    #13

    @d19dotca said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

    sudo du -ch -d 1 /home/yellowtent/platformdata/logs | sort -hr

    for me, /home/yellowtent/platformdata/logs/graphite has more than 580 MB of logs, as well. Is that normal?

    nebulonN 1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by
    #14

    it's from all the crashing

    Life of sky tech

    1 Reply Last reply
    0
  • nebulonN Offline
    nebulonN Offline
    nebulon Staff
    replied to necrevistonnezr on last edited by
    #15

    @necrevistonnezr you can delete those files safely, of course you will lose all logs history, but if you don't have any immediate issue on your server besides log size, you can just remove them via ssh.

    1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by d19dotca
    #16

    The logs are still being poorly maintained it seems, they account for several GBs of disk space which seems crazy high for just logs:

    813M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499/app.log
    788M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6/app.log
    712M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d/app.log
    656M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d/app.log
    631M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a/app.log
    629M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719/app.log
    613M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e/app.log
    596M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20/app.log
    578M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244/app.log
    514M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991/app.log
    

    In my case, it seems the same issue I reported back February remains... there are missing logrotate.conf files for the area where there's far too high of log file sizes.

    Example: Note that logs for 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 was the largest... when I run the following command, there's nothing returned, suggesting the logrotate.conf is missing for this one, and likely all the rest too: ls -alh /home/yellowtent/platformdata/logrotate.d/ | grep 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 returns no files.

    When I do an ls -alh listing for the directory, here is all that exists which seems far too few:

    ls -alh /home/yellowtent/platformdata/logrotate.d
    total 80K
    drwxr-xr-x  2 root       root       4.0K May 29 05:00 .
    drwxr-xr-x 15 yellowtent yellowtent 4.0K May 15 07:27 ..
    -rw-r--r--  1 root       root        946 May 15 20:21 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
    -rw-r--r--  1 root       root        946 May 17 22:43 02b2f762-afa4-4278-afaa-5c07ad29715e
    -rw-r--r--  1 root       root        946 May 20 16:28 200d36ca-8109-409a-9b56-07f46c2f09bd
    -rw-r--r--  1 root       root        946 May 21 17:35 29354a9c-7e1d-4986-b78e-3510fe73b370
    -rw-r--r--  1 root       root        946 May 27 05:00 3167ef7d-e4c2-4072-8e11-8de49c5db970
    -rw-r--r--  1 root       root        946 May 29 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
    -rw-r--r--  1 root       root        946 May 20 16:23 633f2c77-9b33-457a-bc3c-919c87d25ca2
    -rw-r--r--  1 root       root        946 May 17 22:41 64098702-00d1-4d2e-8d27-4597e100d1c9
    -rw-r--r--  1 root       root        946 May 23 06:28 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
    -rw-r--r--  1 root       root        946 May 16 06:37 9f9aba4a-f7b1-44d8-859c-c6953027c39e
    -rw-r--r--  1 root       root        230 May 15 09:00 box
    -rw-r--r--  1 root       root        946 May 17 22:38 c3ddc268-deef-49e0-8e3a-97e174cf0594
    -rw-r--r--  1 root       root        946 May 27 20:55 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
    -rw-r--r--  1 root       root        946 May 27 21:00 ca7827f8-4686-4818-a821-f92fe4f5726d
    -rw-r--r--  1 root       root        946 May 18 05:00 e18fa18b-b176-4b25-bf78-4d82c022b0f5
    -rw-r--r--  1 root       root        946 May 26 15:44 e50f78c7-da6e-4036-a1a3-0cc934448cf1
    -rw-r--r--  1 root       root        946 May 24 00:24 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
    -rw-r--r--  1 root       root       1.1K May 15 09:00 platform
    

    --
    Dustin Dauncey
    www.d19.ca

    nebulonN 1 Reply Last reply
    0
  • nebulonN Offline
    nebulonN Offline
    nebulon Staff
    replied to d19dotca on last edited by
    #17

    @d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

    d19dotcaD 1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    replied to nebulon on last edited by d19dotca
    #18

    @nebulon said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

    @d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

    No, unfortunately not. That has never been the case either, wish it was that easy. 😛 haha. I reported earlier in this thread that re-saving an app with from Location tab will regenerate the logrotate configs for the app. They just go missing randomly, no idea why they disappear, but I at least know how to get them back. Unfortunately I am not aware of any of this until I randomly check sizing on my server and find incredibly large log files which then leads me to discover the apps are missing logrotate configs. Saving the app again so it sort of rebuilds itself will regenerate it. I just looked now and there are a bunch more, for example: I didn't do anything beyond restoring the Cloudron backup to a new Cloudron instance in the Vultr Marketplace last night:

    ls -alh /home/yellowtent/platformdata/logrotate.d
    total 156K
    drwxr-xr-x  2 root       root       4.0K May 31 03:54 .
    drwxr-xr-x 15 yellowtent yellowtent 4.0K May 30 06:27 ..
    -rw-r--r--  1 root       root        946 May 30 06:28 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
    -rw-r--r--  1 root       root        946 May 30 06:28 02b2f762-afa4-4278-afaa-5c07ad29715e
    -rw-r--r--  1 root       root        946 May 30 06:28 05db518f-2c45-411b-865f-813dbf85c244
    -rw-r--r--  1 root       root        946 May 30 06:28 1247a348-c8da-4f30-8a4c-3fa014f092a1
    -rw-r--r--  1 root       root        946 May 30 06:28 134ef7f1-46f8-403e-8ac5-a3297b37893f
    -rw-r--r--  1 root       root        946 May 30 06:28 200d36ca-8109-409a-9b56-07f46c2f09bd
    -rw-r--r--  1 root       root        946 May 30 06:28 21aecd6a-398a-433a-a954-5bcd5f861e20
    -rw-r--r--  1 root       root        946 May 30 06:28 29354a9c-7e1d-4986-b78e-3510fe73b370
    -rw-r--r--  1 root       root        946 May 30 06:28 3167ef7d-e4c2-4072-8e11-8de49c5db970
    -rw-r--r--  1 root       root        946 May 30 06:29 38f883dc-4047-4237-a6a1-3ecd77efcde9
    -rw-r--r--  1 root       root        946 May 30 06:28 40ae3d28-d67f-453e-895d-0f22b539372c
    -rw-r--r--  1 root       root        946 May 30 06:29 4929b73d-cfa4-4246-82a7-4bc8127982f6
    -rw-r--r--  1 root       root        946 May 30 06:29 492b4264-fea8-4d52-bbe6-1f15ce72c53a
    -rw-r--r--  1 root       root        946 May 30 06:29 493c4f88-5d4d-40e5-8361-fd7d4541441d
    -rw-r--r--  1 root       root        946 May 30 06:29 4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
    -rw-r--r--  1 root       root        946 May 30 06:29 6120f84a-0953-4982-9167-ef872c0ee49a
    -rw-r--r--  1 root       root        946 May 30 06:29 633f2c77-9b33-457a-bc3c-919c87d25ca2
    -rw-r--r--  1 root       root        946 May 30 06:29 64098702-00d1-4d2e-8d27-4597e100d1c9
    -rw-r--r--  1 root       root        946 May 31 03:54 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
    -rw-r--r--  1 root       root        946 May 30 06:29 6e21d87b-ea85-420b-8eed-8b14d601438d
    -rw-r--r--  1 root       root        946 May 30 06:30 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
    -rw-r--r--  1 root       root        946 May 30 06:29 8315f008-87e9-4f62-9ae8-1938d7b3f9b2
    -rw-r--r--  1 root       root        946 May 30 06:29 8732aed0-273e-49f5-837f-cd9c600bf7b9
    -rw-r--r--  1 root       root        946 May 30 06:30 8dba8d47-7890-49d3-8dfa-970e1ec52991
    -rw-r--r--  1 root       root        946 May 30 06:30 95c45c33-bd8a-4fae-9718-509c69456abe
    -rw-r--r--  1 root       root        946 May 30 06:30 9f9aba4a-f7b1-44d8-859c-c6953027c39e
    -rw-r--r--  1 root       root        946 May 30 06:30 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
    -rw-r--r--  1 root       root        230 May 28 17:54 box
    -rw-r--r--  1 root       root        946 May 30 06:30 c3ddc268-deef-49e0-8e3a-97e174cf0594
    -rw-r--r--  1 root       root        946 May 30 06:31 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
    -rw-r--r--  1 root       root        946 May 30 06:32 d3121e48-4196-48fe-907d-ee831e11ce5c
    -rw-r--r--  1 root       root        946 May 30 06:32 e18fa18b-b176-4b25-bf78-4d82c022b0f5
    -rw-r--r--  1 root       root        946 May 30 06:33 e50f78c7-da6e-4036-a1a3-0cc934448cf1
    -rw-r--r--  1 root       root        946 May 30 06:31 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
    -rw-r--r--  1 root       root        946 May 30 06:31 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
    -rw-r--r--  1 root       root        946 May 30 06:31 f6b21e27-6d25-4c69-a547-0466e622d719
    -rw-r--r--  1 root       root       1.1K May 28 17:54 platform
    

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    0
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by d19dotca
    #19

    This is still an issue by the way. Noticed my disk space accumulating quickly. There's practically no logrotate's configured. Is this due to 6.3 though perhaps?

    ubuntu@my:~$ ls -alh /home/yellowtent/platformdata/logrotate.d
    total 20K
    drwxr-xr-x  2 root       root       4.0K Jun 30 15:28 .
    drwxr-xr-x 17 yellowtent yellowtent 4.0K Jun 26 05:05 ..
    -rw-r--r--  1 root       root        946 Jun 30 15:28 38f883dc-4047-4237-a6a1-3ecd77efcde9
    -rw-r--r--  1 root       root        230 Jun 30 05:05 box
    -rw-r--r--  1 root       root       1.1K Jun 30 05:05 platform
    
    ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
    4.1G	total
    4.1G	/home/yellowtent/platformdata/
    2.2G	/home/yellowtent/platformdata/mysql
    891M	/home/yellowtent/platformdata/logs
    470M	/home/yellowtent/platformdata/graphite
    421M	/home/yellowtent/platformdata/mongodb
    97M	/home/yellowtent/platformdata/postgresql
    54M	/home/yellowtent/platformdata/backup
    15M	/home/yellowtent/platformdata/redis
    4.0M	/home/yellowtent/platformdata/nginx
    192K	/home/yellowtent/platformdata/update
    176K	/home/yellowtent/platformdata/collectd
    36K	/home/yellowtent/platformdata/addons
    16K	/home/yellowtent/platformdata/sftp
    16K	/home/yellowtent/platformdata/logrotate.d
    8.0K	/home/yellowtent/platformdata/firewall
    4.0K	/home/yellowtent/platformdata/acme
    

    To re-iterate, the logrotate files get generated using the workaround of hitting "Save" on the "Locations" tab of each app, but this is a manual process and I now have to do this for about 30 apps, causing a bit of downtime with each one. 😕 Then I need to run the logrotate function manually to save the disk space again.

    I'm actually a little surprised this would be an issue when all the containers already had to be regenerated as part of the 6.3 upgrade.

    @staff - I'm concerned that this defect hasn't been fixed since it was reported quite a while back. Any possibility of getting to the bottom of this one sooner than later?

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    3
  • d19dotcaD Offline
    d19dotcaD Offline
    d19dotca
    wrote on last edited by
    #20

    I took the time to manually re-save each app Location tab to recreate the logrotate file as per the workaround, which resulted in a bit of downtime for each app. 😕

    I then ran the sudo logrotate -f /home/yellowtent/platformdata/logrotate.d/* command and saved roughly half a GB in logs directory alone (I see the mongodb and graphite folders among others went much smaller too, presumably due to restarting of the apps?). So basically 500 MB in extra logs and that's only on a server that's been running for only about a week, presumably would have grown much further in just another week or two too if the logrotate files weren't regenerated.

    I really hope this will be fixed in the next bug-fix release.

    ubuntu@my:~$ sudo du -ch -d 1 /home/yellowtent/platformdata/ | sort -hr
    3.2G	total
    3.2G	/home/yellowtent/platformdata/
    2.2G	/home/yellowtent/platformdata/mysql
    422M	/home/yellowtent/platformdata/mongodb
    229M	/home/yellowtent/platformdata/graphite
    171M	/home/yellowtent/platformdata/logs
    97M	/home/yellowtent/platformdata/postgresql
    54M	/home/yellowtent/platformdata/backup
    17M	/home/yellowtent/platformdata/redis
    4.0M	/home/yellowtent/platformdata/nginx
    192K	/home/yellowtent/platformdata/update
    176K	/home/yellowtent/platformdata/collectd
    172K	/home/yellowtent/platformdata/logrotate.d
    36K	/home/yellowtent/platformdata/addons
    16K	/home/yellowtent/platformdata/sftp
    8.0K	/home/yellowtent/platformdata/firewall
    4.0K	/home/yellowtent/platformdata/acme
    

    --
    Dustin Dauncey
    www.d19.ca

    1 Reply Last reply
    3

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.