Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Rocket.Chat
  3. The rocketchat_apps_logs.bson collection is huge, can we reduce or remove it?

The rocketchat_apps_logs.bson collection is huge, can we reduce or remove it?

Scheduled Pinned Locked Moved Rocket.Chat
33 Posts 7 Posters 740 Views 7 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      R Offline
      robw
      wrote on last edited by
      #13

      Ok, so that report that "App was updated to {previousVersion}" means the update didn't work, is that right? I'll know that from now on. 🙂

      I was able to get Rocket.Chat updated to the latest 2.54.1 package. I had to skip the backup process, as the snapshot was not going to finish before the disk filled up. Unfortunately it has not fixed the problem. Disk space started growing right away.

      It did make these errors go away though, which is what was happening with 2.54.0:

      image.png

      I note the rocketchat_apps_logs.bson collection did not get updated during this update. This file is now 201Gb.

      1 Reply Last reply
      0
      • R Offline
        R Offline
        robw
        wrote on last edited by
        #14

        Geez... So starting the 2.54.1 package update (which I note upgrades to Rocket.Chat server 7.2.1) appeared to remove the rocketchat_apps_logs.bson file/collection and start over - which didn't happen on the previous update. (I wonder if this has something to do with the server upgrade?) However now I am watching that file grow rapidly during the update process and the update appears to be stuck on this step:

        image.png

        I guess this is going to fill up the disk and then fail. I'm not quite sure what to do next.

        (By the way, I did try running the mongo data clearing process offered by @martinv earlier. However it went into a black hole, which I suppose is not surprising with a 200 Gb collection file, and meanwhile the disk was filling up rapidly so I stopped it. I suppose since that was running inside the container, everything zapped back to its original state when I restarted the app. Was it wrong to run that in the app console, should I access the database from outside instead?)

        If the disk does fill up and the update stops, and if Rocket.Chat is still running at that point, I'll try the mongo data clearing process again. But I suspect the app won't be running at that point so I won't be able to do anything with the database.

        This is not my favourite Rocket.Chat update. 😞

        1 Reply Last reply
        0
        • R Offline
          R Offline
          robw
          wrote on last edited by
          #15

          Um... The problem may have magically disappeared.

          Yep, the disk filled up. The update failed. The app restarted. At which point, storage snapped back to its original state - 300Gb gone in the blink of an eye. I started the 2.54.2 package update again, skipping the backup just in case the snapshot process hung like earlier. It ran. Rocket.Chat is now on the latest version. The disk filling process does not appear to have re-commenced.

          I've seen this before. Don't watch Highlander II, it was a horrible update that should never have been made. Just go directly to Highlander III.

          1 Reply Last reply
          0
          • R Offline
            R Offline
            robw
            wrote on last edited by robw
            #16

            Oh, the problem is not gone.

            image.png

            image.png

            I'm not sure when that happened, Rocket.Chat had stopped working again this morning when I logged in so that's when I checked. So I assume it's Rocket.Chat again. Since the app wasn't working the logs are full of errors, so I couldn't easily see anything useful there. Since the Rocket.Chat container's file system changed in the last update, I can't seem to see the .bson files any more so I can't confirm that was the problem and I'm not sure what else to check.

            There were some other automatic app updates over the last two nights (though not Rocket.Chat because it's up to date), so I suppose it's possibly not Rocket.Chat this time, I'm just assuming.

            I can't see that data anywhere on the Cloudron disk from the server console. A du -h --max-depth=1 at the root level on the server shows Cloudron is using in expected the amount of space taken up by the apps in the coloured parts of the bar above even though df says the disk is full. The 'everything else' data doesn't seem to exist. So I can't tell what the data is. It appears to be living in the overlay filesystem until a Cloudron reboot, at which time it disappears. (Sorry I didn't take a screenshot.) I know nothing of these virtual file systems, so apologies if I'm explaining that wrong, just reporting what I can see.

            So after a reboot it's all good again. After 30 minutes or so, there's no evidence so far that the disk is filling up or anything is changing on the disk yet:

            99f01949-d7cc-412f-8394-57c3c2240aeb-image.png
            1bb11ae3-c03b-49ec-a52b-3cc2d49146e0-image.png

            girishG 1 Reply Last reply
            1
            • R robw

              Oh, the problem is not gone.

              image.png

              image.png

              I'm not sure when that happened, Rocket.Chat had stopped working again this morning when I logged in so that's when I checked. So I assume it's Rocket.Chat again. Since the app wasn't working the logs are full of errors, so I couldn't easily see anything useful there. Since the Rocket.Chat container's file system changed in the last update, I can't seem to see the .bson files any more so I can't confirm that was the problem and I'm not sure what else to check.

              There were some other automatic app updates over the last two nights (though not Rocket.Chat because it's up to date), so I suppose it's possibly not Rocket.Chat this time, I'm just assuming.

              I can't see that data anywhere on the Cloudron disk from the server console. A du -h --max-depth=1 at the root level on the server shows Cloudron is using in expected the amount of space taken up by the apps in the coloured parts of the bar above even though df says the disk is full. The 'everything else' data doesn't seem to exist. So I can't tell what the data is. It appears to be living in the overlay filesystem until a Cloudron reboot, at which time it disappears. (Sorry I didn't take a screenshot.) I know nothing of these virtual file systems, so apologies if I'm explaining that wrong, just reporting what I can see.

              So after a reboot it's all good again. After 30 minutes or so, there's no evidence so far that the disk is filling up or anything is changing on the disk yet:

              99f01949-d7cc-412f-8394-57c3c2240aeb-image.png
              1bb11ae3-c03b-49ec-a52b-3cc2d49146e0-image.png

              girishG Do not disturb
              girishG Do not disturb
              girish
              Staff
              wrote on last edited by
              #17

              It appears to be living in the overlay filesystem until a Cloudron reboot,

              @robw Maybe the data is somewhere in the rocket.chat container. you have to open a web terminal and inspect the /run and /tmp of the app . Another location might be inside the mongodb container. For this, docker exec -ti mongodb /bin/bash and inspect there. As you found, volume data in docker can be hard to get to the bottom of.

              1 Reply Last reply
              0
              • S Offline
                S Offline
                stoccafisso
                wrote on last edited by
                #18

                I have the same "filling up space" problem on two cloudrons I administer. They are not working, as space is totally filled up. Is there a fix? What can be done?

                1 Reply Last reply
                0
                • girishG Do not disturb
                  girishG Do not disturb
                  girish
                  Staff
                  wrote on last edited by
                  #19

                  Did you change the log destination to filesystem? Did you try the mongodb commands suggestered earlier?

                  S 1 Reply Last reply
                  0
                  • girishG girish

                    Did you change the log destination to filesystem? Did you try the mongodb commands suggestered earlier?

                    S Offline
                    S Offline
                    stoccafisso
                    wrote on last edited by
                    #20

                    @girish I have cloudron version 8.2.3, but can't find any "apps" konfiguration under "Settings". Where do I find the settings to change the log destination?

                    1 Reply Last reply
                    0
                    • girishG Do not disturb
                      girishG Do not disturb
                      girish
                      Staff
                      wrote on last edited by
                      #21

                      @stoccafisso the issue is not related to Cloudron. It seems rocket.chat is filling up the logs (specifically, it seems apps inside rocket.chat are putting lots of logs). See https://forum.cloudron.io/post/100242 for the setting, it's inside rocket.chat

                      1 Reply Last reply
                      1
                      • girishG girish

                        @robw just guessing here, so do this with a backup. In settings -> apps, change the log destination to filesystem. Change it to something like /app/data/app_logs (which cloudron as owner) . By default, it seems to save logs to the mongodb database. Once you verify that something is getting created in the filesystem, maybe you can run some mogodb commands to remove the app_logs collection.

                        image.png

                        S Offline
                        S Offline
                        stoccafisso
                        wrote on last edited by stoccafisso
                        #22

                        @girish thanks a lot for helping.

                        I have had some frustrating hours, had to reinstall the whole cloudron in a 16GB ram droplet in order to be able to reinstall latest backup of Rocket.Chat, that backup was over 7GB, while only 2GB before the logtrouble startet.

                        Anyway, have it installed and working, and in Rocket.Chat made the exact setting you advice above. But in the Rocket.Chat terminal /app/data I can see no "app_logs" file?

                        Also when I go into mongodb and issue the commands you suggest, I get the following error message:

                        root@1116b395-909e-4f51-bd9e-9ad9bed3846e:/app/data# mongosh -u "${CLOUDRON_MONGODB_USERNAME}" -p "${CLOUDRON_MONGODB_PASSWORD}" ${CLOUDRON_MONGODB_HOST}:${CLOUDRON_MONGODB_PORT}/${CLOUDRON_MONGODB_DATABASE} db.getCollection('rocketchat_apps_logs').countDocuments()
                        bash: syntax error near unexpected token `('
                        

                        I would like to have a look in mongodb, and delete logentries there if there are any. So what to do with these two issues?

                        S 1 Reply Last reply
                        0
                        • S stoccafisso

                          @girish thanks a lot for helping.

                          I have had some frustrating hours, had to reinstall the whole cloudron in a 16GB ram droplet in order to be able to reinstall latest backup of Rocket.Chat, that backup was over 7GB, while only 2GB before the logtrouble startet.

                          Anyway, have it installed and working, and in Rocket.Chat made the exact setting you advice above. But in the Rocket.Chat terminal /app/data I can see no "app_logs" file?

                          Also when I go into mongodb and issue the commands you suggest, I get the following error message:

                          root@1116b395-909e-4f51-bd9e-9ad9bed3846e:/app/data# mongosh -u "${CLOUDRON_MONGODB_USERNAME}" -p "${CLOUDRON_MONGODB_PASSWORD}" ${CLOUDRON_MONGODB_HOST}:${CLOUDRON_MONGODB_PORT}/${CLOUDRON_MONGODB_DATABASE} db.getCollection('rocketchat_apps_logs').countDocuments()
                          bash: syntax error near unexpected token `('
                          

                          I would like to have a look in mongodb, and delete logentries there if there are any. So what to do with these two issues?

                          S Offline
                          S Offline
                          stoccafisso
                          wrote on last edited by
                          #23

                          Sorry, I forgot to create the directory and make cloudron the owner. Done 😀

                          1 Reply Last reply
                          0
                          • R Offline
                            R Offline
                            robw
                            wrote on last edited by
                            #24

                            @stoccafisso I can confirm that for me this is Rocket.Chat flakiness, not Cloudron. I fear your restoration process won't help if Rocket.Chat is just going to kick off its log filling process again.

                            @girish I haven't yet tried moving logs to the filesystem, because (very strangely) my Rocket.Chat settings page currently appears to be empty, just a search box:

                            6a705bac-8cc0-4a93-9388-5336b1f80132-image.png

                            That's a Rocket.Chat mystery I haven't figured out yet. 🙂

                            I did try the scheduled mongodb log clearing idea directly in the app container, but it didn't seem to help. Because I can't see what's actually happening in the database, I don't know if that's because the data is filling up faster than the query runs, or because of database locking because the database is so big (100s of Gb), or something to do with virtual container file system strangeness, or something else.

                            At this stage my server seems to have fallen into this pattern:

                            • Rocket.Chat fills the disk
                            • Rocket.Chat attempts an update (I have auto updates on)
                            • Update fails because the backup fails (no disk space to prepare the backup, even though my backups go offsite to Backblaze)
                            • Rocket.Chat restarts, at which point the "filled" disk space moves from within the Rocket.Chat container to the virtual file system (reported as "everything else" on the Cloudron System Info page)
                            • A reboot releases the disk space from the virtual file system
                            • Start over...

                            The disk filling doesn't seem to start immediately any more. I don't know what triggers it.

                            So that means with a reboot every day or so, the server is more or less operational for general usage, except for a while right at the end when the disk is full.

                            I'm hoping a Rocket.Chat update will arrive soon that makes this go away.

                            1 Reply Last reply
                            1
                            • R Offline
                              R Offline
                              robw
                              wrote on last edited by
                              #25

                              Quick update: We now have Rocket.Chat on the 2.54.3 package (ran a manual update without a backup), i.e. the small update which sets the Deno cache directory. It has not fixed the problem. Nothing useful from the Rocket.Chat forums yet. So still working on it.

                              1 Reply Last reply
                              0
                              • R Offline
                                R Offline
                                robw
                                wrote on last edited by
                                #26

                                Quick update: Upgrade to package 2.55.0 with Rocket.Chat server 7.3.0 did not fix the problem (though I didn't think it would based on the RC release notes).

                                Attempts to get help on the Rocket.Chat community forum appear to have failed because I encountered the rudest and most self centred and unhelpful "Community Liaison Officer" I've ever had the misfortune to meet, a genuinely toxic individual. However, my glass is half full: the silver lining is that it reinforced my sincere appreciation of the community in this forum and the Cloudron team. 🙂

                                1 Reply Last reply
                                1
                                • J Online
                                  J Online
                                  joseph
                                  Staff
                                  wrote on last edited by
                                  #27

                                  @robw a bit late to the thread but what is your current configuration for logs inside rocket.chat ? Are you using filesystem? i.e the screenshot that @girish posted at https://forum.cloudron.io/post/100242

                                  1 Reply Last reply
                                  0
                                  • R Offline
                                    R Offline
                                    robw
                                    wrote on last edited by
                                    #28

                                    Good question @joseph - Unfortunately I can't see the settings in Rocket.Chat right now, my settings screen is blank. (I double checked and I do have admin user access.) So I assume it is whatever is default. A suggestion above is to change the logs storage and see what happens. I will give that a go when I get some time to work out how to alter the invisible settings. 🙂

                                    image.png

                                    Meanwhile, I have observed that while the disk filling process does not kick off straight away, it always kicks off when the backup process starts. Taking a backup seems to trigger the process with 100% reliability. (In fact this might be why the disk fills up overnight, as I had auto backups enabled.) Then the disk fills and the backup fails. So I have no app level backups for Rocket.Chat at the moment.

                                    Regarding the separate problem of blank settings, I found that the Admin group had checkboxes DISABLED for all the settings configurations. I don't know what caused that or whether it's important, we've never touched this screen. I have switched these on but my settings are still blank so far.

                                    89d80bf6-114d-4cd0-91f7-8d899e0e9e9d-image.png

                                    J 1 Reply Last reply
                                    0
                                    • R robw

                                      Good question @joseph - Unfortunately I can't see the settings in Rocket.Chat right now, my settings screen is blank. (I double checked and I do have admin user access.) So I assume it is whatever is default. A suggestion above is to change the logs storage and see what happens. I will give that a go when I get some time to work out how to alter the invisible settings. 🙂

                                      image.png

                                      Meanwhile, I have observed that while the disk filling process does not kick off straight away, it always kicks off when the backup process starts. Taking a backup seems to trigger the process with 100% reliability. (In fact this might be why the disk fills up overnight, as I had auto backups enabled.) Then the disk fills and the backup fails. So I have no app level backups for Rocket.Chat at the moment.

                                      Regarding the separate problem of blank settings, I found that the Admin group had checkboxes DISABLED for all the settings configurations. I don't know what caused that or whether it's important, we've never touched this screen. I have switched these on but my settings are still blank so far.

                                      89d80bf6-114d-4cd0-91f7-8d899e0e9e9d-image.png

                                      J Online
                                      J Online
                                      joseph
                                      Staff
                                      wrote on last edited by
                                      #29

                                      @robw a "hack" is to maybe install a fresh instance and then compare what is different 🙂 Will give a better idea of what you are supposed to see normally vs what you are seeing in your instance...

                                      1 Reply Last reply
                                      0
                                      • R Offline
                                        R Offline
                                        robw
                                        wrote on last edited by
                                        #30

                                        Thanks @joseph - I will try that!

                                        Meanwhile, a small update: The disk filling process appears to have magically stopped. I don't know when. So our Rocket.Chat appears to be back to normal operation. I don't see any obvious clues in the package updates, although there are updates to the apps engine, maybe it was that. Our instance is backing up and updating happily again.

                                        It wasn't me! 🙂

                                        1 Reply Last reply
                                        2
                                        • G Offline
                                          G Offline
                                          gilburner27
                                          wrote on last edited by
                                          #31

                                          Hi, I have a similar experience but also unable to remove it

                                          1 Reply Last reply
                                          0
                                          • K Offline
                                            K Offline
                                            kawaiipunk
                                            wrote last edited by
                                            #32

                                            Hi, I'm having a similar problem.

                                            Our .bson file is 135GB and immediately fills back up again after being pruned. I haven't yet found any reports of this in the upstream RC issue tracker. Possibly we can we assume this is a Cloudron specific issue?

                                            1 Reply Last reply
                                            0
                                            Reply
                                            • Reply as topic
                                            Log in to reply
                                            • Oldest to Newest
                                            • Newest to Oldest
                                            • Most Votes


                                              • Login

                                              • Don't have an account? Register

                                              • Login or register to search.
                                              • First post
                                                Last post
                                              0
                                              • Categories
                                              • Recent
                                              • Tags
                                              • Popular
                                              • Bookmarks
                                              • Search