The rocketchat_apps_logs.bson collection is huge, can we reduce or remove it?

[girish]

@martinv ah so, rocketchat_apps_logs.bson can also be a file on the filesystem and not part of mongodb! that is news to me. I guess this is some app issue and nothing we can do. But I can suggest a hack: run a daily or hourly cron in the container to truncate the file or something ?

[martinv]

@girish I tried with this in Cloudron Crontab:

0 * * * * mongosh -u "${CLOUDRON_MONGODB_USERNAME}" -p "${CLOUDRON_MONGODB_PASSWORD}" ${CLOUDRON_MONGODB_HOST}:${CLOUDRON_MONGODB_PORT}/${CLOUDRON_MONGODB_DATABASE} --eval "const collection = db.getCollection('rocketchat_apps_logs'); const toDelete = collection.find().sort({ _createdAt: -1 }).skip(1000); toDelete.forEach(doc => collection.deleteOne({ _id: doc._id }));"

It seems to work.

Update: Fixed the schedule to happen once per hour.

[girish]

@martinv that one runs every minute btw thanks for posting the solution!

[robw]

Firstly, thank you @girish and @martinv for being far more useful than a compass at the South Pole! Still nothing useful from the Rocket.Chat forum.

I have set up that cron task to run every 15 minutes, since our log file seemed to be growing very quickly. (After freeing up 40 Gb of disk space yesterday from other apps, our Cloudron server filled up again overnight.) However the cron task seems to be a workaround rather than a solution...

Also in very strange news, the rocketchat_apps_logs.bson file does not seem to be growing any more, even between cron task runs. I'm still not sure what's in that data, I haven't been able to look at it yet, but I'm guessing it's log data for third party Rocket.Chat apps.

Something else is now filling up the disk very quickly, a Gb every few minutes. I haven't figured out what yet. It does seem to correspond to running Rocket.Chat.

I noticed some other strange things and potentially useful notes while working through it:

Firstly, the disk space problem must be very new or we would've seen it ages ago.

I see there was an auto update to the 2.53.0 package last week.

Then the next day, an update to 2.54.0... Or was there?! (I think there was.)

I'm now seeing this behavior on each update, and it wasn't happening before. 2.54.1 landed a few days ago when the same backup error and strange event log effect. I think this is when the problem most likely started.

I still see the Update button after the update and clicking it offers to upgrade to the same version again. I notice this is not the latest package (which is 2.54.2.)

I noticed the 2.54.x series of updates do contain updates to the apps management system. Could it be related?

I noticed that when I manually ran the update, that's when the rocketchat_apps_logs.bson file seemed to diminish in size: it was no longer hundreds of Gb after the update. But I also noticed this:

Strangely, I couldn't find that data anywhere on the file system by digging through du reports; the disk usage across all directories did not add up to the disk usage I could see with df.

After a reboot it seemed to disappear. Is this something strange about the overlay file system that I don't understand?

Something that may or may not be relevant: Our app backups are going to a Backblaze data store, we don't keep them locally.

I'm going to try re-running the app update again to see if I can get to 2.54.2, and to see if the same thing happens with our rapidly growing data disappearing as part of the update.

[robw]

Oh... Does a local snapshot get taken before it's shipped off to Backblaze? That would make sense and explain the previous backup errors during the app update then; there wasn't enough local disk space.

Also I was wrong about rocketchat_apps_logs.bson not growing. It was (it's up to 158Gb again in under an hour), but perhaps I wasn't seeing the file size update in my remote console while the container was running? This is what's still filling up disk space.

[robw]

Ok, so that report that "App was updated to {previousVersion}" means the update didn't work, is that right? I'll know that from now on.

I was able to get Rocket.Chat updated to the latest 2.54.1 package. I had to skip the backup process, as the snapshot was not going to finish before the disk filled up. Unfortunately it has not fixed the problem. Disk space started growing right away.

It did make these errors go away though, which is what was happening with 2.54.0:

I note the rocketchat_apps_logs.bson collection did not get updated during this update. This file is now 201Gb.

[robw]

Geez... So starting the 2.54.1 package update (which I note upgrades to Rocket.Chat server 7.2.1) appeared to remove the rocketchat_apps_logs.bson file/collection and start over - which didn't happen on the previous update. (I wonder if this has something to do with the server upgrade?) However now I am watching that file grow rapidly during the update process and the update appears to be stuck on this step:

I guess this is going to fill up the disk and then fail. I'm not quite sure what to do next.

(By the way, I did try running the mongo data clearing process offered by @martinv earlier. However it went into a black hole, which I suppose is not surprising with a 200 Gb collection file, and meanwhile the disk was filling up rapidly so I stopped it. I suppose since that was running inside the container, everything zapped back to its original state when I restarted the app. Was it wrong to run that in the app console, should I access the database from outside instead?)

If the disk does fill up and the update stops, and if Rocket.Chat is still running at that point, I'll try the mongo data clearing process again. But I suspect the app won't be running at that point so I won't be able to do anything with the database.

This is not my favourite Rocket.Chat update.

[robw]

Um... The problem may have magically disappeared.

Yep, the disk filled up. The update failed. The app restarted. At which point, storage snapped back to its original state - 300Gb gone in the blink of an eye. I started the 2.54.2 package update again, skipping the backup just in case the snapshot process hung like earlier. It ran. Rocket.Chat is now on the latest version. The disk filling process does not appear to have re-commenced.

I've seen this before. Don't watch Highlander II, it was a horrible update that should never have been made. Just go directly to Highlander III.

[robw]

Oh, the problem is not gone.

I'm not sure when that happened, Rocket.Chat had stopped working again this morning when I logged in so that's when I checked. So I assume it's Rocket.Chat again. Since the app wasn't working the logs are full of errors, so I couldn't easily see anything useful there. Since the Rocket.Chat container's file system changed in the last update, I can't seem to see the .bson files any more so I can't confirm that was the problem and I'm not sure what else to check.

There were some other automatic app updates over the last two nights (though not Rocket.Chat because it's up to date), so I suppose it's possibly not Rocket.Chat this time, I'm just assuming.

I can't see that data anywhere on the Cloudron disk from the server console. A du -h --max-depth=1 at the root level on the server shows Cloudron is using in expected the amount of space taken up by the apps in the coloured parts of the bar above even though df says the disk is full. The 'everything else' data doesn't seem to exist. So I can't tell what the data is. It appears to be living in the overlay filesystem until a Cloudron reboot, at which time it disappears. (Sorry I didn't take a screenshot.) I know nothing of these virtual file systems, so apologies if I'm explaining that wrong, just reporting what I can see.

So after a reboot it's all good again. After 30 minutes or so, there's no evidence so far that the disk is filling up or anything is changing on the disk yet:

[girish]

It appears to be living in the overlay filesystem until a Cloudron reboot,

@robw Maybe the data is somewhere in the rocket.chat container. you have to open a web terminal and inspect the /run and /tmp of the app . Another location might be inside the mongodb container. For this, docker exec -ti mongodb /bin/bash and inspect there. As you found, volume data in docker can be hard to get to the bottom of.

[stoccafisso]

I have the same "filling up space" problem on two cloudrons I administer. They are not working, as space is totally filled up. Is there a fix? What can be done?

[girish]

Did you change the log destination to filesystem? Did you try the mongodb commands suggestered earlier?

[stoccafisso]

@girish I have cloudron version 8.2.3, but can't find any "apps" konfiguration under "Settings". Where do I find the settings to change the log destination?

[girish]

@stoccafisso the issue is not related to Cloudron. It seems rocket.chat is filling up the logs (specifically, it seems apps inside rocket.chat are putting lots of logs). See https://forum.cloudron.io/post/100242 for the setting, it's inside rocket.chat