The rocketchat_apps_logs.bson collection is huge, can we reduce or remove it?

[girish]

@stoccafisso the issue is not related to Cloudron. It seems rocket.chat is filling up the logs (specifically, it seems apps inside rocket.chat are putting lots of logs). See https://forum.cloudron.io/post/100242 for the setting, it's inside rocket.chat

[stoccafisso]

@girish thanks a lot for helping.

I have had some frustrating hours, had to reinstall the whole cloudron in a 16GB ram droplet in order to be able to reinstall latest backup of Rocket.Chat, that backup was over 7GB, while only 2GB before the logtrouble startet.

Anyway, have it installed and working, and in Rocket.Chat made the exact setting you advice above. But in the Rocket.Chat terminal /app/data I can see no "app_logs" file?

Also when I go into mongodb and issue the commands you suggest, I get the following error message:

root@1116b395-909e-4f51-bd9e-9ad9bed3846e:/app/data# mongosh -u "${CLOUDRON_MONGODB_USERNAME}" -p "${CLOUDRON_MONGODB_PASSWORD}" ${CLOUDRON_MONGODB_HOST}:${CLOUDRON_MONGODB_PORT}/${CLOUDRON_MONGODB_DATABASE} db.getCollection('rocketchat_apps_logs').countDocuments()
bash: syntax error near unexpected token `('

I would like to have a look in mongodb, and delete logentries there if there are any. So what to do with these two issues?

[stoccafisso]

Sorry, I forgot to create the directory and make cloudron the owner. Done

[robw]

@stoccafisso I can confirm that for me this is Rocket.Chat flakiness, not Cloudron. I fear your restoration process won't help if Rocket.Chat is just going to kick off its log filling process again.

@girish I haven't yet tried moving logs to the filesystem, because (very strangely) my Rocket.Chat settings page currently appears to be empty, just a search box:

That's a Rocket.Chat mystery I haven't figured out yet.

I did try the scheduled mongodb log clearing idea directly in the app container, but it didn't seem to help. Because I can't see what's actually happening in the database, I don't know if that's because the data is filling up faster than the query runs, or because of database locking because the database is so big (100s of Gb), or something to do with virtual container file system strangeness, or something else.

At this stage my server seems to have fallen into this pattern:

• Rocket.Chat fills the disk
• Rocket.Chat attempts an update (I have auto updates on)
• Update fails because the backup fails (no disk space to prepare the backup, even though my backups go offsite to Backblaze)
• Rocket.Chat restarts, at which point the "filled" disk space moves from within the Rocket.Chat container to the virtual file system (reported as "everything else" on the Cloudron System Info page)
• A reboot releases the disk space from the virtual file system
• Start over...

The disk filling doesn't seem to start immediately any more. I don't know what triggers it.

So that means with a reboot every day or so, the server is more or less operational for general usage, except for a while right at the end when the disk is full.

I'm hoping a Rocket.Chat update will arrive soon that makes this go away.