Rocket.Chat freezing

[martinv]

Hello,

we are running in an issue where our Rocket.Chat instance is unable to properly send messages after about 20 hours of uptime. Whenever I try to send a message to somebody the message appears gray, indicating that it was not yet sent. The user does actually get the message though, but it impossible to chat like that.

I wanted to check some log files, but I only found that Rocket.Chat logs errors to stdout here: https://github.com/RocketChat/Docker.Official.Image/issues/16 So I tried to run the command, but it have me the error:

# docker logs e5648bb067fe
Error response from daemon: configured logging driver does not support reading

I tried to attach to the docket stdout, but nothing would ever appear:

 docker attach e5648bb067fe

I tried to inspect the files on docker too by logging in:

docker exec -it -u root e5648bb067fe /bin/bash

I tried to check the Mongo containers as well:

# docker exec -it -u root 48f5e963462c /bin/bash

Once there I was able to connect to the database:

root@mongodb:/# mongo

But then even basic command like list of tables failed:

rs0:PRIMARY> use rocketchat
switched to db rocketchat
rs0:PRIMARY> show tables;
Warning: unable to run listCollections, attempting to approximate collection names by parsing connectionStatus

Could you please provide any insight?

Thanks,
Martin

[nebulon]

Hi and welcome here.

Cloudron does not use the upstream Docker images for various reasons. Also the logging infrastructure is different. You can get to the app logs in the logviewer (top right of the app configuration view) of your Cloudron dashboard and the mongodb logs can be found in the "services" view of the dashboard.

I suspect this is related to a regression in the new RocketChat version which we also hit https://github.com/RocketChat/Rocket.Chat/issues/20553
For some reason RocketChat requires a lot more memory since that release and we are waiting for an upstream fix for that.

Can you check you memory usage on the server and potentially see if temporarily increasing the memory available to the app instance helps?

I guess rolling back to the old version is not really an option, since all messages in-between would be lost and generally meteor apps like rocketchat do frequently change database schemas.

[dew579]

https://github.com/RocketChat/Rocket.Chat/releases

3.12.0-rc.0 is released. Does this update fix this issue?

This bug is terrible as my chat has become nearly unusable now due to constant freezing.

[girish]

Doesn't look like https://github.com/RocketChat/Rocket.Chat/issues/20553 is fixed.

[dew579]

@girish is there anyway to reduce the amount of memory that cloudron, rocketchat, or mongodb uses in the mean time while we are waiting for a fix? My chat runs out of memory and freezes now after a few hours. I use to get the message that MongoDB would run out of memory every couple months but would never freeze. A quick restart would make this slight annoyance go away for a couple more months. Now I can't go more than 2-3 hours without a memory error and total freeze. Server has 4gb of ram and has nothing running on it except rocketchat. Is there a ENV option we can change or something?

[dew579]

@girish I think I'm giving RocketChat 3.5GB out of the 4GB currently, but I noticed in the CloudRon admin panel under "Services" that MongoDB is limited to 256MB. So maybe I should scale RocketChat back to 3GB and increase MongoDB ram limit under services? What do you suggest?

[girish]

@dew579 Yes, increasing MongoDB memory and reducing Rocket.Chat's app memory sounds like a good idea. I have left a note in the github issue asking if 3.12 fixes this. Switching between channels takes forever in our chat as well

[nebulon]

We've just updated the rocket chat package to latest upstream release and at least in our instance this issue appears to be solved.

[girish]

Can confirm. Switching channels is more reasonable now.