Trouble importing mongodb data to Rocket.Chat

[robw]

We're attempting to migrate our Rockat.Chat instance from an Ubuntu Snap installation onto our Cloudron instance. Following the instructions at https://cloudron.io/documentation/guides/import-mongodb/ (we used the backupdb command from https://docs.rocket.chat/installation/snaps to get our Mongo DB backup as it uses mongodump under the hood) almost works but we ran into some roadblocks:

1. The first time we attempted a restore it resulted in an error at the end of the import. Sorry I don't have the exact error as it disappeared from the terminal window before I copied it. It was something similar to "failed: Could not import index bio_1 because it already exists with different options".

Turning the app back on, it spins up correctly and we are able to login. Everything seems to work except that the most recent chat messages are from over a year ago. The chat channels are correct, users are correct, everything else seems to be correct - although we didn't run extensive tests.

2. Due to the previous error, we tried running the mongorestore command using the "--drop" option. We thought that might get around the previous error. It seemed to work, but now we receive this error instead:

2020-08-07T01:23:03.863+0000 Failed: d1c4b380-6f69-46f9-8517-609de87b8407.rocketchat_livechat_inquiry: error creating indexes for d1c4b380-6f69-46f9-8517-609de87b8407.rocketchat_livechat_inquiry: cannot restore index with namespace 'd1c4b380-6f69-46f9-8517-609de87b8407.rocketchat_livechat_inquiry.$queueOrder_1_estimatedWaitingTimeQueue_1_estimatedServiceTimeAt_1': namespace is too long (max size is 127 bytes)

After turning the app back on, we get a similar result to above. Everything seems to work except that the most recent messages are from over a year ago (but a different date this time).

Doing some research, it appears that Mongo DB 3.6.3 has a hard limit on the index namespace size that we can't get around:

https://docs.mongodb.com/manual/reference/limits/

Perhaps the BSON file size limit is also affecting us here. Our rocketchat_uploads.chunks.bson file in the Mongo backup is > 400 Mb. But we don't know.

So my question is... Is there a way to upgrade the Mongo DB instance in the Cloudron Addon to a newer version (4.2+) that doesn't have some of these limits? Or can you suggest another way to import our data?

It's important to some of our stakeholders that we maintain the chat history in our migration.

[robw]

Update: Tried running this from the mongodb console:
https://docs.mongodb.com/manual/reference/command/setFeatureCompatibilityVersion/#view-fcv

But permission is denied. I'm not sure that would help anyway, as from what I read the limit is still there but just ignored on the import, so I suspect the index could not be used by the app even if it was imported.

Also tried the --bypassDocumentValidation option on mongorestore from https://docs.mongodb.com/v3.6/reference/program/mongorestore/ but the import log indicates permission is denied on that too. (Not sure if it would help, just some trial and error. Thought it might help if our BSON files were too big.)

[nebulon]

Looking at the docs you provided, it indeed seems that there is no way around updating mongodb in the addon to support that case.
This may take some time though to test and possibly find any backwards compatibility issues.

I've added an issue to track this at https://git.cloudron.io/cloudron/box/-/issues/725

[robw]

Ok, thanks for the response @nebulon - We will wait patiently and see. I understand that a Mongo upgrade in the platform is a significant change.

Of course we are just assuming that the index import failure is the problem. There might be other problems too. But it does seem like exactly the kind of problem that would cause everything else to work but the messages to stop at a certain date... Anyway it would certainly be nice to update that index size capability to make migrations from other platforms easier. It looks like a showstopper at the moment.

I'm guessing the problem is the length of the container name coupled with a long index name in the Rocket.Chat app. And I guess that's also not something that can be changed.

For further info:
Our snap on the source server is running Mongo DB 3.6.14 - so database compatibility shouldn't be an issue I suppose. Probably not too relevant. Do you need any other info?

[nebulon]

Depending on how sensitive your data is, it might be very helpful to get a copy of this to debug and verify a potential fix on our side. If that is ok with you, maybe send a mail to support@cloudron.io with a possible download link for the data set. Otherwise we may have to find a way how to produce such a large set in the first place.

[robw]

Ah... Thank for for being willing to check that out, and I'd love to be able to help (really would), but one of the main reasons we run Rocket.Chat is to keep our internal comms and client data behind our own firewall and never let it out.

I did a quick search and there doesn't seem to be any existing way to generate a larger data set for testing.

Can I help in another way? Would it help to see a list of file sizes in the backup set? Most of them aren't that big, the one I mentioned is the biggest as I guess it is storage of the uploaded files.

[girish]

@robw I have updated monogodb to 4.0 for Cloudron 5.5. We can then upgrade to 4.2 for the release after. (4.0 is a necessary mid-step before upgrading to 4.2).

[alex-adestech]

@girish are there any plans for upgrading mongodb to 4.2 soon? I have the same error even with Cloudron 5.5. I cannot use Rocket.Chat at all, it says "Not Responding" and already submitted an app_error issue.

[girish]

@alex-adestech Yes, the 4.2 update is scheduled for next Cloudron release (in a couple of weeks). The 5.5 release only updated Mongo to 4.0. This was a necessary step to upgrade to 4.2. I put it here - https://forum.cloudron.io/topic/3018/what-s-coming-in-5-6

[girish]

@robw @alex-adestech An update on this.

Crux of the issue is that the indices name generated are too long. @alex-adestech pointed out to me that this issue only sorted out in MongoDB 4.4 and not even MongoDB 4.2. Searching through the rocket.chat issue tracker I saw only person hit this and nobody replied. Search further, I found that Rocket.Chat doesn't even support MongoDB 4.2. The releases page mentions the versions they test with - https://github.com/RocketChat/Rocket.Chat/releases. This got me wondering why we hit this issue. Well, the answer it turns out is that Cloudron generates 32 byte mongodb database names for apps. This in turn ends up affecting the length of the index.

I have made a fix now for Cloudron 5.6 that generates shorter names. (We have had to do something similar for apps using MySQL as well in the past).