Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Rocket.Chat
  3. Trouble importing mongodb data to Rocket.Chat

Trouble importing mongodb data to Rocket.Chat

Scheduled Pinned Locked Moved Rocket.Chat
mongodbimport
10 Posts 4 Posters 2.4k Views 4 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      R Offline
      robw
      wrote on last edited by girish
      #1

      We're attempting to migrate our Rockat.Chat instance from an Ubuntu Snap installation onto our Cloudron instance. Following the instructions at https://cloudron.io/documentation/guides/import-mongodb/ (we used the backupdb command from https://docs.rocket.chat/installation/snaps to get our Mongo DB backup as it uses mongodump under the hood) almost works but we ran into some roadblocks:

      1. The first time we attempted a restore it resulted in an error at the end of the import. Sorry I don't have the exact error as it disappeared from the terminal window before I copied it. It was something similar to "failed: Could not import index bio_1 because it already exists with different options".

      Turning the app back on, it spins up correctly and we are able to login. Everything seems to work except that the most recent chat messages are from over a year ago. The chat channels are correct, users are correct, everything else seems to be correct - although we didn't run extensive tests.

      1. Due to the previous error, we tried running the mongorestore command using the "--drop" option. We thought that might get around the previous error. It seemed to work, but now we receive this error instead:

      2020-08-07T01:23:03.863+0000 Failed: d1c4b380-6f69-46f9-8517-609de87b8407.rocketchat_livechat_inquiry: error creating indexes for d1c4b380-6f69-46f9-8517-609de87b8407.rocketchat_livechat_inquiry: cannot restore index with namespace 'd1c4b380-6f69-46f9-8517-609de87b8407.rocketchat_livechat_inquiry.$queueOrder_1_estimatedWaitingTimeQueue_1_estimatedServiceTimeAt_1': namespace is too long (max size is 127 bytes)

      After turning the app back on, we get a similar result to above. Everything seems to work except that the most recent messages are from over a year ago (but a different date this time).

      Doing some research, it appears that Mongo DB 3.6.3 has a hard limit on the index namespace size that we can't get around:

      https://docs.mongodb.com/manual/reference/limits/

      Perhaps the BSON file size limit is also affecting us here. Our rocketchat_uploads.chunks.bson file in the Mongo backup is > 400 Mb. But we don't know.

      So my question is... Is there a way to upgrade the Mongo DB instance in the Cloudron Addon to a newer version (4.2+) that doesn't have some of these limits? Or can you suggest another way to import our data?

      It's important to some of our stakeholders that we maintain the chat history in our migration.

      1 Reply Last reply
      2
      • R Offline
        R Offline
        robw
        wrote on last edited by
        #2

        Update: Tried running this from the mongodb console:
        https://docs.mongodb.com/manual/reference/command/setFeatureCompatibilityVersion/#view-fcv

        But permission is denied. I'm not sure that would help anyway, as from what I read the limit is still there but just ignored on the import, so I suspect the index could not be used by the app even if it was imported.

        Also tried the --bypassDocumentValidation option on mongorestore from https://docs.mongodb.com/v3.6/reference/program/mongorestore/ but the import log indicates permission is denied on that too. (Not sure if it would help, just some trial and error. Thought it might help if our BSON files were too big.)

        1 Reply Last reply
        0
        • nebulonN Offline
          nebulonN Offline
          nebulon
          Staff
          wrote on last edited by nebulon
          #3

          Looking at the docs you provided, it indeed seems that there is no way around updating mongodb in the addon to support that case.
          This may take some time though to test and possibly find any backwards compatibility issues.

          I've added an issue to track this at https://git.cloudron.io/cloudron/box/-/issues/725

          1 Reply Last reply
          0
          • R Offline
            R Offline
            robw
            wrote on last edited by
            #4

            Ok, thanks for the response @nebulon - We will wait patiently and see. I understand that a Mongo upgrade in the platform is a significant change.

            Of course we are just assuming that the index import failure is the problem. There might be other problems too. 🙂 But it does seem like exactly the kind of problem that would cause everything else to work but the messages to stop at a certain date... Anyway it would certainly be nice to update that index size capability to make migrations from other platforms easier. It looks like a showstopper at the moment.

            I'm guessing the problem is the length of the container name coupled with a long index name in the Rocket.Chat app. And I guess that's also not something that can be changed.

            For further info:
            Our snap on the source server is running Mongo DB 3.6.14 - so database compatibility shouldn't be an issue I suppose. Probably not too relevant. Do you need any other info?

            1 Reply Last reply
            0
            • nebulonN Offline
              nebulonN Offline
              nebulon
              Staff
              wrote on last edited by
              #5

              Depending on how sensitive your data is, it might be very helpful to get a copy of this to debug and verify a potential fix on our side. If that is ok with you, maybe send a mail to support@cloudron.io with a possible download link for the data set. Otherwise we may have to find a way how to produce such a large set in the first place.

              1 Reply Last reply
              0
              • R Offline
                R Offline
                robw
                wrote on last edited by
                #6

                Ah... Thank for for being willing to check that out, and I'd love to be able to help (really would), but one of the main reasons we run Rocket.Chat is to keep our internal comms and client data behind our own firewall and never let it out. 🙂

                I did a quick search and there doesn't seem to be any existing way to generate a larger data set for testing.

                Can I help in another way? Would it help to see a list of file sizes in the backup set? Most of them aren't that big, the one I mentioned is the biggest as I guess it is storage of the uploaded files.

                1 Reply Last reply
                0
                • girishG Do not disturb
                  girishG Do not disturb
                  girish
                  Staff
                  wrote on last edited by
                  #7

                  @robw I have updated monogodb to 4.0 for Cloudron 5.5. We can then upgrade to 4.2 for the release after. (4.0 is a necessary mid-step before upgrading to 4.2).

                  1 Reply Last reply
                  0
                  • alex-adestechA Offline
                    alex-adestechA Offline
                    alex-adestech
                    wrote on last edited by
                    #8

                    @girish are there any plans for upgrading mongodb to 4.2 soon? I have the same error even with Cloudron 5.5. I cannot use Rocket.Chat at all, it says "Not Responding" and already submitted an app_error issue.

                    1 Reply Last reply
                    0
                    • girishG Do not disturb
                      girishG Do not disturb
                      girish
                      Staff
                      wrote on last edited by
                      #9

                      @alex-adestech Yes, the 4.2 update is scheduled for next Cloudron release (in a couple of weeks). The 5.5 release only updated Mongo to 4.0. This was a necessary step to upgrade to 4.2. I put it here - https://forum.cloudron.io/topic/3018/what-s-coming-in-5-6

                      1 Reply Last reply
                      1
                      • girishG Do not disturb
                        girishG Do not disturb
                        girish
                        Staff
                        wrote on last edited by
                        #10

                        @robw @alex-adestech An update on this.

                        Crux of the issue is that the indices name generated are too long. @alex-adestech pointed out to me that this issue only sorted out in MongoDB 4.4 and not even MongoDB 4.2. Searching through the rocket.chat issue tracker I saw only person hit this and nobody replied. Search further, I found that Rocket.Chat doesn't even support MongoDB 4.2. The releases page mentions the versions they test with - https://github.com/RocketChat/Rocket.Chat/releases. This got me wondering why we hit this issue. Well, the answer it turns out is that Cloudron generates 32 byte mongodb database names for apps. This in turn ends up affecting the length of the index.

                        I have made a fix now for Cloudron 5.6 that generates shorter names. (We have had to do something similar for apps using MySQL as well in the past).

                        1 Reply Last reply
                        1
                        Reply
                        • Reply as topic
                        Log in to reply
                        • Oldest to Newest
                        • Newest to Oldest
                        • Most Votes


                          • Login

                          • Don't have an account? Register

                          • Login or register to search.
                          • First post
                            Last post
                          0
                          • Categories
                          • Recent
                          • Tags
                          • Popular
                          • Bookmarks
                          • Search