Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. wiredtiger & mongodb restarting loop

wiredtiger & mongodb restarting loop

Scheduled Pinned Locked Moved Solved Support
mongodb
5 Posts 3 Posters 1.3k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • luckowL Offline
      luckowL Offline
      luckow
      translator
      wrote on last edited by girish
      #1

      Sometimes you need to restart your Cloudron instance.
      This situation happens after the last reboot.
      Mongodb in a loop.
      Does anyone have a genius idea?

      2020-11-15T18:15:36.000Z ==> Detected existing installation
      2020-11-15T18:15:36.000Z ==> Removing existing lock file
      2020-11-15T18:15:44.000Z exception: connect failed
      2020-11-15T18:15:44.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:45.000Z exception: connect failed
      2020-11-15T18:15:45.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:46.000Z 2020-11-15T18:15:46.863+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
      2020-11-15T18:15:47.000Z exception: connect failed
      2020-11-15T18:15:47.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] MongoDB starting : pid=10 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] db version v4.0.19
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] allocator: tcmalloc
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] modules: none
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] build environment:
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distarch: x86_64
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     target_arch: x86_64
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] 512 MB of memory available to the process out of 32167 MB total system memory
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { oplogSizeMB: 30, replSet: "rs0" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, journal: { enabled: true }, mmapv1: { smallFiles: true } } }
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] 
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
      2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.437+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
      2020-11-15T18:15:48.000Z exception: connect failed
      2020-11-15T18:15:48.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:49.000Z exception: connect failed
      2020-11-15T18:15:49.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:50.000Z exception: connect failed
      2020-11-15T18:15:50.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.866+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:866707][10:0x7ff5d2074a40], txn-recover: Main recovery loop: starting at 27/54258048 to 28/256
      2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.867+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:867489][10:0x7ff5d2074a40], txn-recover: Recovering log 27 through 28
      2020-11-15T18:15:51.000Z exception: connect failed
      2020-11-15T18:15:51.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:52.000Z exception: connect failed
      2020-11-15T18:15:52.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.000+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:878][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Recovering log 28 through 28
      2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.056+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:56545][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Set global recovery timestamp: 5fb16fa300000008
      2020-11-15T18:15:53.000Z exception: connect failed
      2020-11-15T18:15:53.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:54.000Z exception: connect failed
      2020-11-15T18:15:54.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:55.000Z exception: connect failed
      2020-11-15T18:15:55.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:56.000Z exception: connect failed
      2020-11-15T18:15:56.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:57.000Z exception: connect failed
      2020-11-15T18:15:57.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:58.000Z exception: connect failed
      2020-11-15T18:15:58.000Z ==> wait: mongodb not running yet
      2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1605463971, 8)
      2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1605463971, 8) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1605463971, 8)
      2020-11-15T18:15:59.000Z  Version: Unable to find metadata for table:1091d04d.45c585.4542ab.459de8.45c2a72bb8c4f4/collection-8-5104612647747623411
      2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 664
      2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] 
      2020-11-15T18:15:59.000Z 
      2020-11-15T18:15:59.000Z ***aborting after fassert() failure
      2020-11-15T18:15:59.000Z 
      2020-11-15T18:15:59.000Z 
      2020-11-15T18:15:59.000Z exception: connect failed
      2020-11-15T18:15:59.000Z ==> wait: mongodb not running yet
      

      Pronouns: he/him | Primary language: German

      1 Reply Last reply
      0
      • nebulonN Offline
        nebulonN Offline
        nebulon
        Staff
        wrote on last edited by
        #3

        That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

        luckowL rmdesR 2 Replies Last reply
        1
        • luckowL Offline
          luckowL Offline
          luckow
          translator
          wrote on last edited by
          #2

          Wow. What a ride.
          Something was deeply broken.

          My solution was:

          • ssh into Cloudron
          • rm -rf /home/yellowtent/platformdata/mongodb
          • Install a fresh Cloudron on a different vps
          • rsync the mongodb folder from the newly installed Cloudron to the broken instance
          • restore app backups (in my case rocket.chat & wekan)

          Pronouns: he/him | Primary language: German

          1 Reply Last reply
          1
          • nebulonN Offline
            nebulonN Offline
            nebulon
            Staff
            wrote on last edited by
            #3

            That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

            luckowL rmdesR 2 Replies Last reply
            1
            • nebulonN nebulon

              That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

              luckowL Offline
              luckowL Offline
              luckow
              translator
              wrote on last edited by
              #4

              @nebulon nope. next time I will try it this way 🙂
              I deleted some of the files in the mongodb folder, which looked like the database, configuration and index of the applications. But afterwards it was the same behaviour (restarting the mongodb app itself). With a completely empty mongodb folder I never tried it.

              Pronouns: he/him | Primary language: German

              1 Reply Last reply
              1
              • nebulonN nebulon

                That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

                rmdesR Offline
                rmdesR Offline
                rmdes
                wrote on last edited by
                #5

                @nebulon I was having a similar issue with Mongodb since a few days, I backuped the mongodb folder, created a new one, restarted the mongodb service, restarted the apps and the problem is solved 🙂

                1 Reply Last reply
                1
                Reply
                • Reply as topic
                Log in to reply
                • Oldest to Newest
                • Newest to Oldest
                • Most Votes


                  • Login

                  • Don't have an account? Register

                  • Login or register to search.
                  • First post
                    Last post
                  0
                  • Categories
                  • Recent
                  • Tags
                  • Popular
                  • Bookmarks
                  • Search