Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. wiredtiger & mongodb restarting loop

wiredtiger & mongodb restarting loop

Scheduled Pinned Locked Moved Solved Support
mongodb
5 Posts 3 Posters 1.4k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • luckowL Offline
    luckowL Offline
    luckow
    translator
    wrote on last edited by girish
    #1

    Sometimes you need to restart your Cloudron instance.
    This situation happens after the last reboot.
    Mongodb in a loop.
    Does anyone have a genius idea?

    2020-11-15T18:15:36.000Z ==> Detected existing installation
    2020-11-15T18:15:36.000Z ==> Removing existing lock file
    2020-11-15T18:15:44.000Z exception: connect failed
    2020-11-15T18:15:44.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:45.000Z exception: connect failed
    2020-11-15T18:15:45.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:46.000Z 2020-11-15T18:15:46.863+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
    2020-11-15T18:15:47.000Z exception: connect failed
    2020-11-15T18:15:47.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] MongoDB starting : pid=10 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] db version v4.0.19
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] allocator: tcmalloc
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] modules: none
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] build environment:
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distarch: x86_64
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     target_arch: x86_64
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] 512 MB of memory available to the process out of 32167 MB total system memory
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { oplogSizeMB: 30, replSet: "rs0" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, journal: { enabled: true }, mmapv1: { smallFiles: true } } }
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] 
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.437+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
    2020-11-15T18:15:48.000Z exception: connect failed
    2020-11-15T18:15:48.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:49.000Z exception: connect failed
    2020-11-15T18:15:49.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:50.000Z exception: connect failed
    2020-11-15T18:15:50.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.866+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:866707][10:0x7ff5d2074a40], txn-recover: Main recovery loop: starting at 27/54258048 to 28/256
    2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.867+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:867489][10:0x7ff5d2074a40], txn-recover: Recovering log 27 through 28
    2020-11-15T18:15:51.000Z exception: connect failed
    2020-11-15T18:15:51.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:52.000Z exception: connect failed
    2020-11-15T18:15:52.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.000+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:878][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Recovering log 28 through 28
    2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.056+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:56545][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Set global recovery timestamp: 5fb16fa300000008
    2020-11-15T18:15:53.000Z exception: connect failed
    2020-11-15T18:15:53.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:54.000Z exception: connect failed
    2020-11-15T18:15:54.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:55.000Z exception: connect failed
    2020-11-15T18:15:55.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:56.000Z exception: connect failed
    2020-11-15T18:15:56.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:57.000Z exception: connect failed
    2020-11-15T18:15:57.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:58.000Z exception: connect failed
    2020-11-15T18:15:58.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1605463971, 8)
    2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1605463971, 8) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1605463971, 8)
    2020-11-15T18:15:59.000Z  Version: Unable to find metadata for table:1091d04d.45c585.4542ab.459de8.45c2a72bb8c4f4/collection-8-5104612647747623411
    2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 664
    2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] 
    2020-11-15T18:15:59.000Z 
    2020-11-15T18:15:59.000Z ***aborting after fassert() failure
    2020-11-15T18:15:59.000Z 
    2020-11-15T18:15:59.000Z 
    2020-11-15T18:15:59.000Z exception: connect failed
    2020-11-15T18:15:59.000Z ==> wait: mongodb not running yet
    

    Pronouns: he/him | Primary language: German

    1 Reply Last reply
    0
    • nebulonN Offline
      nebulonN Offline
      nebulon
      Staff
      wrote on last edited by
      #3

      That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

      luckowL rmdesR 2 Replies Last reply
      1
      • luckowL Offline
        luckowL Offline
        luckow
        translator
        wrote on last edited by
        #2

        Wow. What a ride.
        Something was deeply broken.

        My solution was:

        • ssh into Cloudron
        • rm -rf /home/yellowtent/platformdata/mongodb
        • Install a fresh Cloudron on a different vps
        • rsync the mongodb folder from the newly installed Cloudron to the broken instance
        • restore app backups (in my case rocket.chat & wekan)

        Pronouns: he/him | Primary language: German

        1 Reply Last reply
        1
        • nebulonN Offline
          nebulonN Offline
          nebulon
          Staff
          wrote on last edited by
          #3

          That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

          luckowL rmdesR 2 Replies Last reply
          1
          • nebulonN nebulon

            That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

            luckowL Offline
            luckowL Offline
            luckow
            translator
            wrote on last edited by
            #4

            @nebulon nope. next time I will try it this way 🙂
            I deleted some of the files in the mongodb folder, which looked like the database, configuration and index of the applications. But afterwards it was the same behaviour (restarting the mongodb app itself). With a completely empty mongodb folder I never tried it.

            Pronouns: he/him | Primary language: German

            1 Reply Last reply
            1
            • nebulonN nebulon

              That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

              rmdesR Offline
              rmdesR Offline
              rmdes
              wrote on last edited by
              #5

              @nebulon I was having a similar issue with Mongodb since a few days, I backuped the mongodb folder, created a new one, restarted the mongodb service, restarted the apps and the problem is solved 🙂

              1 Reply Last reply
              1
              Reply
              • Reply as topic
              Log in to reply
              • Oldest to Newest
              • Newest to Oldest
              • Most Votes


              • Login

              • Don't have an account? Register

              • Login or register to search.
              • First post
                Last post
              0
              • Categories
              • Recent
              • Tags
              • Popular
              • Bookmarks
              • Search