Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    SOLVED wiredtiger & mongodb restarting loop

    Support
    mongodb
    3
    5
    63
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • luckow
      luckow last edited by girish

      Sometimes you need to restart your Cloudron instance.
      This situation happens after the last reboot.
      Mongodb in a loop.
      Does anyone have a genius idea?

      2020-11-15T18:15:36.000Z ==> Detected existing installation
2020-11-15T18:15:36.000Z ==> Removing existing lock file
2020-11-15T18:15:44.000Z exception: connect failed
2020-11-15T18:15:44.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:45.000Z exception: connect failed
2020-11-15T18:15:45.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:46.000Z 2020-11-15T18:15:46.863+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-11-15T18:15:47.000Z exception: connect failed
2020-11-15T18:15:47.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] MongoDB starting : pid=10 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] db version v4.0.19
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] modules: none
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] build environment:
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] 512 MB of memory available to the process out of 32167 MB total system memory
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { oplogSizeMB: 30, replSet: "rs0" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, journal: { enabled: true }, mmapv1: { smallFiles: true } } }
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] 
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.437+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-11-15T18:15:48.000Z exception: connect failed
2020-11-15T18:15:48.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:49.000Z exception: connect failed
2020-11-15T18:15:49.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:50.000Z exception: connect failed
2020-11-15T18:15:50.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.866+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:866707][10:0x7ff5d2074a40], txn-recover: Main recovery loop: starting at 27/54258048 to 28/256
2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.867+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:867489][10:0x7ff5d2074a40], txn-recover: Recovering log 27 through 28
2020-11-15T18:15:51.000Z exception: connect failed
2020-11-15T18:15:51.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:52.000Z exception: connect failed
2020-11-15T18:15:52.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.000+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:878][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Recovering log 28 through 28
2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.056+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:56545][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Set global recovery timestamp: 5fb16fa300000008
2020-11-15T18:15:53.000Z exception: connect failed
2020-11-15T18:15:53.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:54.000Z exception: connect failed
2020-11-15T18:15:54.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:55.000Z exception: connect failed
2020-11-15T18:15:55.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:56.000Z exception: connect failed
2020-11-15T18:15:56.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:57.000Z exception: connect failed
2020-11-15T18:15:57.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:58.000Z exception: connect failed
2020-11-15T18:15:58.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1605463971, 8)
2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1605463971, 8) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1605463971, 8)
2020-11-15T18:15:59.000Z  Version: Unable to find metadata for table:1091d04d.45c585.4542ab.459de8.45c2a72bb8c4f4/collection-8-5104612647747623411
2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 664
2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] 
2020-11-15T18:15:59.000Z 
2020-11-15T18:15:59.000Z ***aborting after fassert() failure
2020-11-15T18:15:59.000Z 
2020-11-15T18:15:59.000Z 
2020-11-15T18:15:59.000Z exception: connect failed
2020-11-15T18:15:59.000Z ==> wait: mongodb not running yet
      1 Reply Last reply Reply Quote 0
      • nebulon
        nebulon Staff last edited by

        That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

        luckow rmdes 2 Replies Last reply Reply Quote 1
        • luckow
          luckow last edited by

          Wow. What a ride.
          Something was deeply broken.

          My solution was:

          • ssh into Cloudron
          • rm -rf /home/yellowtent/platformdata/mongodb
          • Install a fresh Cloudron on a different vps
          • rsync the mongodb folder from the newly installed Cloudron to the broken instance
          • restore app backups (in my case rocket.chat & wekan)
          1 Reply Last reply Reply Quote 1
          • nebulon
            nebulon Staff last edited by

            That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

            luckow rmdes 2 Replies Last reply Reply Quote 1
            • luckow
              luckow @nebulon last edited by

              @nebulon nope. next time I will try it this way 🙂
              I deleted some of the files in the mongodb folder, which looked like the database, configuration and index of the applications. But afterwards it was the same behaviour (restarting the mongodb app itself). With a completely empty mongodb folder I never tried it.

              1 Reply Last reply Reply Quote 1
              • rmdes
                rmdes @nebulon last edited by

                @nebulon I was having a similar issue with Mongodb since a few days, I backuped the mongodb folder, created a new one, restarted the mongodb service, restarted the apps and the problem is solved 🙂

                1 Reply Last reply Reply Quote 1
                • First post
                  Last post