Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


SOLVED wiredtiger & mongodb restarting loop


  • Sometimes you need to restart your Cloudron instance.
    This situation happens after the last reboot.
    Mongodb in a loop.
    Does anyone have a genius idea?

    2020-11-15T18:15:36.000Z ==> Detected existing installation
    2020-11-15T18:15:36.000Z ==> Removing existing lock file
    2020-11-15T18:15:44.000Z exception: connect failed
    2020-11-15T18:15:44.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:45.000Z exception: connect failed
    2020-11-15T18:15:45.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:46.000Z 2020-11-15T18:15:46.863+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
    2020-11-15T18:15:47.000Z exception: connect failed
    2020-11-15T18:15:47.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] MongoDB starting : pid=10 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] db version v4.0.19
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] allocator: tcmalloc
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] modules: none
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] build environment:
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distarch: x86_64
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     target_arch: x86_64
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] 512 MB of memory available to the process out of 32167 MB total system memory
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { oplogSizeMB: 30, replSet: "rs0" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, journal: { enabled: true }, mmapv1: { smallFiles: true } } }
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] 
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
    2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.437+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
    2020-11-15T18:15:48.000Z exception: connect failed
    2020-11-15T18:15:48.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:49.000Z exception: connect failed
    2020-11-15T18:15:49.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:50.000Z exception: connect failed
    2020-11-15T18:15:50.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.866+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:866707][10:0x7ff5d2074a40], txn-recover: Main recovery loop: starting at 27/54258048 to 28/256
    2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.867+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:867489][10:0x7ff5d2074a40], txn-recover: Recovering log 27 through 28
    2020-11-15T18:15:51.000Z exception: connect failed
    2020-11-15T18:15:51.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:52.000Z exception: connect failed
    2020-11-15T18:15:52.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.000+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:878][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Recovering log 28 through 28
    2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.056+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:56545][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Set global recovery timestamp: 5fb16fa300000008
    2020-11-15T18:15:53.000Z exception: connect failed
    2020-11-15T18:15:53.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:54.000Z exception: connect failed
    2020-11-15T18:15:54.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:55.000Z exception: connect failed
    2020-11-15T18:15:55.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:56.000Z exception: connect failed
    2020-11-15T18:15:56.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:57.000Z exception: connect failed
    2020-11-15T18:15:57.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:58.000Z exception: connect failed
    2020-11-15T18:15:58.000Z ==> wait: mongodb not running yet
    2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1605463971, 8)
    2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1605463971, 8) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1605463971, 8)
    2020-11-15T18:15:59.000Z  Version: Unable to find metadata for table:1091d04d.45c585.4542ab.459de8.45c2a72bb8c4f4/collection-8-5104612647747623411
    2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 664
    2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] 
    2020-11-15T18:15:59.000Z 
    2020-11-15T18:15:59.000Z ***aborting after fassert() failure
    2020-11-15T18:15:59.000Z 
    2020-11-15T18:15:59.000Z 
    2020-11-15T18:15:59.000Z exception: connect failed
    2020-11-15T18:15:59.000Z ==> wait: mongodb not running yet
    
  • Staff

    That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.


  • Wow. What a ride.
    Something was deeply broken.

    My solution was:

    • ssh into Cloudron
    • rm -rf /home/yellowtent/platformdata/mongodb
    • Install a fresh Cloudron on a different vps
    • rsync the mongodb folder from the newly installed Cloudron to the broken instance
    • restore app backups (in my case rocket.chat & wekan)
  • Staff

    That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.


  • @nebulon nope. next time I will try it this way 🙂
    I deleted some of the files in the mongodb folder, which looked like the database, configuration and index of the applications. But afterwards it was the same behaviour (restarting the mongodb app itself). With a completely empty mongodb folder I never tried it.