wiredtiger & mongodb restarting loop

[luckow]

Sometimes you need to restart your Cloudron instance.
This situation happens after the last reboot.
Mongodb in a loop.
Does anyone have a genius idea?

2020-11-15T18:15:36.000Z ==> Detected existing installation
2020-11-15T18:15:36.000Z ==> Removing existing lock file
2020-11-15T18:15:44.000Z exception: connect failed
2020-11-15T18:15:44.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:45.000Z exception: connect failed
2020-11-15T18:15:45.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:46.000Z 2020-11-15T18:15:46.863+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-11-15T18:15:47.000Z exception: connect failed
2020-11-15T18:15:47.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] MongoDB starting : pid=10 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] db version v4.0.19
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] modules: none
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] build environment:
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] 512 MB of memory available to the process out of 32167 MB total system memory
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.106+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "0.0.0.0", port: 27017 }, replication: { oplogSizeMB: 30, replSet: "rs0" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, journal: { enabled: true }, mmapv1: { smallFiles: true } } }
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] 
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.388+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-11-15T18:15:47.000Z 2020-11-15T18:15:47.437+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-11-15T18:15:48.000Z exception: connect failed
2020-11-15T18:15:48.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:49.000Z exception: connect failed
2020-11-15T18:15:49.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:50.000Z exception: connect failed
2020-11-15T18:15:50.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.866+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:866707][10:0x7ff5d2074a40], txn-recover: Main recovery loop: starting at 27/54258048 to 28/256
2020-11-15T18:15:50.000Z 2020-11-15T18:15:50.867+0000 I STORAGE  [initandlisten] WiredTiger message [1605464150:867489][10:0x7ff5d2074a40], txn-recover: Recovering log 27 through 28
2020-11-15T18:15:51.000Z exception: connect failed
2020-11-15T18:15:51.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:52.000Z exception: connect failed
2020-11-15T18:15:52.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.000+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:878][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Recovering log 28 through 28
2020-11-15T18:15:53.000Z 2020-11-15T18:15:53.056+0000 I STORAGE  [initandlisten] WiredTiger message [1605464153:56545][10:0x7ff5d2074a40], file:sizeStorer.wt, txn-recover: Set global recovery timestamp: 5fb16fa300000008
2020-11-15T18:15:53.000Z exception: connect failed
2020-11-15T18:15:53.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:54.000Z exception: connect failed
2020-11-15T18:15:54.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:55.000Z exception: connect failed
2020-11-15T18:15:55.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:56.000Z exception: connect failed
2020-11-15T18:15:56.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:57.000Z exception: connect failed
2020-11-15T18:15:57.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:58.000Z exception: connect failed
2020-11-15T18:15:58.000Z ==> wait: mongodb not running yet
2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1605463971, 8)
2020-11-15T18:15:58.000Z 2020-11-15T18:15:58.958+0000 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1605463971, 8) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1605463971, 8)
2020-11-15T18:15:59.000Z  Version: Unable to find metadata for table:1091d04d.45c585.4542ab.459de8.45c2a72bb8c4f4/collection-8-5104612647747623411
2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] Fatal Assertion 34433 at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 664
2020-11-15T18:15:59.000Z 2020-11-15T18:15:59.732+0000 F -        [initandlisten] 
2020-11-15T18:15:59.000Z 
2020-11-15T18:15:59.000Z ***aborting after fassert() failure
2020-11-15T18:15:59.000Z 
2020-11-15T18:15:59.000Z 
2020-11-15T18:15:59.000Z exception: connect failed
2020-11-15T18:15:59.000Z ==> wait: mongodb not running yet

[nebulon]

That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

[luckow]

Wow. What a ride.
Something was deeply broken.

My solution was:

• ssh into Cloudron
• rm -rf /home/yellowtent/platformdata/mongodb
• Install a fresh Cloudron on a different vps
• rsync the mongodb folder from the newly installed Cloudron to the broken instance
• restore app backups (in my case rocket.chat & wekan)

[nebulon]

That is indeed strange and quite the huge workaround. Did you try to empty that mongodb folder, then restart the mongodb addon and then restore the apps? This should also recreate the mongodb with an empty database.

[luckow]

@nebulon nope. next time I will try it this way
I deleted some of the files in the mongodb folder, which looked like the database, configuration and index of the applications. But afterwards it was the same behaviour (restarting the mongodb app itself). With a completely empty mongodb folder I never tried it.

[rmdes]

@nebulon I was having a similar issue with Mongodb since a few days, I backuped the mongodb folder, created a new one, restarted the mongodb service, restarted the apps and the problem is solved