Mongo DB not starting

logo

Hey there,
at some point the mongo DB seems to have broken on my installation. I just noticed it, because the SSL certificates did not update automatically. Furthermore the app updates are stuck updating now. I checked the service tab, where the mongo DB stays on yellow and does not come up as it should.

In the logs I get the following:

I tried rebooting, updating Ubuntu to 20.04 and worked my way through some troubleshooting steps found here without luck. How can I get this fixed, without losing my data like here..?

Kind regards

logo

Here the cloudron log:

nebulon

Is there any error shown in the mongodb service itself? There is a logviewer button in the services view itself for each service.

If others read this, generally rebooting the server is usually not helping much, but may put still running apps at risk to be down aftertwards as well, without understanding why something does not start.

logo

Indeed, I just restartet the mongodb service, to get fresh logs and following showed (can't share the entire log, since it is to long and I am not allowed to upload txt files, so here what seemed most important):

Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.655+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1689665604:655246][14:0x7ff7eea7acc0], file:local/collection-10-4461053802942612146.wt, txn rollback_to_stable: __rollback_to_stable_btree_apply, 1530: file:local/collection-10-4461053802942612146.wt: unable to open handle: WT_ERROR: non-specific WiredTiger error"}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.657+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1689665604:657723][14:0x7ff7eea7acc0], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 1, snapshot max: 1 snapshot count: 0, oldest timestamp: (1681933318, 1) , meta checkpoint timestamp: (1681933323, 1) base write gen: 16574542"}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.661+00:00"},"s":"I", "c":"STORAGE", "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":996}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.661+00:00"},"s":"I", "c":"RECOVERY", "id":23987, "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":1681933323,"i":1}}}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.663+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1689665604:663636][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_block_read_off, 286: sizeStorer.wt: potential hardware corruption, read checksum error for 4096B block at offset 32768: block header checksum of 0x47873a4d doesn't match expected checksum of 0x7a76f50f"}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.663+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1689665604:663763][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_bm_corrupt_dump, 140: {0: 32768, 4096, 0x7a76f50f}: (chunk 1 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 33 00 00 00 0b 00 00 00 01 00 00 01 00 10 00 00 4d 3a 87 47 01 00 00 00 e2 f5 1a 80 cf c0 e2 1f c0 80 81 e3 fc b1 d9 e2 6f c0 00 00 00 00 00 00 c4 90 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 62 6c 6f 63 6b 5f 6d 65 74 61 64 61 74 61 5f 65 6e 63 72 79 70 74 65 64 3d 66 61 6c 73 65 2c 62 6c 6f 63 6b 5f 6d 65 74 61 64 61 74 61 3d 5b 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 64 75 72 61 62 6c 65 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 73 75 6d 3d 75 6e 63 6f 6d 70 72 65 73 73 65 64 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 69 6d 61 67 65 5f 6d 61 78 3d 30 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 30 39 37 39 31 39 3d 28 61 64 64 72 3d 22 30 31 38 31 38 31 65 34 61 65 34 63 37 37 34 39 38 32 38 31 65 34 31 35 34 36 62 64 31 36 38 33 38 31 65 34 66 63 64 66 33 31 33 64 38 30 38 30 38 30 65 "}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1689665604:664002][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_bm_corrupt_dump, 140: {0: 32768, 4096, 0x7a76f50f}: (chunk 2 of 4): 32 36 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 31 30 39 37 39 31 39 2c 74 69 6d 65 3d 31 36 38 31 38 35 39 30 38 37 2c 73 69 7a 65 3d 38 31 39 32 2c 6e 65 77 65 73 74 5f 73 74 61 72 74 5f 64 75 72 61 62 6c 65 5f 74 73 3d 30 2c 6f 6c 64 65 73 74 5f 73 74 61 72 74 5f 74 73 3d 30 2c 6e 65 77 65 73 74 5f 74 78 6e 3d 30 2c 6e 65 77 65 73 74 5f 73 74 6f 70 5f 64 75 72 61 62 6c 65 5f 74 73 3d 30 2c 6e 65 77 65 73 74 5f 73 74 6f 70 5f 74 73 3d 2d 31 2c 6e 65 77 65 73 74 5f 73 74 6f 70 5f 74 78 6e 3d 2d 31 31 2c 70 72 65 70 61 72 65 3d 30 2c 77 72 69 74 65 5f 67 65 6e 3d 31 36 35 36 35 31 33 34 2c 72 75 6e 5f 77 72 69 74 65 5f 67 65 6e 3d 31 36 35 36 35 31 33 32 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 62 61 63 6b 75 70 5f 69 6e 66 6f 3d 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 39 33 2c 34 36 30 38 29 5d c0 fb 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 30 39 39 31 35 34 3d 28 61 64 64 72 3d 22 30 31 38 35 38 31 65 34 30 32 37 35 30 35 62 35 38 36 38 31 65 34 34 32 34 31 30 37 62 34 38 30 38 30 38 30 38 30 38 30 38 30 65 32 36 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 31 30 39 39 31 35 34 2c 74 69 6d 65 3d 31 36 38 31 39 33 33 32 30 34 2c 73 69 7a 65 3d 38 31 39 32 2c 6e 65 77 65 73 74 5f 73 74 61 72 74 5f 64 75 72 61 62 6c 65 5f 74 73 3d 30 2c 6f 6c 64 65 73 74 5f 73 74 61 72 74 5f 74 73 3d 30 2c 6e 65 77 65 73 74 5f 74 78 6e 3d 30 2c 6e 65 77 65 73 74 5f 73 74 6f 70 5f 64 75 72 61 62 6c 65 5f 74 73 3d 30 2c 6e 65 77 65 73 74 5f 73 74 6f 70 5f 74 73 3d 2d 31 2c 6e 65 77 65 73 74 5f 73 74 6f 70 5f 74 78 6e 3d 2d 31 31 2c 70 72 65 70 61 72 65 3d 30 2c 77 72 69 74 65 5f 67 65 6e 3d 31 36 35 36 38 38 35 36 2c 72 75 6e 5f 77 72 69 74 65 5f 67 65 6e 3d 31 36 35 36 35 31 35 32 29 29 9a 01 85 81 e4 02 75 05 b5 86 81 e4 42 41 07 b4 80 80 80 80 80 80 e2 6f c0 cf c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1689665604:664402][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_bm_corrupt_dump, 140: {0: 32768, 4096, 0x7a76f50f}: (chunk 3 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":0,"message":"[1689665604:664537][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_bm_corrupt_dump, 140: {0: 32768, 4096, 0x7a76f50f}: (chunk 4 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1689665604:664552][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_block_read_off, 295: sizeStorer.wt: fatal read error: WT_ERROR: non-specific WiredTiger error"}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1689665604:664559][14:0x7ff7eea7acc0], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_block_read_off, 295: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"F", "c":"-", "id":23089, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":481}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.664+00:00"},"s":"F", "c":"-", "id":23090, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.665+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
Jul 18 09:33:24 {"t":{"$date":"2023-07-18T07:33:24.787+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"initandlisten","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"556C2CAB354A","b":"556C29C9B000","o":"2E1854A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"556C2CAB4FD9","b":"556C29C9B000","o":"2E19FD9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"556C2CAB2366","b":"556C29C9B000","o":"2E17366","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7FF7EF9B3520","b":"7FF7EF971000","o":"42520","s":"__sigaction","s+":"50"},{"a":"7FF7EFA07A7C","b":"7FF7EF971000","o":"96A7C","s":"pthread_kill","s+":"12C"},{"a":"7FF7EF9B3476","b":"7FF7EF971000","o":"42476","s":"raise","s+":"16"},{"a":"7FF7EF9997F3","b":"7FF7EF971000","o":"287F3","s":"abort","s+":"D3"},{"a":"556C2ABEF8E4","b":"556C29C9B000","o":"F548E4","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"},{"a":"556C2A8C0BAF","b":"556C29C9B000","o":"C25BAF","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1052","s+":"16"},{"a":"556C2ADCDF8B","b":"556C29C9B000","o":"1132F8B","s":"__eventv","s+":"3FB"},{"a":"556C2A8D2C14","b":"556C29C9B000","o":"C37C14","s":"__wt_panic_func","s+":"10C"},{"a":"556C2A8E05FD","b":"556C29C9B000","o":"C455FD","s":"__wt_block_read_off.cold.5","s+":"85"},{"a":"556C2AF01546","b":"556C29C9B000","o":"1266546","s":"__wt_block_extlist_read","s+":"96"},{"a":"556C2AF01A6B","b":"556C29C9B000","o":"1266A6B","s":"__wt_block_extlist_read_avail","s+":"2B"},{"a":"556C2AEFC6B0","b":"556C29C9B000","o":"12616B0","s":"__wt_block_checkpoint_load","s+":"1E0"},{"a":"556C2AEF4A37","b":"556C29C9B000","o":"1259A37","s":"__bm_checkpoint_load","s+":"37"},{"a":"556C2AE237AA","b":"556C29C9B000","o":"11887AA","s":"__wt_btree_open","s+":"D9A"},{"a":"556C2AD4C6A2","b":"556C29C9B000","o":"10B16A2","s":"__wt_conn_dhandle_open","s+":"8D2"},{"a":"556C2ADCC889","b":"556C29C9B000","o":"1131889","s":"__wt_session_get_dhandle","s+":"E9"},{"a":"556C2ADCCF32","b":"556C29C9B000","o":"1131F32","s":"__wt_session_get_dhandle","s+":"792"},{"a":"556C2ADCD224","b":"556C29C9B000","o":"1132224","s":"__wt_session_get_btree_ckpt","s+":"154"},{"a":"556C2AD6447A","b":"556C29C9B000","o":"10C947A","s":"__wt_curfile_open","s+":"5A"},{"a":"556C2ADC786B","b":"556C29C9B000","o":"112C86B","s":"__session_open_cursor_int","s+":"2DB"},{"a":"556C2ADC7528","b":"556C29C9B000","o":"112C528","s":"__wt_open_cursor","s+":"58"},{"a":"556C2AEB648E","b":"556C29C9B000","o":"121B48E","s":"__wt_curtable_open","s+":"3BE"},{"a":"556C2ADC79E0","b":"556C29C9B000","o":"112C9E0","s":"__session_open_cursor_int","s+":"450"},{"a":"556C2ADC81AE","b":"556C29C9B000","o":"112D1AE","s":"__session_open_cursor","s+":"71E"},{"a":"556C2AD29097","b":"556C29C9B000","o":"108E097","s":"_ZN5mongo12_GLOBAL__N_111_openCursorEP12__wt_sessionRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKcPP11__wt_cursor.constprop.929","s+":"37"},{"a":"556C2AD295E7","b":"556C29C9B000","o":"108E5E7","s":"_ZN5mongo17WiredTigerSession15getCachedCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmPKc","s+":"67"},{"a":"556C2ACE4B02","b":"556C29C9B000","o":"1049B02","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE","s+":"172"},{"a":"556C2AD2CA43","b":"556C29C9B000","o":"1091A43","s":"_ZNK5mongo20WiredTigerSizeStorer4loadEPNS_16OperationContextENS_10StringDataE","s+":"153"},{"a":"556C2AD18300","b":"556C29C9B000","o":"107D300","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_18WiredTigerKVEngineEPNS_16OperationContextENS0_6ParamsE","s+":"6C0"},{"a":"556C2AD187C4","b":"556C29C9B000","o":"107D7C4","s":"_ZN5mongo29StandardWiredTigerRecordStoreC1EPNS_18WiredTigerKVEngineEPNS_16OperationContextENS_21WiredTigerRecordStore6ParamsE","s+":"C4"},{"a":"556C2ACF851D","b":"556C29C9B000","o":"105D51D","s":"_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE","s+":"3FD"},{"a":"556C2B5F8104","b":"556C29C9B000","o":"195D104","s":"_ZN5mongo17StorageEngineImpl11loadCatalogEPNS_16OperationContextE","s+":"1344"},{"a":"556C2B5FB18C","b":"556C29C9B000","o":"196018C","s":"_ZN5mongo17StorageEngineImplC1EPNS_8KVEngineENS_20StorageEngineOptionsE","s+":"25C"},{"a":"556C2ACDF0B5","b":"556C29C9B000","o":"10440B5","s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","s+":"1A5"},{"a":"556C2B605AC7","b":"556C29C9B000","o":"196AAC7","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE","s+":"4F7"},{"a":"556C2AC8B1C7","b":"556C29C9B000","o":"FF01C7","s":"_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1585","s+":"3F7"},{"a":"556C2AC8F3FA","b":"556C29C9B000","o":"FF43FA","s":"_ZN5mongo12_GLOBAL__N_111mongoDbMainEiPPc","s+":"A5A"},{"a":"556C2ABFEFB9","b":"556C29C9B000","o":"F63FB9","s":"main","s+":"9"},{"a":"7FF7EF99AD90","b":"7FF7EF971000","o":"29D90","s":"__libc_init_first","s+":"90"},{"a":"7FF7EF99AE40","b":"7FF7EF971000","o":"29E40","s":"__libc_start_main","s+":"80"},{"a":"556C2AC8A5FE","b":"556C29C9B000","o":"FEF5FE","s":"_start","s+":"2E"}],"processInfo":{"mongodbVersion":"4.4.19","gitVersion":"9a996e0ad993148b9650dc402e6d3b1804ad3b8a","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-153-generic","version":"#170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023","machine":"x86_64"},"somap":[{"b":"556C29C9B000","elfType":3,"buildId":"0A9DC1A0996DF21B80723D4010693B05ED2C96B2"},{"b":"7FF7EF971000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"69389D485A9793DBE873F0EA2C93E02EFAA9AA3D"}]}}}}
...

Jul 18 09:33:25 /app/code/start.sh: line 18: 14 Aborted (core dumped) $mongod --noauth 2> /dev/null

Thanks, good to know, seems I got lucky nothing further broke in my case

nebulon

Seems like some possible disk corruption. Are you actually using any apps which use mongodb? If not you could clear out the mongodb addon storage folder (/home/yellowtent/platformdata/mongodb/) and restart the service.

girish

@logo you can try something like:

mv /home/yellowtent/platformdata/mongodb/ /home/yellowtent/platformdata/mongodb-save
mkdir /home/yellowtent/platformdata/mongodb/
systemctl restart box

Then, mongodb should come up or go to Services -> Restart Mongodb.

If you have apps that used mongodb, then go into each app -> backups -> restore from latest backup.

logo

@nebulon and @girish thank you for the analysis and suggestions. How do I know if an app uses mongodb? Right now I am using nextcloud and limesurvey, but I could find any mongodb references in the app settings.

girish

@logo both don't use mongodb.

Though @nebulon asked, It's not straightforward to know if an app uses mongodb You have to look into the package code at https://git.cloudron.io/cloudron/<appname>-app . Inside, each repo there is CloudronManifest.json file. Inside that file , it will list under the addons section whether the app needs mongodb or not.

logo

@girish alright, thanks a lot. Always useful to learn more about the things we use

I did what you wrote above and now everything works just like usual
If disk issues keep on happening, I will have to get in contact with the provider..

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Mongo DB not starting