Issue Starting NodeBB on Cloudron After Server Restart

[archos]

Hi, after restarting the server today, I have an issue with NodeBB running on Cloudron. The application status shows "Not Responding", and the terminal in Cloudron UI is not working.

Is there any way to fix this issue? Thanks a lot for your help!

Jan 30 16:33:13 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:33:23 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:33:32 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:33:43 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:33:53 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:34:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.18.198:4567
Jan 30 16:34:17 => Healtheck error: Error: Timeout of 7000ms exceeded
Jan 30 16:34:27 => Healtheck error: Error: Timeout of 7000ms exceeded
Jan 30 16:34:32 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:34:43 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.198:4567
Jan 30 16:33:58 2025-01-30T16:33:58+01:00
Jan 30 16:33:58 2025-01-30T16:33:58+01:00
Jan 30 16:33:58 2025-01-30T16:33:58+01:00
Jan 30 16:33:58 2025-01-30T16:33:58+01:00
Jan 30 16:33:58 DeprecationWarning: Collection.update() is deprecated. Use updateOne, updateMany, or bulkWrite.
Jan 30 16:33:58 Telemetry is now disabled.
Jan 30 16:33:58 Using Mongosh: 1.10.6
Jan 30 16:33:58 Using Mongosh: 1.10.6
Jan 30 16:33:58 ]0;mongosh Current Mongosh Log ID: 679b9be6007078407cd9800e
Jan 30 16:33:58 ]0;mongosh mongodb://<credentials>@mongodb:27017/6227182edf48e3d1?directConnection=trueCurrent Mongosh Log ID: 679b9be688bebc16ac8d5b07
Jan 30 16:33:58 => Bringing package.json up to speed
Jan 30 16:33:58 => Creating directories
Jan 30 16:33:58 => Setting up email
Jan 30 16:33:58 Connecting to: mongodb://<credentials>@mongodb:27017/6227182edf48e3d1?directConnection=true&appName=mongosh+1.10.6
Jan 30 16:33:58 For mongosh info see: https://docs.mongodb.com/mongodb-shell/
Jan 30 16:33:58 For mongosh info see: https://docs.mongodb.com/mongodb-shell/
Jan 30 16:33:58 Using MongoDB: 6.0.13
Jan 30 16:33:58 acknowledged: true,
Jan 30 16:33:58 insertedId: null,
Jan 30 16:33:58 matchedCount: 1,
Jan 30 16:33:58 modifiedCount: 0,
Jan 30 16:33:58 upsertedCount: 0
Jan 30 16:33:58 {
Jan 30 16:33:58 }
Jan 30 16:33:59 2025-01-30T16:33:59+01:00
Jan 30 16:33:59 2025-01-30T16:33:59+01:00
Jan 30 16:33:59 2025-01-30T16:33:59+01:00
Jan 30 16:33:59 2025-01-30T16:33:59+01:00
Jan 30 16:33:59
Jan 30 16:33:59 npm ERR!
Jan 30 16:33:59 npm ERR!
Jan 30 16:33:59 npm ERR!
Jan 30 16:33:59 npm ERR!
Jan 30 16:33:59 npm ERR!
Jan 30 16:33:59 npm ERR!
Jan 30 16:33:59 npm ERR! node_modules/nodebb-plugin-extended-markdown
Jan 30 16:33:59 npm ERR! node_modules/nodebb-plugin-markdown
Jan 30 16:33:59 npm ERR! node_modules/nodebb-plugin-markdown
Jan 30 16:33:59 npm ERR! peer nodebb-plugin-markdown@"^12.0.2" from nodebb-plugin-extended-markdown@2.0.1
Jan 30 16:33:59 npm ERR! nodebb-plugin-extended-markdown@"2.0.1" from the root project
Jan 30 16:33:59 npm ERR! nodebb-plugin-markdown@"13.0.0" from the root project
Jan 30 16:33:59 npm ERR! node_modules/nodebb-plugin-extended-markdown
Jan 30 16:33:59 npm ERR! peer nodebb-plugin-markdown@"^12.0.2" from nodebb-plugin-extended-markdown@2.0.1
Jan 30 16:33:59 npm ERR! nodebb-plugin-extended-markdown@"2.0.1" from the root project
Jan 30 16:33:59 npm ERR! /home/cloudron/.npm/_logs/2025-01-30T15_33_59_322Z-eresolve-report.txt
Jan 30 16:33:59 npm ERR! Conflicting peer dependency: nodebb-plugin-markdown@12.2.9
Jan 30 16:33:59 npm ERR! Could not resolve dependency:
Jan 30 16:33:59 npm ERR! Fix the upstream dependency conflict, or retry
Jan 30 16:33:59 npm ERR! For a full report see:
Jan 30 16:33:59 npm ERR! Found: nodebb-plugin-markdown@13.0.0
Jan 30 16:33:59 npm ERR! While resolving: nodebb-plugin-extended-markdown@2.0.1
Jan 30 16:33:59 npm ERR! this command with --force or --legacy-peer-deps
Jan 30 16:33:59 npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
Jan 30 16:33:59 npm ERR! ERESOLVE could not resolve
Jan 30 16:33:59 Error installing dependencies!
Jan 30 16:33:59 1. Bringing base dependencies up to date... started
Jan 30 16:33:59 1. Updating package.json file with defaults... OK
Jan 30 16:33:59 NodeBB Upgrade Complete!
Jan 30 16:33:59
Jan 30 16:33:59 [#########.........] / idealTree:code: timing idealTree:#root Completed in 1ms npm ERR! code ERESOLVE
Jan 30 16:33:59
Jan 30 16:33:59
Jan 30 16:33:59 Updating NodeBB...
Jan 30 16:33:59 Updating NodeBB...
Jan 30 16:33:59 npm ERR! A complete log of this run can be found in: /home/cloudron/.npm/_logs/2025-01-30T15_33_59_322Z-debug-0.log
Jan 30 16:33:59 2025-01-30T15:33:59.733Z [4567/54] - error: uncaughtException: Command failed: npm install --omit=dev
Jan 30 16:33:59 => Installing packages
Jan 30 16:33:59 Error occurred during upgrade: Error: Command failed: npm install --omit=dev
Jan 30 16:33:59 Error: Command failed: npm install --omit=dev
Jan 30 16:33:59 at /app/code/node_modules/commander/lib/command.js:1502:14
Jan 30 16:33:59 at /app/code/node_modules/commander/lib/command.js:1502:14 {"date":"Thu Jan 30 2025 15:33:59 GMT+0000 (Coordinated Universal Time)","error":{"output":[null,null,null],"pid":69,"signal":null,"status":1,"stderr":null,"stdout":null},"exception":true,"os":{"loadavg":[0.55,0.78,0.97],"uptime":33730.1},"process":{"argv":["/usr/local/node-18.18.0/bin/node","/app/code/nodebb","upgrade","--install"],"cwd":"/app/code","execPath":"/usr/local/node-18.18.0/bin/node","gid":1000,"memoryUsage":{"arrayBuffers":530315,"external":1195535,"heapTotal":21671936,"heapUsed":16562376,"rss":67280896},"pid":54,"uid":1000,"version":"v18.18.0"},"stack":"Error: Command failed: npm install --omit=dev\n at checkExecSyncError (node:child_process:890:11)\n at Object.execSync (node:child_process:962:15)\n at pkgInstall.installAll (/app/code/src/cli/package-install.js:129:9)\n at Object.handler (/app/code/src/cli/upgrade.js:22:19)\n at runSteps (/app/code/src/cli/upgrade.js:55:16)\n at Object.runUpgrade [as upgrade] (/app/code/src/cli/upgrade.js:98:9)\n at Command.<anonymous> (/app/code/src/cli/index.js:303:24)\n at Command.listener [as _actionHandler] (/app/code/node_modules/commander/lib/command.js:542:17)\n at /app/code/node_modules/commander/lib/command.js:1502:14","trace":[{"column":11,"file":"node:child_process","function":"checkExecSyncError","line":890,"method":null,"native":false},{"column":15,"file":"node:child_process","function":"Object.execSync","line":962,"method":"execSync","native":false},{"column":9,"file":"/app/code/src/cli/package-install.js","function":"pkgInstall.installAll","line":129,"method":"installAll","native":false},{"column":19,"file":"/app/code/src/cli/upgrade.js","function":"Object.handler","line":22,"method":"handler","native":false},{"column":16,"file":"/app/code/src/cli/upgrade.js","function":"runSteps","line":55,"method":null,"native":false},{"column":9,"file":"/app/code/src/cli/upgrade.js","function":"Object.runUpgrade [as upgrade]","line":98,"method":"runUpgrade [as upgrade]","native":false},{"column":24,"file":"/app/code/src/cli/index.js","function":null,"line":303,"method":null,"native":false},{"column":17,"file":"/app/code/node_modules/commander/lib/command.js","function":"Command.listener [as _actionHandler]","line":542,"method":"listener [as _actionHandler]","native":false},{"column":14,"file":"/app/code/node_modules/commander/lib/command.js","function":null,"line":1502,"method":null,"native":false}]}
Jan 30 16:33:59 at Command.<anonymous> (/app/code/src/cli/index.js:303:24)
Jan 30 16:33:59 at Command.<anonymous> (/app/code/src/cli/index.js:303:24)

[joseph]

It seems nodebb-plugin-markdown is causing some problem. Can you try disabling it ? https://docs.cloudron.io/apps/nodebb/#disabling-plugins . You might have to revert the app to a working state or put the app in recovery mode.

[archos]

@joseph said in Issue Starting NodeBB on Cloudron After Server Restart:

It seems nodebb-plugin-markdown is causing some problem. Can you try disabling it ? https://docs.cloudron.io/apps/nodebb/#disabling-plugins . You might have to revert the app to a working state or put the app in recovery mode.

I tried disabling the plugin, but I cannot type in the terminal. It looks like the application is frozen. I also tried enabling recovery mode, but the problem was not resolved.

[archos]

I tried running the command in recovery mode, and I got the following output in the terminal:

root@630e742c-440d-409a-97ef-45a5161aa337:/app/code# cd /app/code
root@630e742c-440d-409a-97ef-45a5161aa337:/app/code# ./nodebb reset -p nodebb-plugin-markdown
Warning: Failed to detect linux-musl, fallback to linux-gnu: The "offset" argument must be of type number. Received an instance of Object
2025-01-30T18:38:12.386Z [undefined/19] - error: uncaughtException: ENOENT: no such file or directory, mkdir 'logs'
Error: ENOENT: no such file or directory, mkdir 'logs'
    at Object.mkdirSync (node:fs:1398:3)
    at File._createLogDirIfNotExist (/app/code/node_modules/winston/lib/winston/transports/file.js:759:10)
    at new File (/app/code/node_modules/winston/lib/winston/transports/file.js:94:28)
    at Object.<anonymous> (/app/code/install/web.js:45:3)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:119:18) {"date":"Thu Jan 30 2025 18:38:12 GMT+0000 (Coordinated Universal Time)","error":{"code":"ENOENT","errno":-2,"path":"logs","syscall":"mkdir"},"exception":true,"os":{"loadavg":[0.46,0.81,0.94],"uptime":44782.76},"process":{"argv":["/usr/local/node-18.18.0/bin/node","/app/code/nodebb","reset","-p","nodebb-plugin-markdown"],"cwd":"/app/code","execPath":"/usr/local/node-18.18.0/bin/node","gid":0,"memoryUsage":{"arrayBuffers":327376,"external":2265472,"heapTotal":30515200,"heapUsed":23418856,"rss":79994880},"pid":19,"uid":0,"version":"v18.18.0"},"stack":"Error: ENOENT: no such file or directory, mkdir 'logs'\n    at Object.mkdirSync (node:fs:1398:3)\n    at File._createLogDirIfNotExist (/app/code/node_modules/winston/lib/winston/transports/file.js:759:10)\n    at new File (/app/code/node_modules/winston/lib/winston/transports/file.js:94:28)\n    at Object.<anonymous> (/app/code/install/web.js:45:3)\n    at Module._compile (node:internal/modules/cjs/loader:1256:14)\n    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)\n    at Module.load (node:internal/modules/cjs/loader:1119:32)\n    at Module._load (node:internal/modules/cjs/loader:960:12)\n    at Module.require (node:internal/modules/cjs/loader:1143:19)\n    at require (node:internal/modules/cjs/helpers:119:18)","trace":[{"column":3,"file":"node:fs","function":"Object.mkdirSync","line":1398,"method":"mkdirSync","native":false},{"column":10,"file":"/app/code/node_modules/winston/lib/winston/transports/file.js","function":"File._createLogDirIfNotExist","line":759,"method":"_createLogDirIfNotExist","native":false},{"column":28,"file":"/app/code/node_modules/winston/lib/winston/transports/file.js","function":"new File","line":94,"method":null,"native":false},{"column":3,"file":"/app/code/install/web.js","function":null,"line":45,"method":null,"native":false},{"column":14,"file":"node:internal/modules/cjs/loader","function":"Module._compile","line":1256,"method":"_compile","native":false},{"column":10,"file":"node:internal/modules/cjs/loader","function":"Module._extensions..js","line":1310,"method":".js","native":false},{"column":32,"file":"node:internal/modules/cjs/loader","function":"Module.load","line":1119,"method":"load","native":false},{"column":12,"file":"node:internal/modules/cjs/loader","function":"Module._load","line":960,"method":"_load","native":false},{"column":19,"file":"node:internal/modules/cjs/loader","function":"Module.require","line":1143,"method":"require","native":false},{"column":18,"file":"node:internal/modules/cjs/helpers","function":"require","line":119,"method":null,"native":false}]}

[Axtremus]

I am observing similar issue after upgrading to NodeBB package version 2.14.x.
Among error logs I am seeing are "ERR!" logs about not being able to resolve the nodebb-plugin-emoji plugins, and the application being "not responding."

Had to downgrade to NodeBB package version 2.13.4 to resolve the error and get the application back up and running.

Funny thing is, if I "upgrade" only to package version 2.14.0, I get only "WARN" logs about not being able to resolve the nodebb-plugin-emoji plugins, and the application will eventually get up and run. But if I take one more step to upgrade to 2.14.1, then the "ERR!" logs come back and the application will be stuck "not responding" again. Downgrading from 2.14.1 back to 2.14.0 did not help -- I get the same "ERR!" logs as mentioned above and the application remains "not responding." I have to downgrade back to 2.13.4 to get out of this hole.

For now, I will disable auto-update and hope that smarter people figure this out and put out a revised NodeBB 2.14.x package that won't have the same or similar issue. (Not really sure how this works ... if 2.14.0 and 2.14.1 really have issues, I'm hoping they will be removed such that auto-update will skip over them and move on to a working update when one becomes available.)

[joseph]

Can you run /app/pkg/start.sh in the web terminal once and then do the nodebb reset ? The reason is that /app/pkg/start.sh will "initialize" the app with necessary run time configuration and directories.

[hpz24]

@joseph Hi, facing similar problems and tried your solution but it does not fix the issue.

[nebulon]

Looks like a dependency issue between plugins. Have you tried to disable some of the mentioned plugins? Maybe they need to be updated upstream and will start working in some time then.

See https://docs.cloudron.io/apps/nodebb/#disabling-plugins how to do this. Otherwise restore the app and only update once those plugins have a new fixed release.

[hpz24]

Disabling the plugins has not been successful: The main problem is, that the terminal stops working when nodebb crashes.
So, I fiddled around a bit more:
I´ve the same problem as @Axtremus:
I downgraded to version 2.13.4 and the app works. Then I updated to 2.14.0 and had also the WARN entries but the app works again. Then I restarted the app manually and the behaviour was the same as at the beginning: WARN entries changed to ERR entries and nothing works.
That means: The problem is in Version 2.14: The container works well till the next restart, then it breaks. Even if the app has daily backups, the backups are useless because the backuped a failed state.
Conclusion: I restored back to 2.13.4, disabled Auto-Update and have lost 9 days of posts and threads ..

[joseph]

@hpz24 said in Issue Starting NodeBB on Cloudron After Server Restart:

The main problem is, that the terminal stops working when nodebb crashes.

Right, so you need to use the recovery mode for this - https://docs.cloudron.io/apps/#recovery-mode . Put app in recovery mode and then if you run /app/pkg/start.sh it will set up the run time directories. After that, you can run nodebb cli and disable plugins.

(cc @girish , please put above in docs)

[Axtremus]

Newbie questions here ...

1. How do I file ticket for this issue?
2. How/what do I monitor to know when this issue is resolved (such that I can reenable Auto-Update again)?

[joseph]

@Axtremus Unfortunately, there is no real way to support issues related to plugins. AFAIK, neither upstream nor Cloudron team tests all the combinations of various plugins and plugin versions and nodebb versions . IOW, one has to debug and find out how to resolve this.

I suggest something like this:

• Take a backup
• Clone the forum - https://docs.cloudron.io/backups/#clone-app
• Experiment with the clone to figure out what the issue is wrt deps. If this is getting too complex, I simply recommend not using those plugins.

At the end of the day, plugins is "code" just dropped into the main code codebase and anything can happen. It's not some simple configuration . For this forum which is nodebb, we barely use any plugins to prevent breakage.

[julian]

Hi, Julian from NodeBB team here. Sorry to hear there are dependency issues, they're tricky to debug.

Can I get a full record of the WARN and ERR messages? From what I can see in the image above, both the emoji-android plugin and reactions plugin have already been updated so that they request the correct version as a peer dependency.