Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!

[qwinter]

First, I really appreciate cloudron. What it does is really amazing; a quantum jump from any other solution. I would not selfhost without it.

I have a nodebb forum that just silently broke when I wanted to remove the quill plugin. It seems the current nodebb version is buggy:
https://sudonix.com/topic/275/moving-away-from-the-quill-js-plugin/7

This brings me to the topic: do we get the latest version of everything (within reason)? Is anyone tracking whether that version is reportedly buggy? With my desktop OS, manjaro, I check the forums before they dump a big update. I install it a few days after I've seen the issues reported on the forums' "stable update" channel. I imagine there will be similar things for most of the apps cloudron supports.

We self-hosters probably cannot afford a devops person (let alone team). If a new version of an app gets published to cloudron, and borks a public production site, there's absolutely nothing we can do other than revert to a backup. This situation makes me uneasy. I want to have peace of mind more than I want to be independent and self-hosting!

Anyone thinking the same? What's your take? How much of your week goes to babysitting things that break with updates when you self host (even with cloudron)? I'm starting to feel this could be an uncomfortably high number.

[robi]

Cloudron isn't the issue here, although I love the slogan:

Don't Self Host Without It!

NodeBB updates have been mostly bulletproof, thanks to Cloudron and team. However the plugins are another matter and distinctly a NodeBB issue.

You may want to restart NodeBB or do a rebuild from within the forum admin options to correct any breaking change quill left.

Until the NodeBB team figures out a better way to support poor, mediocre and good plugins, it remains fragile and less predictable.

Hence the Cloudron team doesn't recommend adding too many such plugins, and only testing one at a time.

[BrutalBirdie]

@qwinter said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

This brings me to the topic: do we get the latest version of everything (within reason)?

If a Software provider releases a public stable release, it is expected to be stable!
But if that is not the case you really can't blame the one deploying it (In this case Cloudron).
If a release is still buggy it should be a release candidate and not be published.
(Not counting in plugins!)

@qwinter said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

Is anyone tracking whether that version is reportedly buggy?

Given the scope of all the apps cloudron has and the limited amount of resources we have, there are lifecycle tests for each app.
https://git.cloudron.io/cloudron/nodebb-app/-/blob/master/test/test.js
These tests can always be improved since the scope of testing can be quite huge.
For example, we can't test each existing plugin for NodeBB, Discourse, Nextcloud yada yada since the room for errors will increase exponentially with each extra plugin.

@qwinter said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

We self-hosters probably cannot afford a devops person (let alone team). If a new version of an app gets published to cloudron, and borks a public production site, there's absolutely nothing we can do other than revert to a backup. This situation makes me uneasy. I want to have peace of mind more than I want to be independent and self-hosting!

If the lifecycle tests is successful we would consider it ready to go.

However, what you are describing could be solved.
Disable automatic updates for apps which you deem production necessary, which are not allowed to break. ever! aka. never change a running system.

You can always clone an app and test the update yourself on the cloned version before upgrading the production one.
But this is one of the points which I don't want to do with cloudron and I am happy not to have to do.

@qwinter said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

Anyone thinking the same? What's your take? How much of your week goes to babysitting things that break with updates when you self host (even with cloudron)? I'm starting to feel this could be an uncomfortably high number.

Little to none when it comes to cloudron.
My latest 30 minutes of 'babysitting' was my private cloudron which I thought crashed. But no, the mainboard of the server simply died and the tech crew in the datacenter switched it out in less in 10 Minutes after my support call and mail.

Here is a screenshot of one my main cloudrons availability report for my company with all internal tools for this year so far.

1 = Up and Running and 0 = Down
And all the downtimes you can see there are from scheduled security reboots.
I could click together a graph for that with the linked trigger which reports a needed reboot, I am lazy now.

I also have graphs of each docker container in cloudron, like cpu/ram/disk i/o, exit codes, network i/o yada YADA.
I am monitoring everything from hardware level components to software level services.

For me, if a Cloudron Server sends out notifications to me that something is not right, then something is really not right.

[qwinter]

@BrutalBirdie said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

Here is a screenshot of one my main cloudrons availability report for my company with all internal tools for this year so far.

That's amazingly detailed. It shows you are a pro with servers. And that you really care about keeping things running. Which is not something I am or want to be. I'm a businessman nowadays and while I love to solve tech problems every now and then, it's not a good use of my time. I really believe selfhosting is a better way to deal with technology (and it foments freedom from bigTech), but the price of admission is too high at the moment for me.

[girish]

@qwinter said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

This brings me to the topic: do we get the latest version of everything (within reason)? Is anyone tracking whether that version is reportedly buggy?

We provide the latest releases of apps as quickly as we can. There are automated tests for all app packages. The tests install the app, backup/restart/restore the app, check if update works against the previous app package etc. What is not tested is if the app itself broke internally. For example, maybe the latest version of nodebb is unable to upload images or render emojis or Firefox 99 doesn't work. This QA/testing is the responsibility of upstream authors. It is assumed they make releases based on their own quality test suites.

Finally, for plugins, this is not tested and from experience, I can tell you that I don't think anyone tests all versions and combinations - this includes plugin authors and the NodeBB authors.

Coming back to your problem: given above, if you rely on plugins, best approach is to disable automatic updates for NodeBB - https://docs.cloudron.io/updates/#app-updates . Then, when you have some time available, clone the app, update plugins and nodebb and see if things are good. If it is, then you update the main app manually.

[girish]

@qwinter On the general topic of plugins, this is a very hard problem to solve (even if you pay someone else).

If you forget selfhosting and consider managed hosting - they don't allow most plugins. See GoDaddy plugin list, WPEngine, Blue host, discourse managed hosting to name a few. SaaS has no concept of plugins anyway.

This leaves us with self-hosting. So far, practically, no app implements plugin sandboxing properly (only haraka tries). This is the reason the whole app breaks when a single plugin breaks. It also means that if you are not up to date enough, well, your app gets hacked.

For your specific nodebb quill plugin , this was reported and fixed only 4 days ago. You have to realize it has only 13 stars on GitHub and that's also the eyeballs it gets.

Maybe I am missing some other case here (which is what you are referring to) ? How do you think things can be improved?

[qwinter]

@girish I know, it's a terribly hard to fix problem (supporting plugins).
I don't have a solution. It's just that in my case I have to weight in this and make a decision.
Either nodebb without plugins, or do not self host.

Interesting, never thought it would come to this

[girish]

@qwinter said in Nodebb reported to be buggy. Not having control over cloudron's release pace could be a problem!:

Either nodebb without plugins, or do not self host.

I think just being prudent about which plugins you use pretty much keeps things manageable (for this forum, anyway...).

[qwinter]

@girish looks like nodebb self healed. The 'broken' state it was in is not broken anymore. This is good news