Suggestion for prereleases

[d19dotca]

@girish, just a suggestion for consideration... It'd be really nice if when you release 7.5.0 for early testing by your advanced fearless (lol) users that the bugs we bring to your attention are addressed in a quick patch release that doesn't have too long of a waiting period, rather than the beta testers having to wait for a point release which also comes with new features and thus takes a long time to release.

For example, it's been quite a while since 7.5.0 came out as pre-release, and many of us reported various errors with it in our real-world testing. It was observed in those related posts that in most cases you resolved the issues right away within hours to days. But we're waiting quite a long time for those bug fixes to be delivered since they're being included in a new feature release at the same time. I think it'd be better for your users if you released 7.5.1 pre-release with the bug fixes quickly before allowing the branch with the new features being added yet in 7.5.x.

Hopefully the above makes sense. I do also recognize that we are beta testing and ideally would be using separate servers entirely for those tests or reverting back right away, but for one reason or another (bug fixes or a particular new feature we've been waiting on and really need if seen as low risk) we'll risk the update to pre-release software in production and we'll report back with any bugs we find. If anything catastrophic of course we'd just revert back to a backup, but many times the bugs we find aren't too high-impacting, however waiting a month or more for fixes you already made to code a day or two after it's reported seems like a long time to wait in my opinion. Thinking in that scenario it might be good to do it something like this...

1. New 7.5.0 pre-release
2. Gather feedback from beta testers on 7.5.0, generate bug fixes for those that are low-hanging fruit or high-impacting, after maybe just one to two weeks the bug fixes you've created can be released as 7.x.1 (or 7.x.0.1), even as pre-release still
3. Then continue with adding the new features again in the next point release and call it stable (ideally) by that time since the bug fixes would already have been released too

Not meant to be a criticism at all though, I assume you've chosen to release in the way you do currently for a reason, but just wanted to throw it an optional suggestion from a beta testers point of view. Thanks again for all the awesome improvements!! You guys are awesome!

[warg]

@d19dotca said in Suggestion for prereleases:

For example, it's been quite a while since 7.5.0 came out as pre-release, and many of us reported various errors with it in our real-world testing.

I think - regardless of beta or stable - there should be minor version fixes delivered, e. g. every 2 weeks. This is something I noticed as a disadvantage as well: v7.5.0 was called likely stable but at the moment the log file download is broken, websockets can run into timeout (and don't reconnect) and a few apps wait for bugfixes that are not that irrelevant. So basically waiting multiple weeks (or months?) to have this fixed or downgrading again is a bit unlucky.

So basically we suggest the same just different naming. This feels a bit like DirectAdmin where no real stable exists (stable is too outdated or lacks stuff you actually need while their beta updates are happening too often and break stuff too often.

[girish]

Thanks for the feedback! We discussed this internally a bit.

• Some are updating to pre-release on production instances (despite the warnings about unstable) and expecting fixes quickly. To us, quickly even means 1 month. We understand the idea of pushing new releases every week but this is very hard to do. Cloudron is really like an OS, there are lots of packages, docker images and deps and is not a simple tarball on github and let the user figure it out. As a comparison, there is a reason distros are not pushing new releases every week or two.

• Providing a way to update to unstable easily (just click a button) and then not providing fixes quickly is bound to disappoint at some point and it seems that point is now given this is upvoted a bit.

• As a lesson, we will become more conservative about pre-releases i.e we will release only after we have tested extensively ourselves and also completed the features. This is how we did it before and we actually didn't have many complaints. We only implemented pre-releases because people were enthusiastic to test early and seems like this is more trouble than it's worth.

[girish]

@warg said in Suggestion for prereleases:

v7.5.0 was called likely stable but at the moment the log file download is broken, websockets can run into timeout (and don't reconnect) and a few apps wait for bugfixes that are not that irrelevant.

7.5 is actually not released but is offered as a pre-release. Did you upgrade from 7.4 to 7.5 ? It should have informed you that this is unstable. Can you tell me how your upgrade flow (as in, how did you end up with 7.5)?

(But as mentioned, we agree that pre-releases should never have been offered. And releasing every 2 weeks doesn't fit in our product's model unless it's a security update).

[msbt]

@girish I for one have 4 machines that are on v7.5.0 and only one of those got updated manually (a free one), the other three updated on their own via cron, I was a bit surprised back then (although always happy to help with debugging) and upvoted this post because of that:

[avatar1024]

I'm also pretty sure that one of my instance updated automatically to 7.5.0.

@girish your post made a lot of sense, and indeed if people choose to update to something that's clearly mark as a pre-release then they should not expect stability or quick fixes...maybe the label 7.5.0 instead of something like 7.4.90 still felt "pretty stable" to most of us even with the pre-release flag since we're used to your excellent work on new releases ). This issue is really if the update got pushed automatically for some of us.

I get that 7.5.0 was a different beast to other major releases, but still it reminds me of other discussions on Cloudron update strategy.

I personally would be in favour anyway to have two options:

• leave as is: auto-update happens for all platform releases for the most enthusiastic users / least sensitive instances
• auto-update happens for all platform releases except x.x.0 (update can still be done manually if needed).

It feels that this would overall allow flexibility for users to choose their update strategy depending on their instances and their desire to test new releases / features, and flexibility for devs to push new releases when they see fit without having to worry about being too conservative or not enough (so if they need more user testing on some new stuff they can push a release a little earlier like they did for 7.5.0 without putting users at risks).

See discussions on here and here.

[nebulon]

I think we made a mistake with 7.5 where it was pushed out for some Cloudrons unintentionally for auto-updates for a brief moment, and it seems to me this is what caused more disruption that usual. Once a release is out, we push out updates slowly simply based on the first character alphabetically of the dashboard domain. That filter was too open for a short timeframe, where some Cloudrons already picked up the update info. This is a manual approach on our end, and we will be more careful and restrictive in the future.

It's great for us to have early feedback of course but if relevant Cloudrons break due to this, it is also a time-consuming, often manual support task to fix up affected ones. So it's a thin line to walk.

[warg]

@girish said in Suggestion for prereleases:

7.5 is actually not released but is offered as a pre-release. Did you upgrade from 7.4 to 7.5 ? It should have informed you that this is unstable. Can you tell me how your upgrade flow (as in, how did you end up with 7.5)?

I started with 7.4.x and read in the forum that it fixed something that did annoy me during the evaluation of the product so I thought that a upgrade sounds good and I just checked the logs: Looks like it did update itself via cronjob:

25.06.2023 cron Cloudron updated to version 7.5.0

(Like others reported.) In general v7.5 looks like a good improvement to v7.4 although some stuff is a bit broken (nothing too critcial).

[avatar1024]

This post is deleted!

[avatar1024]

@nebulon said in Suggestion for prereleases:

It's great for us to have early feedback of course but if relevant Cloudrons break due to this, it is also a time-consuming, often manual support task to fix up affected ones. So it's a thin line to walk.

@nebulon Makes perfect sense. And so what do you think of the idea above to allow for flexibility on both ends (users and devs)?

[d19dotca]

@girish said in Suggestion for prereleases:

Some are updating to pre-release on production instances (despite the warnings about unstable) and expecting fixes quickly. To us, quickly even means 1 month.

Just to clarify from my perspective a couple of things and add my two cents...

• Pre-releases in the past have nearly always gone very smoothly (which is awesome! ) and I suspect this is why more and more people are open to running the pre-release builds even on production instances, because major issues are thankfully so rare. Combine that with long-awaited features or fixes users have been eager for, it's all a great incentive for more users to test out the pre-release. And personally I think that's a really good situation to have experienced users on the pre-release as we've been able to help the Cloudron team identify many issues that likely would not have been caught as soon otherwise. This helps improve then Cloudron product for everyone.
• I think there's perhaps a difference of opinion with different users and Cloudron team on what "quickly" means to them. In most of the software I use (and I suspect this applies to others too), software updates are often seen every 1-3 weeks especially soon after a major version release since they tend to be more buggy and need an initial bug fix patch released quickly rather than waiting for the next major release months later. On top of that, I think part of the "quick release" expectation by some users comes from seeing how quickly fixes are completed by the amazing Cloudron team here when we report issues, often fixed in 24-48 hours or less I find, so when we see that being so quick then waiting 1-2+ months for the fix just seems understandably unusual or unexpected.

@girish said in Suggestion for prereleases:

Cloudron is really like an OS, there are lots of packages, docker images and deps and is not a simple tarball on github and let the user figure it out. As a comparison, there is a reason distros are not pushing new releases every week or two.

While I tend to agree, even on an OS distro there is usually an easy way to get later versions of various applications. This isn't something that's possible in Cloudron currently so we're more reliant on the "OS" (aka Cloudron) here for bug fixes.

@girish said in Suggestion for prereleases:

Providing a way to update to unstable easily (just click a button) and then not providing fixes quickly is bound to disappoint at some point and it seems that point is now given this is upvoted a bit.

Maybe something to consider... two different branches. Pre-release and stable. Users can pick which one they want to run on their systems, and pre-releases get updates every 1-2 weeks perhaps, and stable every few months or whenever a new release is ready and considered stable. More and more software applications seem to be doing it this way, such as 1Password for example and nearly any browser, and more. This might satisfy those users who want to stick to stable releases and those who want quicker bug fixes and such and enjoy reporting issues on pre-release software to better help the Cloudron community. I see this also as an opportunity for Cloudron to grow better quicker by getting more rapid feedback from experienced users.

@girish said in Suggestion for prereleases:

As a lesson, we will become more conservative about pre-releases i.e we will release only after we have tested extensively ourselves and also completed the features. This is how we did it before and we actually didn't have many complaints. We only implemented pre-releases because people were enthusiastic to test early and seems like this is more trouble than it's worth.

I tend to disagree on this one. In software - as you're well aware - beta testing is a critically important piece of the development cycle so that the final release is as least-buggy as possible. I know Cloudron used to beta test with only new deployments of Cloudron installs back in the day if I recall correctly, but the unfortunate part of that is that it isn't targeting the advanced users who want to test and report these types of issues encountered (I'm guessing most people installing Cloudron who would unknowingly get the new release would be newer users).

I think the Cloudron team should really embrace the beta testing / pre-release process and embrace working with experienced users in the forum here who are actively testing and reporting issues and enjoy helping improve the software with the fantastic Cloudron team. Tightening the process up I fear will only make things worse / more buggy before the final release as more things may go uncaught. You've got a good amount of people willing to test the product for you for basically free, why not take advantage it? It may create more bug reports which in-turn may be more work, sure, but that's a good thing IMO because they may otherwise go unnoticed and will then be sure to improve it for those who are not willing to beta test.

Side note: Maybe calling it a "beta" or even an "alpha" may be better because pre-release to some people implies it's close to a "release candidate" which in-turn implies it should be fairly stable. If the Cloudron team doesn't want so many people testing, then perhaps naming it as "beta" will give some users a bit more pause on installing it and only those experienced would be more willing to do it.

I know that's a long write-up (I offer my typical Canadian apology, lol), but I hope you can tell that I've given this a lot of thought and I hope that some of the notes above will be considered for how to handle releases in the future. I'm always happy to discuss further if any questions at all.