8.3.1 upgrade

[ivan-petro]

Hello @girish , @joseph

After a second attempt to upgrade to version 8.3.1, I'm still encountering serious issues.

I’ve already spent a significant amount of time creating a new server and restoring from backups. Once again, during the upgrade to 8.3.1, all apps with PostgreSQL are stuck in the state:

Restarting - Waiting for platform to initialize

Checking the box logs, I see repeated PostgreSQL backup and export operations, for example:

2025-03-24T20:10:48.541Z box:services Backing up postgresql  
2025-03-24T20:10:49.129Z box:services pipeRequestToFile: connected with status code 200  
2025-03-24T20:10:49.152Z box:services exportDatabase: Exporting addon postgresql of app 36863357-62c8-4750-9f7e-a20040bc1d15  
...  
2025-03-24T20:11:34.186Z box:services exportDatabase: Exporting addon postgresql of app 42f73f7a-f67b-4690-871a-f422c56bf4fd  

I'm unsure if I should just wait — and if so, how long?
Why is there no way to stop this process or manually recover my apps?

From a user experience perspective, this feels unfriendly and very frustrating.

Has anyone successfully upgraded to 8.3.1?
At this point, I’ve already spent over a week trying to make it work.

[ivan-petro]

@joseph , @girish and one more question:
how to quickly rollback cloudron update? I see in docs only new setup. why I can't just revert update ? I can't even remove this status "Platform status
Starting services, this can take a while". Cloudron is frozen.

[ivan-petro]

@joseph, @girish — let me help clarify some real-world use cases your clients face:

We use NocoDB, Directus, n8n, and similar tools with large datasets.
Cloudron is an excellent hosting platform and fully capable of being used in production environments.

However, it’s concerning to see how the update logic currently works — for example, during a version upgrade, the script upgrades the database and freezes all other apps.
Imagine a client running NocoDB with 500 GB of data — even if it's the only app, this can be a serious issue.

You've added powerful applications to the Cloudron App Store that are clearly meant for serious, data-heavy production use. But this makes it even more critical to improve update and recovery mechanisms.

Please consider implementing the following improvements:

• Add the ability to force-stop an update and roll back to the previous stable state.
• Ensure that other apps are not frozen during an upgrade, especially if they are not directly affected.

I’m even willing to pay for such improvements as an optional feature or extension.

I’ve already invested a lot of time trying to resolve these issues, and I’m still stuck — any help or solution would be highly appreciated.

[ivan-petro]

Also, just to emphasize how critical this is:

It’s crazy — I’ve been sitting here for 6 hours, watching all my apps stuck in

Restarting - Waiting for platform to initialize

Meanwhile, the upgrade script is running endlessly, doing some kind of background database operations.
Then come the Redis errors, and even after restarting the server, the system starts upgrading the PostgreSQL database again.

This feels like a never-ending loop with no control or visibility into what’s happening. For production environments, this is just not acceptable.

I really hope that people who haven’t upgraded yet will read this post and maybe think twice — or better, go find two days to enjoy life instead of debugging this mess.

[ivan-petro]

@girish @joseph

One more note:

I strongly recommend everyone to use Cloudron in real-case scenarios — not just for testing.

For example: try running n8n + NocoDB (both available in your App Marketplace) with 300–500 GB of data.

Let’s imagine a realistic use case:

• n8n logging is enabled to keep execution history for 1 month
• You have 100+ workflows running for your business
• In NocoDB, you're storing orders, products, customer data, etc.

Now try upgrading Cloudron in that environment…
It would be interesting to see how the platform handles it.

If you’d like, I’d be happy to help create real-world test cases like these — they might help uncover issues before production users hit them.

[ivan-petro]

Another question:

Why am I seeing so many Redis errors after the PostgreSQL upgrade?

Here’s a snippet from the logs:

2025-03-24T22:06:01.057Z box:services Waiting for redis-39d103ac-9d2b-44f1-9e38-63605e9da5d0  
2025-03-24T22:06:01.059Z box:services Attempt 1 failed. Will retry: Network error waiting for redis-39d103ac-9d2b-44f1-9e38-63605e9da5d0: connect ECONNREFUSED 172.18.0.21:3000  
2025-03-24T22:06:16.211Z box:services startRedis: stopping and deleting previous redis container redis-3a4d573e-8262-4609-b56e-ecb2837a53d7  
2025-03-24T22:06:16.631Z box:services startRedis: starting redis container redis-3a4d573e-8262-4609-b56e-ecb2837a53d7  
2025-03-24T22:06:16.634Z box:shell services: /bin/bash -c docker run --restart=always -d --name=redis-3a4d573e-8262-4609-b56e-ecb2837a53d7  

And yet, in the Cloudron dashboard, all Redis services appear as running.

This is insane — I rebooted the server, and the PostgreSQL upgrade process restarted again.
And still, all my apps remain stuck in the state:

Restarting - Waiting for platform to initialize

It’s a complete deadlock, and I have no visibility or control over what’s happening.

[mazarian]

Did you update your system to 8.3.1? This issue has already been addressed. If I were a mod I'd find your abrupt tone pretty off-putting. With your level of expectation and use case, you should be paying for support contracts on enterprise plans. Your uptime/support expectations don't match up with Cloudron.

[nebulon]

Since the update also updates postgres all postgres databases are first exported, then the db container is rebuilt with the new version, then the databases for each app are re-imported. This may take quite some time depending on the system resources (memory and disk I/O mostly here). Do you see any errors in the postgres service logs?

As @mazarian mentioned, we do have a pricing plan for business (which you also mention you are using Cloudron for) where we can give more timely and direct support. Just to set expectation clear.

[ivan-petro]

@mazarian didn't understand you.

Yes, I did update to 8.3.1 — and I’m highlighting the issues that happen after the upgrade.
This is a community forum, and I’m not contacting support asking anyone to urgently fix my server.
I handled everything myself, including a full recovery from backups and getting all apps back online manually.

I posted here to share the issue, specifically that apps were unavailable for hours, and to suggest that maybe the upgrade scripts could be improved or at least include warnings for such cases.

Yes, I’m already considering moving to a paid support plan — that’s clear.
But I fail to see how faster support would help with the fact that an upgrade process locks up all apps, shows endless Redis errors, and triggers long PostgreSQL operations.

This forum exists exactly for this kind of discussion.
If moderators feel my messages are out of place — they’re free to delete them.

But please don’t jump in if you don’t understand the issue. I’m sharing a real-world problem that any production user could face — and I believe it’s important to speak up.

[ivan-petro]

Thanks for the quick reply.

I performed several service restarts, and finally — after about 12 hours — the upgrade completed successfully.

Memory isn't an issue in my case. I’ve allocated the maximum allowed RAM for all services. The server has 128 GB of RAM and 32 CPU cores, so the system resources are more than sufficient.

That said, it wasn’t immediately clear that everything was progressing normally — it looked like the process was stuck for a long time.
I apologize if some of my earlier comments came off the wrong way — I was just genuinely frustrated, as this was the first time an upgrade took so long in production, with no way to stop or roll back.

Also, having to migrate to a new server after the first failed attempt added to the pressure.

I've made my conclusions — and for the next production Cloudron server, I’m seriously considering switching to a different pricing plan with support included.

If any of my previous posts seemed too harsh or inappropriate in tone, please feel free to delete them — no hard feelings.
Thanks again for the clarification and support.

[nebulon]

Glad it worked out in the end. The process could be a bit more chatty to give database export/import progress.

That upgrade moved Postrgres from version 14 to version 16 btw, so such large Postgres upgrades are hopefully done for some time now.

[robi]

@nebulon I think it's important to highlight how impressive this actually is.

A DB upgrade is no small feat for one app, let alone 60-80 apps and multiple DBs that many Cloudrons here run.

Then to jump across two DB version upgrades, all while seamlessly restitching everything back up so apps function properly.

Traditionally there would be a team of SREs doing this for a week. Yet it happens in a few hours if not minutes.

So there will be some red timeout messages and UI glitches while the process is going on (the red messages were waiting for me this AM while the system upgrade happened over night as I left my dashboard tab open), however with some patience and understanding everything comes back up as if by self-healing magic.

Sure it could be more polished (how about making a browser refresh after a successful upgrade to clear all the red messages?), but once you know, it's just more important to be in gratitude that it all just works and business continues as usual.

I for one appreciate this magic and hence the makers of this wonderful self-hosting system from the time I first met them during a meetup in Silicon Valley, CA.

Cloudron gives me TONS of time back, which is also why I can take the time to write about it, run fun experiments, and not have to be stuck troubleshooting standard app deployment woes.

Thank you Cloudron & Team for another seamless upgrade and intangibly so much more.