Auto-update to 8.3 - various apps down - database issue

[fbartels]

https://forum.cloudron.io/topic/13412/8.3.0-postgres-upgrade-failure

I ran into the same issue. For me all postgres databases were empty and the fix was to restore these specific apps from a backup from before the upgrade.

[robi]

One of mine upgraded with no issues.

[imc67]

Last night my big production Cloudron (out of 3) was updated but I'd noticed that on forehand so after reading issues I increased postgres memory a lot and made sure all apps were updated before. Luckily it went all good but it would've been a disaster if this one had longer downtime.

[robi]

@imc67 @staff perhaps that's a new feature Cloudron upgrades should have, especially when upgrading major database versions and schema updates which may need a bump in the memory requirements temporarily.

[shrey]

My Cloudron instance for auto upgraded last night and now more than 50 apps are down!

Retry > Restart isn't working either, for a lot of them.

[shrey]

@staff

This is a rather incomprehensible blunder on the part of Cloudron! The postgres databases have all been nuked!

I'm having to manually restore almost all of the affected apps, which is a very time-intensive task, not to mention the several hours of unscheduled downtime these services have been under.

[d19dotca]

This is strange to read, as I updated to 8.3.0 over the weekend without any issues (other than a couple of SpamAssassin rules I needed to update because of deprecations in the latest version of SA being used in 8.30 Cloudron). Definitely no outages other than a slightly delayed startup while the databases migrated.

For what it’s worth, I had 4 GB capacity allocated to the PostgreSQL database service, so maybe that helped?

[shrey]

@d19dotca I too, have 4GB allocated to the Postgres service. Besides that, the resource graphs didn't even go anywhere near the threshold, during the update process.

[d19dotca]

@shrey Interesting, I wonder what the magic number might be for memory then if that was the root cause. It likely depends on each server's environment too, it's probably something like X times the current/average use. My PG usage is typically only running around 512 MB at most, often less than that. So 4 GB was several times more than my daily usage numbers seem to represent.

On a side note with regards to the graphs, in my experience I find the graphs to be not too valuable, I think because it (correct me if I'm wrong though) only updates the usage every 5 minutes, so if it hits its max memory usage immediately in under 5 minutes and restarts for example then it won't necessarily be recorded. I may be wrong, but just my general experience, I tend to view the graphs with a grain of salt.

[nebulon]

@shrey do you have any errors in the postgres service logs or do you see similar/same errors in app logs? Did the postgres service start fine after the platform update?

[girish]

@shrey said in Auto-update to 8.3 - various apps down - database issue:

The postgres databases have all been nuked!

This is how the upgrade is carried out. The databases are exported, a new postgres is started from fresh and then they are all reimported. During this process postgres does have unlimited memory.

However, for reasons, we are yet to figure out, on some servers, it seems the reimport fails because postgres is somehow busy. So far, we haven't logs as to why this fails. The fix is as you found out - just do the reimport by restoring the apps (the upgrade automated this but the failure makes the end user do this manually).

[shrey]

@girish said in Auto-update to 8.3 - various apps down - database issue:

but the failure makes the end user do this manually

Well, this manual process costed me several hours of downtime of 'production' services, as well as another couple of hours for restoring them (at lot of backup files are really big, e.g. immich) 🫤

[girish]

@shrey yes, there is clearly some bug somewhere unfortunately

[CptPlastic]

SHAME ON YOU! I just woke up to a mess of cloudron servers being all messed up over this 8.3 update. To make matters worse almost every single database had to be restored. Some of this caused data loss because the apps on some accounts are used for logging 24/7. I learned a valuable lesson. Turn off auto update.

[CptPlastic]

@girish Man this is bad

[girish]

@CptPlastic if the cloudron is still in error state, would like to take a look at it. can you email us at support@cloudron.io ?

[nebulon]

Not sure if strong words help the cause, it is not like we introduce bugs or slack on testing on purpose.

I wonder where the data loss comes in though, there should be only a small timeframe between app backup and app being down (so no data can get changed/added) while the app was down.

[timconsidine]

I share the feelings of alarm but we do need to keep it in context - it’s IT - it always goes wrong - we are amazed and pleasantly, nay deliriously, happy when it works smoothly, as mostly Cloudron does.

Apart from root cause stuff and understanding why it is smooth for some and (what feels in the moment) near-disaster for others, the interesting point for me is : when is a backup a backup ? When can it be relied on ?

I couldn’t restore to the pre-update backups, needed to go further back, and then manually add lost stuff. In my case it wasn’t nice, but it wasn’t that hard, but maybe I was lucky, judging by other reports.

But that’s probably an unanswerable question.
I wonder if snapshots offer more than backups in this scenario. But I guess that’s outside Cloudron realm.

Maybe biggest takeaway from all this : don’t make big updates of core services auto-update. Give the user notice, and a chance to make a snapshot or some other cautious approach, and leave them to decide when to run it.

[mazarian]

Add me to the list of people experiencing the same issue. Hoping for a solution soon! While some apps were able to be restored from backup, I have multiple Chatwoot instances that will not restore.

[joseph]

@mazarian what is the error when you try to restore?