Cloudron Forum

We restored from backup to 1.35.0 and it is working again. Have not attempted the upgrade again.

It appears the database crashes during upgrade from 1.35.0 to 1.36.1. Here is what we see:

=> Sourcing /app/data/env.sh
Feb 28 16:21:22 => Starting chatwoot
Feb 28 16:21:22 => Upgrading existing db
Feb 28 16:21:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.20.35:3000
Feb 28 16:21:36 I, [2025-02-28T23:21:36.197228 #10] INFO -- : [rake ip_lookup:setup] IP_LOOKUP_API_KEY empty. Skipping geoip database setup
Feb 28 16:21:37 I, [2025-02-28T23:21:37.247106 #10] INFO -- : Migrating to CreateCaptainTables (20250104200055)
Feb 28 16:21:37 == 20250104200055 CreateCaptainTables: migrating ==============================
Feb 28 16:21:37 -- extension_enabled?("vector")
Feb 28 16:21:37 -> 0.0176s
Feb 28 16:21:37 -- enable_extension("vector")
Feb 28 16:21:37 WARNING: terminating connection because of crash of another server process
Feb 28 16:21:37 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Feb 28 16:21:37 HINT: In a moment you should be able to reconnect to the database and repeat your command.
Feb 28 16:21:37 rails aborted!
Feb 28 16:21:37 ActiveRecord::ConnectionNotEstablished: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (ActiveRecord::ConnectionNotEstablished)
Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 PG::ConnectionBad: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (PG::ConnectionBad) Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (PG::ConnectionBad)
Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (ActiveRecord::StatementInvalid) Feb 28 16:21:37 This probably means the server terminated abnormally Feb 28 16:21:37 before or while processing the request. Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 28 16:21:37 This probably means the server terminated abnormally
Feb 28 16:21:37 before or while processing the request.
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::ConnectionNotEstablished: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (ActiveRecord::ConnectionNotEstablished) Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (PG::ConnectionBad)
Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::StatementInvalid: PG::ConnectionBad: PQsocket() can't get socket descriptor (ActiveRecord::StatementInvalid) Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 StandardError: Failed to enable 'vector' extension. Read more at https://chwt.app/v4/migration (StandardError) Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:30:in rescue in setup_vector_extension'
Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:27:in setup_vector_extension' Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:5:in up'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (ActiveRecord::StatementInvalid) Feb 28 16:21:37 This probably means the server terminated abnormally Feb 28 16:21:37 before or while processing the request. Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:28:in setup_vector_extension'
Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:5:in up' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 28 16:21:37 This probably means the server terminated abnormally
Feb 28 16:21:37 before or while processing the request.
Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:28:in setup_vector_extension' Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:5:in up'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in `block (2 levels) in <main>'
Feb 28 16:21:37 Tasks: TOP => db:migrate
Feb 28 16:21:37 (See full trace by running task with --trace)

@girish Yes, I'm referring to the host MySQL. And I already followed the steps in that document, but even at innodb_force_recovery = 4, it will not start. Regarding backups, the last backup in /var/backups/snapshot is from last November.

And just to reiterate what I said in my post, this is purely educational. I've verified that the Baserow application's data has been copied elsewhere. So if it might be worth it to the community on a whole to dig into this, I'm happy to. But if you deem it a waste of time/resources, don't sweat it. Thanks.

I've run into an interesting issue here that I'm wondering if there may be value in troubleshooting. We have a Cloudron server that we used for Baserow. It's not a critical device, but I suspect it might have a few databases on it that I'd like to get access to. We did not have Cloudron setup to backup, but we did have a full image snapshot image that ran nightly on it (3rd party). Cloudron is running on a KVM VPS and this morning it was suspended due to non-payment (100% my error, forgot to update them with my new credit card). NBD, paid it and the server came back online. However, Cloudron was not starting properly and it appears to be MySQL related. So I set out to the MySQL recovery, but even with innodb_force_recovery = 4 it will not start (Performance schema tables initialization failed).

So I decided to restore to a local VM so I could at least get into Cloudron and see if the databases actually have data I care about (it's Baserow and we moved away to Nocodb, so I suspect it doesn't). The backup is from about 4 hours before the server was suspended, but I'm having the same issue. Now, this could make sense because it is a snapshot of a running system, I'm guessing the "damage" would be similar. Anyone interested in working through this? We can do a lot of messing around on the restored VM and then attempt a final run on the production. Let me know, thanks!

@nebulon I'm chatting with Rocket.Chat via their own server. They've asked me to provide docker setup & version and docker config. Is there a way to get this information from Cloudron?

@nebulon I don't disagree, but unfortunately, it doesn't help my current situation.

And just another note on this. I can't even work around this by setting the business hours in UTC because it doesn't allow going past midnight. So if I set a start time of 2PM and an end time of 2AM, it doesn't work at all.

@girish A user can't set the timezone, it pulls from the server. So the reason I came across this is setting business hours so that when we embed the live chat-functionality on our website, it doesn't show after business hours. I don't believe there is a way to show the set timezone. We just know it's not setting the timezone correctly because our live chat shows offline even though it's within the set business hours.

@girish As it turns out, this didn’t work. It still acts like it’s UTC.

@girish That worked, thank you!

How would I go about setting the timezone for the Rocket.Chat app? Anything I searched for suggested php.ini, but RC is via docker. I'm guessing the env file, but I'm not exactly sure how this should be formatted. RC doesn't have the ability to edit the timezone from within the app and uses the server timezone. Thanks.