Chatwoot 1.36.1 Upgrade Issue

[cleveradmin]

It appears the database crashes during upgrade from 1.35.0 to 1.36.1. Here is what we see:

=> Sourcing /app/data/env.sh
Feb 28 16:21:22 => Starting chatwoot
Feb 28 16:21:22 => Upgrading existing db
Feb 28 16:21:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.20.35:3000
Feb 28 16:21:36 I, [2025-02-28T23:21:36.197228 #10] INFO -- : [rake ip_lookup:setup] IP_LOOKUP_API_KEY empty. Skipping geoip database setup
Feb 28 16:21:37 I, [2025-02-28T23:21:37.247106 #10] INFO -- : Migrating to CreateCaptainTables (20250104200055)
Feb 28 16:21:37 == 20250104200055 CreateCaptainTables: migrating ==============================
Feb 28 16:21:37 -- extension_enabled?("vector")
Feb 28 16:21:37 -> 0.0176s
Feb 28 16:21:37 -- enable_extension("vector")
Feb 28 16:21:37 WARNING: terminating connection because of crash of another server process
Feb 28 16:21:37 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Feb 28 16:21:37 HINT: In a moment you should be able to reconnect to the database and repeat your command.
Feb 28 16:21:37 rails aborted!
Feb 28 16:21:37 ActiveRecord::ConnectionNotEstablished: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (ActiveRecord::ConnectionNotEstablished)
Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 PG::ConnectionBad: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (PG::ConnectionBad) Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (PG::ConnectionBad)
Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (ActiveRecord::StatementInvalid) Feb 28 16:21:37 This probably means the server terminated abnormally Feb 28 16:21:37 before or while processing the request. Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 28 16:21:37 This probably means the server terminated abnormally
Feb 28 16:21:37 before or while processing the request.
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::ConnectionNotEstablished: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (ActiveRecord::ConnectionNotEstablished) Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: connection to server at "fd00:c107:d509::5", port 5432 failed: FATAL: the database system is in recovery mode (PG::ConnectionBad)
Feb 28 16:21:37 connection to server at "172.18.30.2", port 5432 failed: FATAL: the database system is in recovery mode
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::StatementInvalid: PG::ConnectionBad: PQsocket() can't get socket descriptor (ActiveRecord::StatementInvalid) Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 StandardError: Failed to enable 'vector' extension. Read more at https://chwt.app/v4/migration (StandardError) Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:30:in rescue in setup_vector_extension'
Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:27:in setup_vector_extension' Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:5:in up'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>' Feb 28 16:21:37 2025-02-28T23:21:37Z Feb 28 16:21:37 Caused by: Feb 28 16:21:37 ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (ActiveRecord::StatementInvalid) Feb 28 16:21:37 This probably means the server terminated abnormally Feb 28 16:21:37 before or while processing the request. Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:28:in setup_vector_extension'
Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:5:in up' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in block (2 levels) in <main>'
Feb 28 16:21:37 2025-02-28T23:21:37Z
Feb 28 16:21:37 Caused by:
Feb 28 16:21:37 PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly (PG::ConnectionBad)
Feb 28 16:21:37 This probably means the server terminated abnormally
Feb 28 16:21:37 before or while processing the request.
Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:28:in setup_vector_extension' Feb 28 16:21:37 /app/code/db/migrate/20250104200055_create_captain_tables.rb:5:in up'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:26:in block (3 levels) in <main>' Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in each'
Feb 28 16:21:37 /app/code/lib/tasks/db_enhancements.rake:18:in `block (2 levels) in <main>'
Feb 28 16:21:37 Tasks: TOP => db:migrate
Feb 28 16:21:37 (See full trace by running task with --trace)

[cleveradmin]

We restored from backup to 1.35.0 and it is working again. Have not attempted the upgrade again.

[BrutalBirdie]

Please format your logs in a code block like

```
your code goes here
```

Resulting in

your code goes here

Make it less difficult for me and everyone's eyes to read your logs

[BrutalBirdie]

Feb 28 16:21:37 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

Did the app or the Postgres service maybe run out of memory? resulting to this issue?

[nebulon]

@cleveradmin can you try to clone the app instance and then perform the update on the cloned one? This way we can see if it is reproducible.