PostGres SQL Error update from v1.112.4 to v1.113.0

[arnaudguy]

After updating from v1.112.4 to v1.113.0, I encountered a PostgreSQL error. I have resolved the issue (see the bottom of the topic).

Error log at boot.

Mar 02 08:22:50 => Healthcheck error: Error: connect ECONNREFUSED 172.18.20.122:80
Mar 02 08:22:50 `enable_tracing` is now deprecated in favor of `traces_sample_rate = 1.0`.
Mar 02 08:22:50
Mar 02 08:23:00 => Healthcheck error: Error: connect ECONNREFUSED 172.18.20.122:80
Mar 02 08:23:05 main: == [advisory_lock_connection] object_id: 67420, pg_backend_pid: 567
Mar 02 08:23:05 Migrating to ValidateUserAgentDetailsOrganizationIdNullConstraint (20260126183256)
Mar 02 08:23:05 main: == 20260126183256 ValidateUserAgentDetailsOrganizationIdNullConstraint: migrating
Mar 02 08:23:05 main: -- execute("ALTER TABLE user_agent_details VALIDATE CONSTRAINT check_17a3a18e31;")
Mar 02 08:23:05 main: == [advisory_lock_connection] object_id: 67420, pg_backend_pid: 567
Mar 02 08:23:05 rake aborted!
Mar 02 08:23:05 StandardError: An error has occurred, this and all later migrations canceled:
Mar 02 08:23:05 2026-03-02T07:23:05Z
Mar 02 08:23:05 PG::CheckViolation: ERROR: check constraint "check_17a3a18e31" of relation "user_agent_details" is violated by some row
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:56:in `exec'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:56:in `block (2 levels) in raw_execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract_adapter.rb:1024:in `block in with_raw_connection'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.3/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract_adapter.rb:993:in `with_raw_connection'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `block in raw_execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.3/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract_adapter.rb:1141:in `log'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `raw_execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:538:in `internal_execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:137:in `execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/query_cache.rb:27:in `execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `execute'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration/default_strategy.rb:10:in `method_missing'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1059:in `block in method_missing'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1025:in `block in say_with_time'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/benchmark-0.4.1/lib/benchmark.rb:305:in `measure'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1025:in `say_with_time'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1048:in `method_missing'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/constraints_helpers.rb:122:in `block in validate_check_constraint'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/timeout_helpers.rb:26:in `disable_statement_timeout'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/constraints_helpers.rb:119:in `validate_check_constraint'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/constraints_helpers.rb:235:in `validate_not_null_constraint'
Mar 02 08:23:05 /home/git/gitlab/db/post_migrate/20260126183256_validate_user_agent_details_organization_id_null_constraint.rb:7:in `up'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:997:in `public_send'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:997:in `exec_migration'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/query_analyzer.rb:94:in `within'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migration_helpers/require_disable_ddl_transaction_for_multiple_locks.rb:71:in `exec_migration'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:978:in `block (2 levels) in migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/benchmark-0.4.1/lib/benchmark.rb:305:in `measure'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:977:in `block in migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:425:in `with_connection'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:976:in `migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1190:in `migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1539:in `block in execute_migration_in_transaction'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:616:in `block in within_new_transaction'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.3/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:613:in `within_new_transaction'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:361:in `transaction'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:229:in `retry_with_backoff'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:131:in `read_write'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:44:in `ddl_transaction'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1538:in `execute_migration_in_transaction'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1512:in `each'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1512:in `migrate_without_lock'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1457:in `block in migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1610:in `with_advisory_lock'
Mar 02 08:23:05 /home/git/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:14:in `with_advisory_lock'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1457:in `migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1264:in `up'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/migration.rb:1239:in `migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.3/lib/active_record/railties/databases.rake:93:in `block (2 levels) in <top (required)>'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/cli/exec.rb:59:in `load'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/cli/exec.rb:59:in `kernel_load'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/cli/exec.rb:23:in `run'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/cli.rb:451:in `exec'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/cli.rb:35:in `dispatch'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/cli.rb:29:in `start'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/exe/bundle:28:in `block in <top (required)>'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/lib/bundler/friendly_errors.rb:118:in `with_friendly_errors'
Mar 02 08:23:05 /home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/bundler-2.7.1/exe/bundle:20:in `<top (required)>'
Mar 02 08:23:05 /home/git/rbenv-root/versions/3.3.10/bin/bundle:25:in `load'
Mar 02 08:23:05 /home/git/rbenv-root/versions/3.3.10/bin/bundle:25:in `<main>'
Mar 02 08:23:05 Tasks: TOP => db:migrate
Mar 02 08:23:05 (See full trace by running task with --trace)
Mar 02 08:23:06 ==> Configuring OIDC auth
Mar 02 08:23:07 ==> Reusing existing host keys
Mar 02 08:23:07 ==> Fixing secrets
Mar 02 08:23:07 ==> Initializing gitlab shell
Mar 02 08:23:07 ==> Copying schema file
Mar 02 08:23:07 ==> Setting up tmp
Mar 02 08:23:08 ==> Fixing permissions
Mar 02 08:23:09 ==> Creating pg_trgm extension
Mar 02 08:23:09 NOTICE: extension "pg_trgm" already exists, skipping
Mar 02 08:23:09 ==> Upgrading existing db
Mar 02 08:23:10 => Healthcheck error: Error: connect ECONNREFUSED 172.18.20.122:80
Mar 02 08:23:11 `/home/git` is not writable.
Mar 02 08:23:11 Bundler will use `/tmp/bundler20260302-287-bfw5cp287' as your home directory temporarily.

Resolved with

dbc4fe1b1815a444d0a4222f157d4804a0=> SELECT
  tc.constraint_name,
  kcu.column_name,
  ccu.table_name AS foreign_table_name,
  ccu.column_name AS foreign_column_name
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu
  ON tc.constraint_name = kcu.constraint_name
JOIN information_schema.constraint_column_usage ccu
  ON ccu.constraint_name = tc.constraint_name
WHERE tc.table_name = 'user_agent_details'
  AND tc.constraint_type = 'FOREIGN KEY';
 constraint_name |   column_name   | foreign_table_name | foreign_column_name 
-----------------+-----------------+--------------------+---------------------
 fk_74db0b7cc2   | organization_id | organizations      | id
(1 row)

dbc4fe1b1815a444d0a4222f157d4804a0=> 
dbc4fe1b1815a444d0a4222f157d4804a0=> SELECT COUNT(*) AS nb_violations
FROM user_agent_details
WHERE organization_id IS NULL;
 nb_violations 
---------------
             1
(1 row)

dbc4fe1b1815a444d0a4222f157d4804a0=> BEGIN;
DELETE FROM user_agent_details
WHERE organization_id IS NULL;
COMMIT;

ALTER TABLE user_agent_details VALIDATE CONSTRAINT check_17a3a18e31;
BEGIN
DELETE 1
COMMIT
ALTER TABLE
dbc4fe1b1815a444d0a4222f157d4804a0=> SELECT conname, convalidated
FROM pg_constraint
WHERE conname = 'check_17a3a18e31';
     conname      | convalidated 
------------------+--------------
 check_17a3a18e31 | t
(1 row)

[simon]

Had the same problem. Thanks for the post.

[joseph]

This is an upstream bug - https://gitlab.com/gitlab-org/gitlab/-/issues/590908 and also https://gitlab.com/gitlab-org/gitlab/-/issues/591051 .

Somehow git.cloudron.io just fine