GitLab 1.43.0 Not Responding - key must be 32 bytes or longer

[CarbonBee]

Hi,

I have a major issue on both my GitLab servers.I just updated them to 1.43.0 and both are not responding.
Here is the full log :

Dec 02 08:07:22 ==> Enabling LDAP integration
Dec 02 08:07:22 ==> Reusing existing host keys
Dec 02 08:07:22 ==> Fixing secrets
Dec 02 08:07:22 ==> Initializing gitlab shell
Dec 02 08:07:22 ==> Copying schema file
Dec 02 08:07:22 ==> Setting up tmp
Dec 02 08:07:23 ==> Fixing permissions
Dec 02 08:07:23 ==> Creating pg_trgm extension
Dec 02 08:07:23 NOTICE: extension "pg_trgm" already exists, skipping
Dec 02 08:07:23 ==> Upgrading existing db
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:29: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:30: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:24 `/home/git` is not writable.
Dec 02 08:07:24 Bundler will use `/tmp/bundler20201202-44-14wdz8y44' as your home directory temporarily.
Dec 02 08:07:24 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:24 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:24 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:28 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:32 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:34 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:36 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/acts-as-taggable-on-6.5.0/lib/acts_as_taggable_on/tagging.rb:9: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.9.0/lib/marginalia.rb:94: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql_adapter.rb:648: warning: The called method `execute_and_clear_without_marginalia' is defined here
Dec 02 08:07:43 == 20201008013434 GenerateCiJwtSigningKey: migrating ==========================
Dec 02 08:07:43 rake aborted!
Dec 02 08:07:43 StandardError: An error has occurred, this and all later migrations canceled:
Dec 02 08:07:43
Dec 02 08:07:43 key must be 32 bytes or longer
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:60:in `crypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:36:in `encrypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:266:in `encrypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:350:in `encrypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:165:in `block (2 levels) in attr_encrypted'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:77:in `block in attr_encrypted'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:50:in `public_send'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:50:in `_assign_attribute'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:42:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:29:in `perform_attribute_assignment'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/persistence.rb:620:in `block in update'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:212:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/persistence.rb:619:in `update'
Dec 02 08:07:43 /home/git/gitlab/db/migrate/20201008013434_generate_ci_jwt_signing_key.rb:21:in `block in up'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:70:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:70:in `block in find_each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:238:in `block in in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:222:in `loop'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:222:in `in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:135:in `find_in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:69:in `find_each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/querying.rb:21:in `find_each'
Dec 02 08:07:43 /home/git/gitlab/db/migrate/20201008013434_generate_ci_jwt_signing_key.rb:20:in `up'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:831:in `exec_migration'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:811:in `block in migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:810:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1002:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1361:in `block in ddl_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:212:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1361:in `ddl_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1280:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1280:in `migrate_without_lock'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1229:in `block in migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1382:in `with_advisory_lock'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1229:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1061:in `up'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1036:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <top (required)>'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/railties/databases.rake:84:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <top (required)>'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'

I'm filling a support ticket right now, but if anyone has an idea on that issue...

[girish]

@carbonbee I thought this was sorted out ages ago - https://git.cloudron.io/cloudron/gitlab-app/-/issues/9 . I have fixed up your instance, should be OK now.

[RaV]

@girish Looks like I see the same problem. How can this be fixed?

[girish]

@rav Try this:

• Go to App -> Repair. Enable Recovery Mode
• Then, Console -> Terminal.
  • Run "pwgen -1cns 64" . This will give a 64-bit token.
  • Edit /app/data/secrets.yml. There is a key there call db_key_base. Delete the value there and replace it with the token above. (This is what gitlab is complaining about, it's saying that the key here is too short).
• App -> Repair. Disable Recovery Mode

[RaV]

@girish said in GitLab 1.43.0 Not Responding - key must be 32 bytes or longer:

/app/data/secrets.yml

Worked thank you