Gitlab /admin/runner error 500

[RaV]

@girish The error in the log seems to be not so usefull.

Jan 23 10:55:32 172.18.0.1 - - [23/Jan/2021:09:55:32 +0000] "GET /admin/runners HTTP/1.1" 500 2926 "https://git.aaa.bb/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15
Jan 23 10:55:35 172.18.0.1 - - [23/Jan/2021:09:55:35 +0000] "POST /api/v4/jobs/request HTTP/1.1" 500 39 "-" "gitlab-runner 13.7.0 (13-7-stable; go1.13.8; linux/amd64)"

In recovery mode in the webterminal I am unable to type or see anything this is what seems strange. Yes it is /admin/runners sorry.

[girish]

@rav GitLab logs are under /run/gitlab/log/. There are two files there application.log and production.log. Maybe you can load the page and then look for the backtrace in those files.

[RaV]

@girish found this in production.log

Started GET "/admin/runners" for x.x.x.x at 2021-01-26 10:41:57 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 88ms (ActiveRecord: 1.9ms | Elasticsearch: 0.0ms | Allocations: 48002)
  
ActionView::Template::Error ():
    36: 
    37:   .col-sm-6
    38:     .bs-callout
    39:       = render partial: 'ci/runner/how_to_setup_runner',
    40:                 locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
    41:                           type: 'shared',
    42:                           reset_token_url: reset_registration_token_admin_application_settings_path }
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:38:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'
app/models/application_setting_implementation.rb:334:in `runners_registration_token'
lib/gitlab/current_settings.rb:20:in `method_missing'
app/views/admin/runners/index.html.haml:39
app/controllers/application_controller.rb:133:in `render'
app/controllers/application_controller.rb:494:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:485:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:479:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:544:in `sentry_context'
app/controllers/application_controller.rb:472:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:464:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

[girish]

@rav I found the exact same error in https://stackoverflow.com/questions/54216933/internal-server-error-500-while-accessing-gitlab-admin-runners .

As per the instruction in the stackoverflow post, can you try this in Cloudron's Web terminal?:

# su - git  
$ cd gitlab
$ bundle exec rails c -e production # (takes about 10 sec to bring up rails terminal)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!

[RaV]

@girish said in Gitlab /admin/runner error 500:

ApplicationSetting.current.reset_runners_registration_token!

Tried this and the Rails console said => True.
The /admin/runners page is still not available. Any other logs I could check?

[girish]

@rav said in Gitlab /admin/runner error 500:

The /admin/runners page is still not available. Any other logs I could check?

So, same backtrace? Have you tried restarting the app just to be sure?

[RaV]

@girish looks the same to me yes. did a restart of the app but it did not help.

Started GET "/admin/runners" for a.b.c.d at 2021-01-30 10:47:26 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 124ms (ActiveRecord: 29.1ms | Elasticsearch: 0.0ms | Allocations: 96262)
  
ActionView::Template::Error ():
    16:   .table-section.section-10
    17:     .table-mobile-header{ role: 'rowheader' }= _('Runner token')
    18:     .table-mobile-content
    19:       = link_to runner.short_sha, admin_runner_path(runner)
    20: 
    21:   .table-section.section-20
    22:     .table-mobile-header{ role: 'rowheader' }= _('Description')
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:35:in `block in add_authentication_token_field'
app/models/ci/runner.rb:265:in `short_sha'
app/views/admin/runners/_runner.html.haml:19
app/views/admin/runners/index.html.haml:134
app/views/admin/runners/index.html.haml:133
app/controllers/application_controller.rb:125:in `render'
app/controllers/application_controller.rb:482:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:473:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:467:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:532:in `sentry_context'
app/controllers/application_controller.rb:460:in `block in set_current_context'
lib/gitlab/application_context.rb:56:in `block in use'
lib/gitlab/application_context.rb:56:in `use'
lib/gitlab/application_context.rb:22:in `with_context'
app/controllers/application_controller.rb:451:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

[girish]

@rav I can't make out what the issue is. Do you think you can reach out to us at support@cloudron.io and give us SSH access so I can debug gitlab a bit to see what the issue is? Alternately, you have to ask on the GitLab forums on what to do for this.

[RaV]

@girish okay opened the support ticket from my cloudron

[girish]

Fix is documented in https://git.cloudron.io/cloudron/gitlab-app/-/issues/13