Gitlab /admin/runner error 500

[RaV]

Somehow my Gitlab seems to be broken. When I try to access the Admin page for the runners I get a 500 error. While trying to fix it I figured out that my Gitlab does not start when I try to enable the recovery mode. Any way to fix this?

[girish]

@rav Is there any error in the logs when you get a 500? If you disable recovery mode, does the app come back up? It is expected that the app won't start in recovery mode. You have to go the web terminal and then run /app/pkg/start.sh to start the app.

I guess the page is /admin/runners and not /admin/runner ?

[RaV]

@girish The error in the log seems to be not so usefull.

Jan 23 10:55:32 172.18.0.1 - - [23/Jan/2021:09:55:32 +0000] "GET /admin/runners HTTP/1.1" 500 2926 "https://git.aaa.bb/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15
Jan 23 10:55:35 172.18.0.1 - - [23/Jan/2021:09:55:35 +0000] "POST /api/v4/jobs/request HTTP/1.1" 500 39 "-" "gitlab-runner 13.7.0 (13-7-stable; go1.13.8; linux/amd64)"

In recovery mode in the webterminal I am unable to type or see anything this is what seems strange. Yes it is /admin/runners sorry.

[girish]

@rav GitLab logs are under /run/gitlab/log/. There are two files there application.log and production.log. Maybe you can load the page and then look for the backtrace in those files.

[RaV]

@girish found this in production.log

Started GET "/admin/runners" for x.x.x.x at 2021-01-26 10:41:57 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 88ms (ActiveRecord: 1.9ms | Elasticsearch: 0.0ms | Allocations: 48002)
  
ActionView::Template::Error ():
    36: 
    37:   .col-sm-6
    38:     .bs-callout
    39:       = render partial: 'ci/runner/how_to_setup_runner',
    40:                 locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
    41:                           type: 'shared',
    42:                           reset_token_url: reset_registration_token_admin_application_settings_path }
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:38:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'
app/models/application_setting_implementation.rb:334:in `runners_registration_token'
lib/gitlab/current_settings.rb:20:in `method_missing'
app/views/admin/runners/index.html.haml:39
app/controllers/application_controller.rb:133:in `render'
app/controllers/application_controller.rb:494:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:485:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:479:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:544:in `sentry_context'
app/controllers/application_controller.rb:472:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:464:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

[girish]

@rav I found the exact same error in https://stackoverflow.com/questions/54216933/internal-server-error-500-while-accessing-gitlab-admin-runners .

As per the instruction in the stackoverflow post, can you try this in Cloudron's Web terminal?:

# su - git  
$ cd gitlab
$ bundle exec rails c -e production # (takes about 10 sec to bring up rails terminal)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!

[RaV]

@girish said in Gitlab /admin/runner error 500:

ApplicationSetting.current.reset_runners_registration_token!

Tried this and the Rails console said => True.
The /admin/runners page is still not available. Any other logs I could check?

[girish]

@rav said in Gitlab /admin/runner error 500:

The /admin/runners page is still not available. Any other logs I could check?

So, same backtrace? Have you tried restarting the app just to be sure?

[RaV]

@girish looks the same to me yes. did a restart of the app but it did not help.

Started GET "/admin/runners" for a.b.c.d at 2021-01-30 10:47:26 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 124ms (ActiveRecord: 29.1ms | Elasticsearch: 0.0ms | Allocations: 96262)
  
ActionView::Template::Error ():
    16:   .table-section.section-10
    17:     .table-mobile-header{ role: 'rowheader' }= _('Runner token')
    18:     .table-mobile-content
    19:       = link_to runner.short_sha, admin_runner_path(runner)
    20: 
    21:   .table-section.section-20
    22:     .table-mobile-header{ role: 'rowheader' }= _('Description')
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:35:in `block in add_authentication_token_field'
app/models/ci/runner.rb:265:in `short_sha'
app/views/admin/runners/_runner.html.haml:19
app/views/admin/runners/index.html.haml:134
app/views/admin/runners/index.html.haml:133
app/controllers/application_controller.rb:125:in `render'
app/controllers/application_controller.rb:482:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:473:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:467:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:532:in `sentry_context'
app/controllers/application_controller.rb:460:in `block in set_current_context'
lib/gitlab/application_context.rb:56:in `block in use'
lib/gitlab/application_context.rb:56:in `use'
lib/gitlab/application_context.rb:22:in `with_context'
app/controllers/application_controller.rb:451:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

[girish]

@rav I can't make out what the issue is. Do you think you can reach out to us at support@cloudron.io and give us SSH access so I can debug gitlab a bit to see what the issue is? Alternately, you have to ask on the GitLab forums on what to do for this.

[RaV]

@girish okay opened the support ticket from my cloudron

[girish]

Fix is documented in https://git.cloudron.io/cloudron/gitlab-app/-/issues/13