Somehow my Gitlab seems to be broken. When I try to access the Admin page for the runners I get a 500 error. While trying to fix it I figured out that my Gitlab does not start when I try to enable the recovery mode. Any way to fix this?

@rav Is there any error in the logs when you get a 500? If you disable recovery mode, does the app come back up? It is expected that the app won't start in recovery mode. You have to go the web terminal and then run /app/pkg/start.sh to start the app. I guess the page is /admin/runners and not /admin/runner ?

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Solved Gitlab /admin/runner error 500

GitLab

3212

R
RaV last edited by

Somehow my Gitlab seems to be broken. When I try to access the Admin page for the runners I get a 500 error. While trying to fix it I figured out that my Gitlab does not start when I try to enable the recovery mode. Any way to fix this?
1 Reply Last reply
Reply Quote 0
girish Staff @RaV last edited by

@rav Is there any error in the logs when you get a 500? If you disable recovery mode, does the app come back up? It is expected that the app won't start in recovery mode. You have to go the web terminal and then run /app/pkg/start.sh to start the app.

I guess the page is /admin/runners and not /admin/runner ?
R 1 Reply Last reply
Reply Quote 0
R
RaV @girish last edited by

@girish The error in the log seems to be not so usefull.

Jan 23 10:55:32 172.18.0.1 - - [23/Jan/2021:09:55:32 +0000] "GET /admin/runners HTTP/1.1" 500 2926 "https://git.aaa.bb/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15
Jan 23 10:55:35 172.18.0.1 - - [23/Jan/2021:09:55:35 +0000] "POST /api/v4/jobs/request HTTP/1.1" 500 39 "-" "gitlab-runner 13.7.0 (13-7-stable; go1.13.8; linux/amd64)"

In recovery mode in the webterminal I am unable to type or see anything this is what seems strange. Yes it is /admin/runners sorry.
1 Reply Last reply
Reply Quote 0
girish Staff @RaV last edited by

@rav GitLab logs are under /run/gitlab/log/. There are two files there application.log and production.log. Maybe you can load the page and then look for the backtrace in those files.
R 1 Reply Last reply
Reply Quote 0

RaV @girish

@girish found this in production.log

Started GET "/admin/runners" for x.x.x.x at 2021-01-26 10:41:57 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 88ms (ActiveRecord: 1.9ms | Elasticsearch: 0.0ms | Allocations: 48002)
  
ActionView::Template::Error ():
    36: 
    37:   .col-sm-6
    38:     .bs-callout
    39:       = render partial: 'ci/runner/how_to_setup_runner',
    40:                 locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
    41:                           type: 'shared',
    42:                           reset_token_url: reset_registration_token_admin_application_settings_path }
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:38:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'
app/models/application_setting_implementation.rb:334:in `runners_registration_token'
lib/gitlab/current_settings.rb:20:in `method_missing'
app/views/admin/runners/index.html.haml:39
app/controllers/application_controller.rb:133:in `render'
app/controllers/application_controller.rb:494:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:485:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:479:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:544:in `sentry_context'
app/controllers/application_controller.rb:472:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:464:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

girish Staff @RaV last edited by
@rav I found the exact same error in https://stackoverflow.com/questions/54216933/internal-server-error-500-while-accessing-gitlab-admin-runners .

As per the instruction in the stackoverflow post, can you try this in Cloudron's Web terminal?:
```
# su - git  
$ cd gitlab
$ bundle exec rails c -e production # (takes about 10 sec to bring up rails terminal)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
```
R 1 Reply Last reply
Reply Quote 0
R
RaV @girish last edited by

@girish said in Gitlab /admin/runner error 500:

ApplicationSetting.current.reset_runners_registration_token!

Tried this and the Rails console said => True.
The /admin/runners page is still not available. Any other logs I could check?
1 Reply Last reply
Reply Quote 0
girish Staff @RaV last edited by

@rav said in Gitlab /admin/runner error 500:

The /admin/runners page is still not available. Any other logs I could check?

So, same backtrace? Have you tried restarting the app just to be sure?
R 1 Reply Last reply
Reply Quote 0

RaV @girish

@girish looks the same to me yes. did a restart of the app but it did not help.

Started GET "/admin/runners" for a.b.c.d at 2021-01-30 10:47:26 +0000
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 124ms (ActiveRecord: 29.1ms | Elasticsearch: 0.0ms | Allocations: 96262)
  
ActionView::Template::Error ():
    16:   .table-section.section-10
    17:     .table-mobile-header{ role: 'rowheader' }= _('Runner token')
    18:     .table-mobile-content
    19:       = link_to runner.short_sha, admin_runner_path(runner)
    20: 
    21:   .table-section.section-20
    22:     .table-mobile-header{ role: 'rowheader' }= _('Description')
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable.rb:35:in `block in add_authentication_token_field'
app/models/ci/runner.rb:265:in `short_sha'
app/views/admin/runners/_runner.html.haml:19
app/views/admin/runners/index.html.haml:134
app/views/admin/runners/index.html.haml:133
app/controllers/application_controller.rb:125:in `render'
app/controllers/application_controller.rb:482:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:473:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:467:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:532:in `sentry_context'
app/controllers/application_controller.rb:460:in `block in set_current_context'
lib/gitlab/application_context.rb:56:in `block in use'
lib/gitlab/application_context.rb:56:in `use'
lib/gitlab/application_context.rb:22:in `with_context'
app/controllers/application_controller.rb:451:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

girish Staff @RaV last edited by

@rav I can't make out what the issue is. Do you think you can reach out to us at support@cloudron.io and give us SSH access so I can debug gitlab a bit to see what the issue is? Alternately, you have to ask on the GitLab forums on what to do for this.
R 1 Reply Last reply
Reply Quote 0
R
RaV @girish last edited by

@girish okay opened the support ticket from my cloudron
1 Reply Last reply
Reply Quote 0
girish Staff last edited by

Fix is documented in https://git.cloudron.io/cloudron/gitlab-app/-/issues/13
1 Reply Last reply
Reply Quote 0

1 / 1

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Gitlab /admin/runner error 500