Request failed with status code 500
-
It seems it running out of memory (terminated by SIGKILL in the logs). Have you tried giving it more memory? How low is the server on memory?
-
@girish Server has been upgraded to 120GB of memory recently and each Baserow app have been set from default 3GB to 4GB and some with bigger tables with even more (one baserow with 6 and another with 8GB )
I only managed to get them back up and running after indeed giving more memory, saving the location tab (they were loosing their associated app for some reason, with cloudron showing the view when the app is shutdown)
now all Baserow are back on 1.16.1 with updates turned off -
@rmdes thanks, I think we need to report it upstream at https://gitlab.com/baserow/baserow/-/issues (with the backtrace in https://paste.cloudron.io/okobusimap.yaml)
-
I've tried to reproduce it by doing the following:
- Manually install 1.16.1 (Baserow 1.22.1) using
cloudron install -l upgrade-test-2 --image cloudron/io.baserow.cloudronapp:20231221-163612-6274e3e65
- Wait for all the templates to install
- Upgrade to 1.16.2 (Baserow 1.22.2) using
cloudron update --app 30d540a1-9fb8-4d69-b572-5c481df18a7a --image cloudron/io.baserow.cloudronapp:20240115-111341-0207e58dd
But it all works as expected.
- Manually install 1.16.1 (Baserow 1.22.1) using
-
-
@rmdes we're analyzed your logs in more detail, and the only thing that's off, it the
Invalid field name(s) given in select_related: 'profile'
error. We're going to look into that. Meanwhile, it would also be great to receive which environment variables you've configured. -
I'm not sure if it's related, but just now I tried to login again, this time on the latest version, I have allowed the upgrade to enfold, once the container started I tried to login
and I'm getting this :
Jan 27 14:23:06179:M 27 Jan 2024 13:23:06.193 * Background saving terminated with success Jan 27 14:23:063133:C 27 Jan 2024 13:23:06.099 * DB saved on disk Jan 27 14:23:063133:C 27 Jan 2024 13:23:06.100 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB Jan 27 14:23:47172.18.0.1 - - [27/Jan/2024:13:23:47 +0000] "GET /dashboard HTTP/1.1" 400 158553 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" Jan 27 14:23:47ERROR 2024-01-27 13:23:47,137 django.request.log_response:241- Internal Server Error: /api/user/dashboard/ Jan 27 14:23:47File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:23:47File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:23:47File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:23:47File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 448, in __call__ Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 490, in thread_handler Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/exception.py", line 43, in inner Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 302, in __len__ Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 302, in __len__ Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 492, in get Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 492, in get Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__ Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__ Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1158, in get_related_selections Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1158, in get_related_selections Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 268, in get_select Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 268, in get_select Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 55, in setup_query Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 55, in setup_query Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 573, in as_sql Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 573, in as_sql Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 64, in pre_sql_setup Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 64, in pre_sql_setup Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 497, in dispatch Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:23:47File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:23:47File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for Jan 27 14:23:47Traceback (most recent call last): Jan 27 14:23:47auth_response = super().authenticate(request) Jan 27 14:23:47auth_response = super().authenticate(request) Jan 27 14:23:47django.core.exceptions.FieldError: Invalid field name(s) given in select_related: 'profile'. Choices are: (none) Jan 27 14:23:47django.core.exceptions.FieldError: Invalid field name(s) given in select_related: 'profile'. Choices are: (none) Jan 27 14:23:47extra_select, order_by, group_by = self.pre_sql_setup() Jan 27 14:23:47extra_select, order_by, group_by = self.pre_sql_setup() Jan 27 14:23:47num = len(clone) Jan 27 14:23:47num = len(clone) Jan 27 14:23:47raise FieldError( Jan 27 14:23:47raise FieldError( Jan 27 14:23:47raise exc Jan 27 14:23:47raise exc_info[1] Jan 27 14:23:47related_klass_infos = self.get_related_selections(select) Jan 27 14:23:47related_klass_infos = self.get_related_selections(select) Jan 27 14:23:47request.user Jan 27 14:23:47request.user Jan 27 14:23:47response = await get_response(request) Jan 27 14:23:47response = await wrapped_callback( Jan 27 14:23:47response = self.handle_exception(exc) Jan 27 14:23:47result = self.fn(*self.args, **self.kwargs) Jan 27 14:23:47results = compiler.execute_sql( Jan 27 14:23:47results = compiler.execute_sql( Jan 27 14:23:47ret = await asyncio.wait_for(future, timeout=None) Jan 27 14:23:47return await fut Jan 27 14:23:47return func(*args, **kwargs) Jan 27 14:23:47return self.dispatch(request, *args, **kwargs) Jan 27 14:23:47return self.get_user(validated_token), validated_token Jan 27 14:23:47return self.get_user(validated_token), validated_token Jan 27 14:23:47return view_func(*args, **kwargs) Jan 27 14:23:47self._authenticate() Jan 27 14:23:47self._authenticate() Jan 27 14:23:47self._fetch_all() Jan 27 14:23:47self._fetch_all() Jan 27 14:23:47self._result_cache = list(self._iterable_class(self)) Jan 27 14:23:47self._result_cache = list(self._iterable_class(self)) Jan 27 14:23:47self.initial(request, *args, **kwargs) Jan 27 14:23:47self.perform_authentication(request) Jan 27 14:23:47self.perform_authentication(request) Jan 27 14:23:47self.raise_uncaught_exception(exc) Jan 27 14:23:47self.select, self.klass_info, self.annotation_col_map = self.get_select() Jan 27 14:23:47self.select, self.klass_info, self.annotation_col_map = self.get_select() Jan 27 14:23:47self.setup_query() Jan 27 14:23:47self.setup_query() Jan 27 14:23:47sql, params = self.as_sql() Jan 27 14:23:47sql, params = self.as_sql() Jan 27 14:23:47user = self.user_model.objects.select_related("profile").get( Jan 27 14:23:47user = self.user_model.objects.select_related("profile").get( Jan 27 14:23:47user_auth_tuple = authenticator.authenticate(self) Jan 27 14:23:47user_auth_tuple = authenticator.authenticate(self) Jan 27 14:23:50172.18.0.1 - - [27/Jan/2024:13:23:50 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:24:00[2024-01-27 13:24:00,017: INFO/MainProcess] Scheduler: Sending due task baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email() (baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email) Jan 27 14:24:00[2024-01-27 13:24:00,023: INFO/MainProcess] Task baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email[faf13556-2ce3-4cd4-a7bf-4a2e0766b63e] received Jan 27 14:24:00[2024-01-27 13:24:00,027: INFO/ForkPoolWorker-14] Task baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email[faf13556-2ce3-4cd4-a7bf-4a2e0766b63e] succeeded in 0.002572456025518477s: None Jan 27 14:24:00[2024-01-27 13:24:00,028: INFO/MainProcess] Task baserow.core.notifications.tasks.singleton_send_instant_notifications_summary_by_email[54dfa545-32d6-45a1-9c24-503d6aaf8e86] received Jan 27 14:24:00[2024-01-27 13:24:00,061: INFO/ForkPoolWorker-14] Task baserow.core.notifications.tasks.singleton_send_instant_notifications_summary_by_email[54dfa545-32d6-45a1-9c24-503d6aaf8e86] succeeded in 0.00927996402606368s: None Jan 27 14:24:00172.18.0.1 - - [27/Jan/2024:13:24:00 +0000] "GET /_health HTTP/1.1" 200 156762 "-" "Mozilla (CloudronHealth)"
All I see is this on the frontend :
-
This is really odd, I tried to login again and this time, I managed to load the dashboard, but when I try to visit any part of the app I get this for /settings/ for example
ERROR Request failed with status code 500Jan 27 14:33:24172.18.0.1 - - [27/Jan/2024:13:33:24 +0000] "GET /dashboard HTTP/1.1" 500 931 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0" Jan 27 14:33:24ERROR Request failed with status code 500 Jan 27 14:33:24ERROR 2024-01-27 13:33:24,402 django.request.log_response:241- Internal Server Error: /api/settings/ Jan 27 14:33:24File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:33:24File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:33:24File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:33:24File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 448, in __call__ Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 490, in thread_handler Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/exception.py", line 43, in inner Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 302, in __len__ Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 492, in get Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__ Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1158, in get_related_selections Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 268, in get_select Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 55, in setup_query Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 573, in as_sql Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 64, in pre_sql_setup Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 497, in dispatch Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 497, in dispatch Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:33:24File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:33:24File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for Jan 27 14:33:24Traceback (most recent call last): Jan 27 14:33:24at IncomingMessage.emit (node:domain:489:12) Jan 27 14:33:24at IncomingMessage.emit (node:events:525:35) Jan 27 14:33:24at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:545:0) Jan 27 14:33:24at endReadableNT (node:internal/streams/readable:1359:12) Jan 27 14:33:24at process.processTicksAndRejections (node:internal/process/task_queues:82:21) Jan 27 14:33:24at settle (node_modules/axios/lib/core/settle.js:13:0) Jan 27 14:33:24auth_response = super().authenticate(request) Jan 27 14:33:24auth_response = super().authenticate(request) Jan 27 14:33:24django.core.exceptions.FieldError: Invalid field name(s) given in select_related: 'profile'. Choices are: (none) Jan 27 14:33:24extra_select, order_by, group_by = self.pre_sql_setup() Jan 27 14:33:24num = len(clone) Jan 27 14:33:24raise FieldError( Jan 27 14:33:24raise exc Jan 27 14:33:24raise exc Jan 27 14:33:24raise exc_info[1] Jan 27 14:33:24related_klass_infos = self.get_related_selections(select) Jan 27 14:33:24request.user Jan 27 14:33:24request.user Jan 27 14:33:24response = await get_response(request) Jan 27 14:33:24response = await wrapped_callback( Jan 27 14:33:24response = self.handle_exception(exc) Jan 27 14:33:24response = self.handle_exception(exc) Jan 27 14:33:24result = self.fn(*self.args, **self.kwargs) Jan 27 14:33:24results = compiler.execute_sql( Jan 27 14:33:24ret = await asyncio.wait_for(future, timeout=None) Jan 27 14:33:24return await fut Jan 27 14:33:24return func(*args, **kwargs) Jan 27 14:33:24return self.dispatch(request, *args, **kwargs) Jan 27 14:33:24return self.dispatch(request, *args, **kwargs) Jan 27 14:33:24return self.get_user(validated_token), validated_token Jan 27 14:33:24return self.get_user(validated_token), validated_token Jan 27 14:33:24return view_func(*args, **kwargs) Jan 27 14:33:24self._authenticate() Jan 27 14:33:24self._authenticate() Jan 27 14:33:24self._fetch_all() Jan 27 14:33:24self._result_cache = list(self._iterable_class(self)) Jan 27 14:33:24self.initial(request, *args, **kwargs) Jan 27 14:33:24self.initial(request, *args, **kwargs) Jan 27 14:33:24self.perform_authentication(request) Jan 27 14:33:24self.perform_authentication(request) Jan 27 14:33:24self.raise_uncaught_exception(exc) Jan 27 14:33:24self.raise_uncaught_exception(exc) Jan 27 14:33:24self.select, self.klass_info, self.annotation_col_map = self.get_select() Jan 27 14:33:24self.setup_query() Jan 27 14:33:24sql, params = self.as_sql() Jan 27 14:33:24user = self.user_model.objects.select_related("profile").get( Jan 27 14:33:24user = self.user_model.objects.select_related("profile").get( Jan 27 14:33:24user_auth_tuple = authenticator.authenticate(self) Jan 27 14:33:24user_auth_tuple = authenticator.authenticate(self) Jan 27 14:33:30172.18.0.1 - - [27/Jan/2024:13:33:30 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:33:40172.18.0.1 - - [27/Jan/2024:13:33:40 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:33:50172.18.0.1 - - [27/Jan/2024:13:33:50 +0000] "GET /_health HTTP/1.1" 200 156762 "-" "Mozilla (CloudronHealth)"
-
More logs :
Apparently something is wrong with one of my tables ?
it's odd because this table has been operating since 2 years and is being fed by a N8N process just fine until now
this is the error I see now when I try to access any of my previously working tablesJan 27 14:34:00[2024-01-27 13:34:00,018: INFO/MainProcess] Scheduler: Sending due task baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email() (baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email) Jan 27 14:34:00[2024-01-27 13:34:00,027: INFO/MainProcess] Task baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email[fb70243d-a783-4e5d-870c-50b204885463] received Jan 27 14:34:00[2024-01-27 13:34:00,033: INFO/ForkPoolWorker-14] Task baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email[fb70243d-a783-4e5d-870c-50b204885463] succeeded in 0.0042537120170891285s: None Jan 27 14:34:00[2024-01-27 13:34:00,039: INFO/MainProcess] Task baserow.core.notifications.tasks.singleton_send_instant_notifications_summary_by_email[2076bd97-f3b9-4a74-a552-c75ab3d5425e] received Jan 27 14:34:00[2024-01-27 13:34:00,050: INFO/ForkPoolWorker-14] Task baserow.core.notifications.tasks.singleton_send_instant_notifications_summary_by_email[2076bd97-f3b9-4a74-a552-c75ab3d5425e] succeeded in 0.008335583959706128s: None Jan 27 14:34:00172.18.0.1 - - [27/Jan/2024:13:34:00 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:34:10172.18.0.1 - - [27/Jan/2024:13:34:10 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:34:1760|2024-01-27 13:34:17.909|INFO|baserow.core.action.signals:log_action_receiver:28 - do: workspace= action_type=sign_in_user user=1 Jan 27 14:34:17172.18.0.1 - - [27/Jan/2024:13:34:17 +0000] "POST /api/user/token-auth/ HTTP/1.1" 200 1354 "-" "axios/1.6.2" Jan 27 14:34:18172.18.0.1 - - [27/Jan/2024:13:34:18 +0000] "GET /api/database/fields/table/135/ HTTP/1.1" 200 2913 "-" "axios/1.6.2" Jan 27 14:34:18ERROR 2024-01-27 13:34:18,594 django.request.log_response:241- Internal Server Error: /api/database/rows/table/135/ Jan 27 14:34:18Traceback (most recent call last): Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 27 14:34:18raise exc_info[1] Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/exception.py", line 43, in inner Jan 27 14:34:18response = await get_response(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async Jan 27 14:34:18response = await wrapped_callback( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 448, in __call__ Jan 27 14:34:18ret = await asyncio.wait_for(future, timeout=None) Jan 27 14:34:18File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for Jan 27 14:34:18return await fut Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run Jan 27 14:34:18result = self.fn(*self.args, **self.kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 490, in thread_handler Jan 27 14:34:18return func(*args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view Jan 27 14:34:18return view_func(*args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view Jan 27 14:34:18return self.dispatch(request, *args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch Jan 27 14:34:18response = self.handle_exception(exc) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 27 14:34:18self.raise_uncaught_exception(exc) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 27 14:34:18raise exc Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 497, in dispatch Jan 27 14:34:18self.initial(request, *args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:34:18self.perform_authentication(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:34:18request.user Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:34:18self._authenticate() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:34:18user_auth_tuple = authenticator.authenticate(self) Jan 27 14:34:18File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:34:18auth_response = super().authenticate(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:34:18return self.get_user(validated_token), validated_token Jan 27 14:34:18File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:34:18user = self.user_model.objects.select_related("profile").get( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 492, in get Jan 27 14:34:18num = len(clone) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 302, in __len__ Jan 27 14:34:18self._fetch_all() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all Jan 27 14:34:18self._result_cache = list(self._iterable_class(self)) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__ Jan 27 14:34:18results = compiler.execute_sql( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql Jan 27 14:34:18sql, params = self.as_sql() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 573, in as_sql Jan 27 14:34:18extra_select, order_by, group_by = self.pre_sql_setup() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 64, in pre_sql_setup Jan 27 14:34:18self.setup_query() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 55, in setup_query Jan 27 14:34:18self.select, self.klass_info, self.annotation_col_map = self.get_select() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 268, in get_select Jan 27 14:34:18related_klass_infos = self.get_related_selections(select) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1158, in get_related_selections Jan 27 14:34:18raise FieldError( Jan 27 14:34:18django.core.exceptions.FieldError: Invalid field name(s) given in select_related: 'profile'. Choices are: (none) Jan 27 14:34:18ERROR 2024-01-27 13:34:18,594 django.request.log_response:241- Internal Server Error: /api/database/rows/table/135/ Jan 27 14:34:18Traceback (most recent call last): Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 27 14:34:18raise exc_info[1] Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/exception.py", line 43, in inner Jan 27 14:34:18response = await get_response(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async Jan 27 14:34:18response = await wrapped_callback( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 448, in __call__ Jan 27 14:34:18ret = await asyncio.wait_for(future, timeout=None) Jan 27 14:34:18File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for Jan 27 14:34:18return await fut Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run Jan 27 14:34:18result = self.fn(*self.args, **self.kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 490, in thread_handler Jan 27 14:34:18return func(*args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view Jan 27 14:34:18return view_func(*args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view Jan 27 14:34:18return self.dispatch(request, *args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch Jan 27 14:34:18response = self.handle_exception(exc) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 27 14:34:18self.raise_uncaught_exception(exc) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 27 14:34:18raise exc Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 497, in dispatch Jan 27 14:34:18self.initial(request, *args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:34:18self.perform_authentication(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:34:18request.user Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:34:18self._authenticate() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:34:18user_auth_tuple = authenticator.authenticate(self) Jan 27 14:34:18File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:34:18auth_response = super().authenticate(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:34:18return self.get_user(validated_token), validated_token Jan 27 14:34:18File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:34:18user = self.user_model.objects.select_related("profile").get( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 492, in get Jan 27 14:34:18num = len(clone) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 302, in __len__ Jan 27 14:34:18self._fetch_all() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all Jan 27 14:34:18self._result_cache = list(self._iterable_class(self)) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__ Jan 27 14:34:18results = compiler.execute_sql( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql Jan 27 14:34:18sql, params = self.as_sql() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 573, in as_sql Jan 27 14:34:18extra_select, order_by, group_by = self.pre_sql_setup() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 64, in pre_sql_setup Jan 27 14:34:18self.setup_query() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 55, in setup_query Jan 27 14:34:18self.select, self.klass_info, self.annotation_col_map = self.get_select() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 268, in get_select Jan 27 14:34:18related_klass_infos = self.get_related_selections(select) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1158, in get_related_selections Jan 27 14:34:18raise FieldError( Jan 27 14:34:18django.core.exceptions.FieldError: Invalid field name(s) given in select_related: 'profile'. Choices are: (none) Jan 27 14:34:18ERROR 2024-01-27 13:34:18,594 django.request.log_response:241- Internal Server Error: /api/database/rows/table/135/ Jan 27 14:34:18Traceback (most recent call last): Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 486, in thread_handler Jan 27 14:34:18raise exc_info[1] Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/exception.py", line 43, in inner Jan 27 14:34:18response = await get_response(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async Jan 27 14:34:18response = await wrapped_callback( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 448, in __call__ Jan 27 14:34:18ret = await asyncio.wait_for(future, timeout=None) Jan 27 14:34:18File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for Jan 27 14:34:18return await fut Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run Jan 27 14:34:18result = self.fn(*self.args, **self.kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/asgiref/sync.py", line 490, in thread_handler Jan 27 14:34:18return func(*args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view Jan 27 14:34:18return view_func(*args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view Jan 27 14:34:18return self.dispatch(request, *args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch Jan 27 14:34:18response = self.handle_exception(exc) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception Jan 27 14:34:18self.raise_uncaught_exception(exc) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception Jan 27 14:34:18raise exc Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 497, in dispatch Jan 27 14:34:18self.initial(request, *args, **kwargs) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 414, in initial Jan 27 14:34:18self.perform_authentication(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/views.py", line 324, in perform_authentication Jan 27 14:34:18request.user Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 227, in user Jan 27 14:34:18self._authenticate() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework/request.py", line 380, in _authenticate Jan 27 14:34:18user_auth_tuple = authenticator.authenticate(self) Jan 27 14:34:18File "/app/code/backend/src/baserow/api/authentication.py", line 48, in authenticate Jan 27 14:34:18auth_response = super().authenticate(request) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/rest_framework_simplejwt/authentication.py", line 40, in authenticate Jan 27 14:34:18return self.get_user(validated_token), validated_token Jan 27 14:34:18File "/app/code/backend/src/baserow/api/authentication.py", line 28, in get_user Jan 27 14:34:18user = self.user_model.objects.select_related("profile").get( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 492, in get Jan 27 14:34:18num = len(clone) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 302, in __len__ Jan 27 14:34:18self._fetch_all() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 1507, in _fetch_all Jan 27 14:34:18self._result_cache = list(self._iterable_class(self)) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/query.py", line 57, in __iter__ Jan 27 14:34:18results = compiler.execute_sql( Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql Jan 27 14:34:18sql, params = self.as_sql() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 573, in as_sql Jan 27 14:34:18extra_select, order_by, group_by = self.pre_sql_setup() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 64, in pre_sql_setup Jan 27 14:34:18self.setup_query() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 55, in setup_query Jan 27 14:34:18self.select, self.klass_info, self.annotation_col_map = self.get_select() Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 268, in get_select Jan 27 14:34:18related_klass_infos = self.get_related_selections(select) Jan 27 14:34:18File "/app/code/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1158, in get_related_selections Jan 27 14:34:18raise FieldError( Jan 27 14:34:18django.core.exceptions.FieldError: Invalid field name(s) given in select_related: 'profile'. Choices are: (none) Jan 27 14:34:18172.18.0.1 - - [27/Jan/2024:13:34:18 +0000] "POST /api/database/rows/table/135/ HTTP/1.1" 500 145 "-" "axios/1.6.2" Jan 27 14:34:20172.18.0.1 - - [27/Jan/2024:13:34:20 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:34:30172.18.0.1 - - [27/Jan/2024:13:34:30 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:34:40172.18.0.1 - - [27/Jan/2024:13:34:40 +0000] "GET /_health HTTP/1.1" 200 156761 "-" "Mozilla (CloudronHealth)" Jan 27 14:34:4459|2024-01-27 13:34:44.705|INFO|baserow.core.action.signals:log_action_receiver:28 - do: workspace= action_type=sign_in_user user=1 Jan 27 14:34:44172.18.0.1 - - [27/Jan/2024:13:34:44 +0000] "POST /api/user/token-auth/ HTTP/1.1" 200 1354 "-" "axios/1.6.2" Jan 27 14:34:45172.18.0.1 - - [27/Jan/2024:13:34:45 +0000] "GET /api/database/fields/table/135/ HTTP/1.1" 200 2913 "-" "axios/1.6.2"
-
@bram said in Request failed with status code 500:
@rmdes we're analyzed your logs in more detail, and the only thing that's off, it the
Invalid field name(s) given in select_related: 'profile'
error. We're going to look into that. Meanwhile, it would also be great to receive which environment variables you've configured.looks like this is still the same issue, the logs seem to bel the same.
-
Hey, Davide from Baserow here. Here the explanation of the bug, the work in progress to fix it for the long termand an immediate solution for now (taken from https://gitlab.com/baserow/baserow/-/issues/2290#note_1760709216) :
So, it turned out to be a pretty challenging bug. The main problem is that atm ASGI is not fully compatible with how we manage our dynamic models in Django, because every time we clear Django's global apps cache and, if another request tries to access some related field on a model at the same time, it might experience some weird error like the ones reported in this thread.
While a proper fix is in development here: !2105 I'd suggest changing the baserow-app in Cloudron applying the patch attached below. Basically, it uses WSGI to handle the API endpoints, leaving ASGI to handle WebSockets. In this way, every API request will have its process (and so its own Django apps registry), so that concurrent requests will have separate Django instances.Patch:
diff --git a/CloudronManifest.json b/CloudronManifest.json index 1da2e64..c563208 100644 --- a/CloudronManifest.json +++ b/CloudronManifest.json @@ -1,6 +1,6 @@ { "id": "io.baserow.cloudronapp", - "version": "1.17.0", + "version": "1.17.1", "upstreamVersion": "1.22.3", "title": "Baserow", "author": "Bram Wiepjes", diff --git a/nginx.conf b/nginx.conf index f080702..5ddf6ce 100644 --- a/nginx.conf +++ b/nginx.conf @@ -32,7 +32,7 @@ server { proxy_read_timeout 86400; } - location ~ ^/(api|ws)/ { + location ~ ^/(ws)/ { proxy_pass http://127.0.0.1:8000; proxy_pass_request_headers on; proxy_http_version 1.1; @@ -42,6 +42,17 @@ server { proxy_set_header Connection "upgrade"; proxy_read_timeout 86400; } + + location ~ ^/(api)/ { + proxy_pass http://127.0.0.1:8001; + proxy_pass_request_headers on; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_read_timeout 86400; + } location /media/ { root /app/data; diff --git a/supervisor.conf b/supervisor.conf index 70d64c4..6fa5e8e 100644 --- a/supervisor.conf +++ b/supervisor.conf @@ -3,7 +3,7 @@ nodaemon = true logfile=/dev/null logfile_maxbytes=0 -[program:gunicorn] +[program:backend-asgi] user=cloudron directory=/app/code command=/app/code/env/bin/gunicorn -w 3 -b 127.0.0.1:8000 -b [::1]:8000 -k uvicorn.workers.UvicornWorker baserow.config.asgi:application --log-level=debug @@ -14,6 +14,17 @@ stderr_logfile_maxbytes=0 autostart=true autorestart=true +[program:backend-wsgi] +user=cloudron +directory=/app/code +command=/app/code/env/bin/gunicorn -w 3 -b 127.0.0.1:8001 -b [::1]:8001 baserow.config.wsgi:application --log-level=debug +stdout_logfile=/dev/stdout +stdout_logfile_maxbytes=0 +stderr_logfile=/dev/stdout +stderr_logfile_maxbytes=0 +autostart=true +autorestart=true + [program:worker] user=cloudron directory=/app/code
-
-
-
-