SOLVED HELP!!! Latest App update broke Taiga and restore doesn't help. HELP!!!

[imc67]

Just updated Taiga to the latest app package, after that Django gives a 500 and front is gives empty 'white page'. Restore app to latest backup (23 minutes before) results in empty TAIGA.

This is a disaster as there is very important information in it in 2 projects where we're working on!

Please help!!.

Here catched logs after the update:

Apr 13 08:28:49 box:tasks 6386: {"percent":100,"result":null,"error":null}
Apr 13 08:31:46 self._fetch_all()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
Apr 13 08:31:46 self._result_cache = list(self._iterable_class(self))
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/query.py", line 55, in __iter__
Apr 13 08:31:46 results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
Apr 13 08:31:46 cursor = self.connection.cursor()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
Apr 13 08:31:46 return self._cursor()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
Apr 13 08:31:46 self.ensure_connection()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
Apr 13 08:31:46 self.connect()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
Apr 13 08:31:46 raise dj_exc_value.with_traceback(traceback) from exc_value
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
Apr 13 08:31:46 self.connect()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
Apr 13 08:31:46 self.connection = self.get_new_connection(conn_params)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
Apr 13 08:31:46 connection = Database.connect(**conn_params)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
Apr 13 08:31:46 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Apr 13 08:31:46 django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
Apr 13 08:31:46
Apr 13 08:31:46 172.18.0.1 - - [13/Apr/2021:06:31:46 +0000] "GET /api/v1/projects?member=6&order_by=user_order&slight=true HTTP/1.1" 500 27 "https://******" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:87.0) Gecko/20100101 Firefox/87.0"
Apr 13 08:31:46 ERROR:2021-04-13 06:31:46,741: Internal Server Error: /api/v1/projects/by_slug
Apr 13 08:31:46 Traceback (most recent call last):
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
Apr 13 08:31:46 self.connect()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
Apr 13 08:31:46 self.connection = self.get_new_connection(conn_params)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
Apr 13 08:31:46 connection = Database.connect(**conn_params)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
Apr 13 08:31:46 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Apr 13 08:31:46 psycopg2.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
Apr 13 08:31:46
Apr 13 08:31:46
Apr 13 08:31:46 The above exception was the direct cause of the following exception:
Apr 13 08:31:46
Apr 13 08:31:46 Traceback (most recent call last):
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
Apr 13 08:31:46 response = get_response(request)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
Apr 13 08:31:46 response = self.process_exception_by_middleware(e, request)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
Apr 13 08:31:46 response = wrapped_callback(request, *callback_args, **callback_kwargs)
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/viewsets.py", line 104, in view
Apr 13 08:31:46 return self.dispatch(request, *args, **kwargs)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Apr 13 08:31:46 return view_func(*args, **kwargs)
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/views.py", line 458, in dispatch
Apr 13 08:31:46 response = self.handle_exception(exc)
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/views.py", line 447, in dispatch
Apr 13 08:31:46 self.initial(request, *args, **kwargs)
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/views.py", line 377, in initial
Apr 13 08:31:46 self.perform_authentication(request)
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/views.py", line 325, in perform_authentication
Apr 13 08:31:46 request.user
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/request.py", line 262, in user
Apr 13 08:31:46 self._authenticate()
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/base/api/request.py", line 428, in _authenticate
Apr 13 08:31:46 user_auth_tuple = authenticator.authenticate(self)
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/auth/backends.py", line 86, in authenticate
Apr 13 08:31:46 user = get_user_for_token(token, "authentication",
Apr 13 08:31:46 File "/app/code/taiga-back/taiga/auth/tokens.py", line 52, in get_user_for_token
Apr 13 08:31:46 user = model_cls.objects.get(pk=data["user_%s_id" % (scope)])
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
Apr 13 08:31:46 return getattr(self.get_queryset(), name)(*args, **kwargs)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/query.py", line 402, in get
Apr 13 08:31:46 num = len(clone)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/query.py", line 256, in __len__
Apr 13 08:31:46 self._fetch_all()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
Apr 13 08:31:46 self._result_cache = list(self._iterable_class(self))
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/query.py", line 55, in __iter__
Apr 13 08:31:46 results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1140, in execute_sql
Apr 13 08:31:46 cursor = self.connection.cursor()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
Apr 13 08:31:46 return self._cursor()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
Apr 13 08:31:46 self.ensure_connection()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
Apr 13 08:31:46 self.connect()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
Apr 13 08:31:46 raise dj_exc_value.with_traceback(traceback) from exc_value
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
Apr 13 08:31:46 self.connect()
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
Apr 13 08:31:46 self.connection = self.get_new_connection(conn_params)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
Apr 13 08:31:46 connection = Database.connect(**conn_params)
Apr 13 08:31:46 File "/app/code/taiga/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
Apr 13 08:31:46 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Apr 13 08:31:46 django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
Apr 13 08:31:46 

[imc67]

@imc67 At the same time NextCloud also crashed, while I wasn't even touching it. Restarting MYSQL (although there was no issue) didn't help.

Rebooting server solved the issues ....

[imc67]

@imc67 eveything is working fine now, I'm really wondering what happened because "just updating" causing this trouble where reboot was needed is not "comfortable" on a production server

[nebulon]

The logs do indicate a database issue and restarting that might have fixed it as well. I guess you restarted mysql as you mentioned, while both apps use postgresql.
Actually the line of:

FATAL: remaining connection slots are reserved for non-replication superuser connections

gives a good indication here. I have seen that recently, that if apps hog onto the postgres connections, it eventually may exhaust the connection pool, then this message is shown. A postgresql service restart resolves this, as it frees up those connection slots.