Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0

[girish]

@rompo said in Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0:

Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server

Thanks, the above is the root cause. The app is not able to CURL itself for some reason.

• Are you behind Cloudflare?
• Open a web terminal (of any app) and then curl https://my.myurl.cloud/openid/.well-known/openid-configuration . We need to find out why this is not working.

[rompo]

Mar 06 19:32:45The above exception was the direct cause of the following exception:
Mar 06 19:32:45The above exception was the direct cause of the following exception:
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45await cb(*args, **kwargs)
Mar 06 19:32:45await cb(*args, **kwargs)
Mar 06 19:32:45await oidc.load_metadata()
Mar 06 19:32:45await oidc.load_metadata()
Mar 06 19:32:45await p.load_metadata()
Mar 06 19:32:45await p.load_metadata()
Mar 06 19:32:45body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:32:45body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:32:45current.result = callback( # type: ignore[misc]
Mar 06 19:32:45current.result = callback( # type: ignore[misc]
Mar 06 19:32:45metadata_response = await self._http_client.get_json(url)
Mar 06 19:32:45metadata_response = await self._http_client.get_json(url)
Mar 06 19:32:45raise Exception(
Mar 06 19:32:45raise Exception(
Mar 06 19:32:45raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:32:45raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:32:45raise self.value.with_traceback(self.tb)
Mar 06 19:32:45raise self.value.with_traceback(self.tb)
Mar 06 19:32:45response = await make_deferred_yieldable(request_deferred)
Mar 06 19:32:45response = await make_deferred_yieldable(request_deferred)
Mar 06 19:32:45response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:32:45response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:32:45result = context.run(
Mar 06 19:32:45result = context.run(
Mar 06 19:32:45return await f()
Mar 06 19:32:45return await f()
Mar 06 19:32:45return await self._cachedcall.get()
Mar 06 19:32:45return await self._cachedcall.get()
Mar 06 19:32:45return await self._provider_metadata.get()
Mar 06 19:32:45return await self._provider_metadata.get()
Mar 06 19:32:45return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:32:45return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:32:45self._result.raiseException()
Mar 06 19:32:45self._result.raiseException()
Mar 06 19:32:45synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:32:45synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:32:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 19:33:03=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:13=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:23=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:43=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:46==> Ensure we log to console
Mar 06 19:33:46==> Configuring synapse
Mar 06 19:33:46==> Configuring OIDC auth
Mar 06 19:33:46==> Fixing permissions
Mar 06 19:33:46==> Starting synapse
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 355 - WARNING - main - ***** STARTING SERVER *****
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 356 - WARNING - main - Server /app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py version 1.102.0
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 365 - INFO - main - Server hostname: myurl.cloud
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 366 - INFO - main - Instance name: master
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 367 - INFO - main - Twisted reactor: EPollReactor
Mar 06 19:33:482024-03-06 18:33:48,048 - synapse.app.homeserver - 356 - INFO - main - Setting up server
Mar 06 19:33:482024-03-06 18:33:48,049 - synapse.server - 345 - INFO - main - Setting up.
Mar 06 19:33:482024-03-06 18:33:48,075 - synapse.storage.databases - 73 - INFO - main - [database config 'master']: Checking database server
Mar 06 19:33:482024-03-06 18:33:48,076 - synapse.storage.databases - 76 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state']
Mar 06 19:33:482024-03-06 18:33:48,076 - synapse.storage.prepare_database - 135 - INFO - main - ['main', 'state']: Checking existing schema version
Mar 06 19:33:482024-03-06 18:33:48,078 - synapse.storage.prepare_database - 139 - INFO - main - ['main', 'state']: Existing schema is 84 (+3 deltas)
Mar 06 19:33:482024-03-06 18:33:48,078 - synapse.storage.databases.main - 388 - INFO - main - Checking database for consistency with configuration...
Mar 06 19:33:482024-03-06 18:33:48,079 - synapse.storage.prepare_database - 433 - INFO - main - Applying schema deltas for v84
Mar 06 19:33:482024-03-06 18:33:48,080 - synapse.storage.prepare_database - 568 - INFO - main - Schema now up to date
Mar 06 19:33:482024-03-06 18:33:48,081 - synapse.storage.databases - 91 - INFO - main - [database config 'master']: Starting 'main' database
Mar 06 19:33:482024-03-06 18:33:48,093 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for pushers(id): 1
Mar 06 19:33:482024-03-06 18:33:48,094 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for deleted_pushers(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,094 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_stream(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for e2e_cross_signing_keys(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.databases.main.event_push_actions - 1167 - INFO - main - Searching for stream ordering 1 month ago
Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.databases.main.event_push_actions - 1171 - INFO - main - Found stream ordering 1 month ago: it's 0
Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.databases.main.event_push_actions - 1174 - INFO - main - Searching for stream ordering 1 day ago
Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.databases.main.event_push_actions - 1178 - INFO - main - Found stream ordering 1 day ago: it's 0
Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules(id): 1
Mar 06 19:33:482024-03-06 18:33:48,099 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_enable(id): 1
Mar 06 19:33:482024-03-06 18:33:48,100 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for access_tokens(id): 1
Mar 06 19:33:482024-03-06 18:33:48,101 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for refresh_tokens(id): 1
Mar 06 19:33:482024-03-06 18:33:48,102 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for event_reports(id): 1
Mar 06 19:33:482024-03-06 18:33:48,102 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_stream(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for user_signature_stream(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_outbound_pokes(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_in_room(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,104 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_remote_pending(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,104 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,105 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database
Mar 06 19:33:482024-03-06 18:33:48,106 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared
Mar 06 19:33:482024-03-06 18:33:48,106 - synapse.server - 348 - INFO - main - Finished setting up.
Mar 06 19:33:482024-03-06 18:33:48,124 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True
Mar 06 19:33:482024-03-06 18:33:48,124 - synapse.push.pusher - 54 - INFO - main - defined email pusher type
Mar 06 19:33:482024-03-06 18:33:48,125 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update'
Mar 06 19:33:482024-03-06 18:33:48,125 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile'
Mar 06 19:33:482024-03-06 18:33:48,126 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence'
Mar 06 19:33:482024-03-06 18:33:48,127 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing'
Mar 06 19:33:482024-03-06 18:33:48,127 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory'
Mar 06 19:33:482024-03-06 18:33:48,128 - synapse.app.homeserver - 179 - INFO - sentinel - Running
Mar 06 19:33:482024-03-06 18:33:48,129 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576
Mar 06 19:33:482024-03-06 18:33:48,133 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter
Mar 06 19:33:482024-03-06 18:33:48,158 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Mar 06 19:33:482024-03-06 18:33:48,158 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 19:33:48await p.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 19:33:48return await self._provider_metadata.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 19:33:48return await self._cachedcall.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 19:33:48self._result.raiseException()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 19:33:48raise self.value.with_traceback(self.tb)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 19:33:48result = context.run(
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 19:33:48return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 19:33:48return await f()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 19:33:48metadata_response = await self._http_client.get_json(url)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 19:33:48body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 19:33:48response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 19:33:48response = await make_deferred_yieldable(request_deferred)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 19:33:48current.result = callback( # type: ignore[misc]
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 19:33:48raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:33:48synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:33:48
Mar 06 19:33:48The above exception was the direct cause of the following exception:
Mar 06 19:33:48
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 19:33:48await cb(*args, **kwargs)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 19:33:48await oidc.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 19:33:48raise Exception(
Mar 06 19:33:48Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 19:33:48Error during startup:
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 19:33:48await p.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 19:33:48return await self._provider_metadata.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 19:33:48return await self._cachedcall.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 19:33:48self._result.raiseException()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 19:33:48raise self.value.with_traceback(self.tb)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 19:33:48result = context.run(
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 19:33:48return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 19:33:48return await f()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 19:33:48metadata_response = await self._http_client.get_json(url)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 19:33:48body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 19:33:48response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 19:33:48response = await make_deferred_yieldable(request_deferred)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 19:33:48current.result = callback( # type: ignore[misc]
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 19:33:48raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:33:48synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:33:48
Mar 06 19:33:48The above exception was the direct cause of the following exception:
Mar 06 19:33:48
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 19:33:48await cb(*args, **kwargs)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 19:33:48await oidc.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 19:33:48raise Exception(
Mar 06 19:33:48Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 19:33:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 19:34:03=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:13=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:23=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:43=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:48==> Ensure we log to console
Mar 06 19:34:48==> Configuring synapse
Mar 06 19:34:48==> Configuring OIDC auth
Mar 06 19:34:48==> Fixing permissions
Mar 06 19:34:48==> Starting synapse

[rompo]

@girish

No, I'm not behind Cloudflaire.

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[girish]

@rompo you to check why that curl command does not work on your system. If the certificate is valid then it's a DNS related issue. Please check what the DNS of my.myurl.cloud is resolving to.

[rompo]

This post is deleted!

[rompo]

@girish

The DNS resolves correctly from my point of view.
my.myurl.cloud is also accessible from the Internet as well as from the internal network and I can log in to it.

Unreachable ports cannot be a problem either.

However, if I want to access the web interface via the internal IP, I get the following message: You are seeing this page because the DNS record of 172.27.212.20 is set to this server's IP but Cloudron has no app configured for this domain.

[girish]

@rompo you cannot access Cloudron pages by IP address, it has to be by domain name entirely.

Can you try to resolve host my.myurl.cloud from the web terminal? Does it give internal IP or external IP ? If it is the external IP, does your network support hairpinning - the ability to access internal Cloudron when using external IP ? See https://docs.cloudron.io/troubleshooting/#hairpin-nat

[rompo]

@girish
I receive the public IP as output.

With pfSense, there is a "host override" for all Cloudron URLs, which actually resolves correctly (internal 172 addresses within the network).

However, I have now activated NAT Reflection with everything that goes with it as a test, which does not result in any change for Cloudron. Within the 172 network, however, publicIP is now resolved and routed correctly.

However, it is not entirely clear to me why Cloudron outputs the public IP in the original configuration with the "Host Override" entries, because the DNS server address is actually assigned via DHCP.
How can I check this? I can't get to the required interface with ifconfig.

[girish]

@rompo Looks like we get closer. Cloudron has it's own internal recursive DNS resolver. This means that it might be bypassing pfSense DNS. Please see https://docs.cloudron.io/networking/#internal-dns-server to make Cloudron use the internal DNS. This way it will resolve DNS to internal network values .

[rompo]

@girish
Thanks for the tip, I have entered pfSense as DNS, now it works again.

At the end of last week I set up a test server to play with which is connected to a different firewall. I had exactly the same problem with this fresh installation.

Another problem with the new login procedure is that mobile devices with the policy of not allowing cookies can no longer log in to the Element app.
An alternative solution must now be found, possibly deactivating openID if this is possible, but that is another topic.

Many thanks for your help and have a good week!

For all those who need a short summary:

Log in via SSH
Create the file "custom.conf":

sudo touch /etc/unbound/unbound.conf.d/custom.conf

Edit file:

sudo nano /etc/unbound/unbound.conf.d/custom.conf

Insert content into the file:

# this disables DNSSEC
server:
  val-permissive-mode: yes

# forward all queries to the internal DNS
forward-zone:
  name: "."
  forward-addr: 172.xxx.xxx.xxx

Restart:

sudo systemctl restart unbound

Check:

sudo systemctl status unbound