Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0

rompo

Hello everyone

As the title already describes, Matrix can no longer be started completely since an automatic update was carried out.

A complete restart of everything did not help either.

In the logs I have the feeling that it could fail in this part. In any case, it starts to reload immediately after the "Error: connect EHOSTUNREACH 172.18.19.117:8008". What kind of IP this is is also a mystery to me.

Does anyone have any ideas?

Dec 05 23:06:24Exception: Error while initialising OIDC provider 'oidc-cloudron'
Dec 05 23:06:37=> Healtheck error: Error: Timeout of 7000ms exceeded
Dec 05 23:06:47=> Healtheck error: Error: Timeout of 7000ms exceeded
Dec 05 23:06:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Dec 05 23:07:00=> Healtheck error: Error: connect EHOSTUNREACH 172.18.19.117:8008

girish

@rompo said in Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0:

Error while initialising OIDC provider 'oidc-cloudron

It seems this is the error. Do you have any logs before this error message?

girish

So , per https://github.com/matrix-org/synapse/issues/9662 , it seems there might be a cert issue. Do you have valid certs for the Cloudron dashboard ?

rompo

@girish
Thanks for your feedback and sorry for the late feedback, I didn't have the capacity to deal with the problem.

Yes, the certificates are up to date and we don't have a problem with the certificates in the dashboard or other apps.

I'll add the log of an entire cycle below.

Mar 06 18:42:38
Mar 06 18:42:38
Mar 06 18:42:38
Mar 06 18:42:38
Mar 06 18:42:382024-03-06 17:42:38,090 - synapse.server - 348 - INFO - main - Finished setting up.
Mar 06 18:42:382024-03-06 17:42:38,090 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared
Mar 06 18:42:382024-03-06 17:42:38,109 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True
Mar 06 18:42:382024-03-06 17:42:38,110 - synapse.push.pusher - 54 - INFO - main - defined email pusher type
Mar 06 18:42:382024-03-06 17:42:38,111 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update'
Mar 06 18:42:382024-03-06 17:42:38,111 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile'
Mar 06 18:42:382024-03-06 17:42:38,112 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence'
Mar 06 18:42:382024-03-06 17:42:38,112 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing'
Mar 06 18:42:382024-03-06 17:42:38,113 - synapse.app.homeserver - 179 - INFO - sentinel - Running
Mar 06 18:42:382024-03-06 17:42:38,113 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory'
Mar 06 18:42:382024-03-06 17:42:38,115 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576
Mar 06 18:42:382024-03-06 17:42:38,118 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter
Mar 06 18:42:382024-03-06 17:42:38,191 - synapse.handlers.deactivate_account - 250 - INFO - user_parter_loop-0 - User parter finished: stopping
Mar 06 18:42:382024-03-06 17:42:38,317 - synapse.metrics._gc - 127 - INFO - sentinel - Collecting gc 1
Mar 06 18:42:382024-03-06 17:42:38,325 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Mar 06 18:42:382024-03-06 17:42:38,326 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup
Mar 06 18:42:38Error during startup:
Mar 06 18:42:38Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 18:42:38Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 18:42:38The above exception was the direct cause of the following exception:
Mar 06 18:42:38The above exception was the direct cause of the following exception:
Mar 06 18:42:38Traceback (most recent call last):
Mar 06 18:42:38Traceback (most recent call last):
Mar 06 18:42:38Traceback (most recent call last):
Mar 06 18:42:38Traceback (most recent call last):
Mar 06 18:42:38await cb(*args, **kwargs)
Mar 06 18:42:38await cb(*args, **kwargs)
Mar 06 18:42:38await oidc.load_metadata()
Mar 06 18:42:38await oidc.load_metadata()
Mar 06 18:42:38await p.load_metadata()
Mar 06 18:42:38await p.load_metadata()
Mar 06 18:42:38body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 18:42:38body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 18:42:38current.result = callback( # type: ignore[misc]
Mar 06 18:42:38current.result = callback( # type: ignore[misc]
Mar 06 18:42:38metadata_response = await self._http_client.get_json(url)
Mar 06 18:42:38metadata_response = await self._http_client.get_json(url)
Mar 06 18:42:38raise Exception(
Mar 06 18:42:38raise Exception(
Mar 06 18:42:38raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 18:42:38raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 18:42:38raise self.value.with_traceback(self.tb)
Mar 06 18:42:38raise self.value.with_traceback(self.tb)
Mar 06 18:42:38response = await make_deferred_yieldable(request_deferred)
Mar 06 18:42:38response = await make_deferred_yieldable(request_deferred)
Mar 06 18:42:38response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 18:42:38response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 18:42:38result = context.run(
Mar 06 18:42:38result = context.run(
Mar 06 18:42:38return await f()
Mar 06 18:42:38return await f()
Mar 06 18:42:38return await self._cachedcall.get()
Mar 06 18:42:38return await self._cachedcall.get()
Mar 06 18:42:38return await self._provider_metadata.get()
Mar 06 18:42:38return await self._provider_metadata.get()
Mar 06 18:42:38return g.throw(self.value.with_traceback(self.tb))
Mar 06 18:42:38return g.throw(self.value.with_traceback(self.tb))
Mar 06 18:42:38self._result.raiseException()
Mar 06 18:42:38self._result.raiseException()
Mar 06 18:42:38synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 18:42:38synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 18:42:47=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:42:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:43:07=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:43:17=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:43:27=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:43:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.19.117:8008
Mar 06 18:43:38==> Ensure we log to console
Mar 06 18:43:38==> Configuring synapse
Mar 06 18:43:38==> Configuring OIDC auth
Mar 06 18:43:39==> Fixing permissions
Mar 06 18:43:39==> Starting synapse
Mar 06 18:43:40=> Healtheck error: Error: connect ECONNREFUSED 172.18.19.117:8008
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 355 - WARNING - main - ***** STARTING SERVER *****
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 356 - WARNING - main - Server /app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py version 1.102.0
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 365 - INFO - main - Server hostname: myurl.cloud
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 366 - INFO - main - Instance name: master
Mar 06 18:43:402024-03-06 17:43:40,465 - root - 367 - INFO - main - Twisted reactor: EPollReactor
Mar 06 18:43:402024-03-06 17:43:40,466 - synapse.app.homeserver - 356 - INFO - main - Setting up server
Mar 06 18:43:402024-03-06 17:43:40,466 - synapse.server - 345 - INFO - main - Setting up.
Mar 06 18:43:402024-03-06 17:43:40,495 - synapse.storage.databases - 73 - INFO - main - [database config 'master']: Checking database server
Mar 06 18:43:402024-03-06 17:43:40,497 - synapse.storage.databases - 76 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state']
Mar 06 18:43:402024-03-06 17:43:40,497 - synapse.storage.prepare_database - 135 - INFO - main - ['main', 'state']: Checking existing schema version
Mar 06 18:43:402024-03-06 17:43:40,500 - synapse.storage.prepare_database - 139 - INFO - main - ['main', 'state']: Existing schema is 84 (+3 deltas)
Mar 06 18:43:402024-03-06 17:43:40,500 - synapse.storage.databases.main - 388 - INFO - main - Checking database for consistency with configuration...
Mar 06 18:43:402024-03-06 17:43:40,501 - synapse.storage.prepare_database - 433 - INFO - main - Applying schema deltas for v84
Mar 06 18:43:402024-03-06 17:43:40,502 - synapse.storage.prepare_database - 568 - INFO - main - Schema now up to date
Mar 06 18:43:402024-03-06 17:43:40,503 - synapse.storage.databases - 91 - INFO - main - [database config 'master']: Starting 'main' database
Mar 06 18:43:402024-03-06 17:43:40,523 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for pushers(id): 80
Mar 06 18:43:402024-03-06 17:43:40,523 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for deleted_pushers(stream_id): 1
Mar 06 18:43:402024-03-06 17:43:40,524 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_stream(stream_id): 1
Mar 06 18:43:402024-03-06 17:43:40,530 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for e2e_cross_signing_keys(stream_id): 7
Mar 06 18:43:402024-03-06 17:43:40,530 - synapse.storage.databases.main.event_push_actions - 1167 - INFO - main - Searching for stream ordering 1 month ago
Mar 06 18:43:402024-03-06 17:43:40,533 - synapse.storage.databases.main.event_push_actions - 1171 - INFO - main - Found stream ordering 1 month ago: it's 106
Mar 06 18:43:402024-03-06 17:43:40,533 - synapse.storage.databases.main.event_push_actions - 1174 - INFO - main - Searching for stream ordering 1 day ago
Mar 06 18:43:402024-03-06 17:43:40,535 - synapse.storage.databases.main.event_push_actions - 1178 - INFO - main - Found stream ordering 1 day ago: it's 106
Mar 06 18:43:402024-03-06 17:43:40,536 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules(id): 1
Mar 06 18:43:402024-03-06 17:43:40,537 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_enable(id): 1
Mar 06 18:43:402024-03-06 17:43:40,540 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for access_tokens(id): 8
Mar 06 18:43:402024-03-06 17:43:40,540 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for refresh_tokens(id): 1
Mar 06 18:43:402024-03-06 17:43:40,543 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for event_reports(id): 1
Mar 06 18:43:402024-03-06 17:43:40,543 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_stream(stream_id): 26
Mar 06 18:43:402024-03-06 17:43:40,544 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for user_signature_stream(stream_id): 19
Mar 06 18:43:402024-03-06 17:43:40,544 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_outbound_pokes(stream_id): 1
Mar 06 18:43:402024-03-06 17:43:40,545 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_in_room(stream_id): 26
Mar 06 18:43:402024-03-06 17:43:40,545 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_remote_pending(stream_id): 1
Mar 06 18:43:402024-03-06 17:43:40,546 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 26
Mar 06 18:43:402024-03-06 17:43:40,547 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database
Mar 06 18:43:402024-03-06 17:43:40,548 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared
Mar 06 18:43:402024-03-06 17:43:40,548 - synapse.server - 348 - INFO - main - Finished setting up.
Mar 06 18:43:402024-03-06 17:43:40,568 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True
Mar 06 18:43:402024-03-06 17:43:40,568 - synapse.push.pusher - 54 - INFO - main - defined email pusher type
Mar 06 18:43:402024-03-06 17:43:40,569 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update'
Mar 06 18:43:402024-03-06 17:43:40,569 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile'
Mar 06 18:43:402024-03-06 17:43:40,570 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence'
Mar 06 18:43:402024-03-06 17:43:40,571 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing'
Mar 06 18:43:402024-03-06 17:43:40,571 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory'
Mar 06 18:43:402024-03-06 17:43:40,572 - synapse.app.homeserver - 179 - INFO - sentinel - Running
Mar 06 18:43:402024-03-06 17:43:40,573 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576
Mar 06 18:43:402024-03-06 17:43:40,576 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter
Mar 06 18:43:402024-03-06 17:43:40,602 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Mar 06 18:43:402024-03-06 17:43:40,602 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup
Mar 06 18:43:40Traceback (most recent call last):
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 18:43:40await p.load_metadata()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 18:43:40return await self._provider_metadata.get()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 18:43:40return await self._cachedcall.get()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 18:43:40self._result.raiseException()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 18:43:40raise self.value.with_traceback(self.tb)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 18:43:40result = context.run(
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 18:43:40return g.throw(self.value.with_traceback(self.tb))
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 18:43:40return await f()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 18:43:40metadata_response = await self._http_client.get_json(url)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 18:43:40body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 18:43:40response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 18:43:40response = await make_deferred_yieldable(request_deferred)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 18:43:40current.result = callback( # type: ignore[misc]
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 18:43:40raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 18:43:40synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 18:43:40
Mar 06 18:43:40The above exception was the direct cause of the following exception:
Mar 06 18:43:40
Mar 06 18:43:40Traceback (most recent call last):
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 18:43:40await cb(*args, **kwargs)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 18:43:40await oidc.load_metadata()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 18:43:40raise Exception(
Mar 06 18:43:40Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 18:43:40Error during startup:
Mar 06 18:43:40Traceback (most recent call last):
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 18:43:40await p.load_metadata()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 18:43:40return await self._provider_metadata.get()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 18:43:40return await self._cachedcall.get()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 18:43:40self._result.raiseException()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 18:43:40raise self.value.with_traceback(self.tb)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 18:43:40result = context.run(
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 18:43:40return g.throw(self.value.with_traceback(self.tb))
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 18:43:40return await f()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 18:43:40metadata_response = await self._http_client.get_json(url)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 18:43:40body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 18:43:40response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 18:43:40response = await make_deferred_yieldable(request_deferred)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 18:43:40current.result = callback( # type: ignore[misc]
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 18:43:40raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 18:43:40synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 18:43:40
Mar 06 18:43:40The above exception was the direct cause of the following exception:
Mar 06 18:43:40
Mar 06 18:43:40Traceback (most recent call last):
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 18:43:40await cb(*args, **kwargs)
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 18:43:40await oidc.load_metadata()
Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 18:43:40raise Exception(
Mar 06 18:43:40Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 18:43:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:44:07=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:44:17=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:44:27=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 18:44:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.19.117:8008
Mar 06 18:44:41=> Healtheck error: Error: connect ECONNREFUSED 172.18.19.117:8008
Mar 06 18:44:41==> Ensure we log to console
Mar 06 18:44:41==> Configuring synapse
Mar 06 18:44:41==> Configuring OIDC auth
Mar 06 18:44:41==> Fixing permissions
Mar 06 18:44:41==> Starting synapse

Thank you very much for your support.

girish

@rompo one idea is to install a test synapse instance with cloudron authentication and see if it starts. Just want to check if this is a migration issue or a general issue with even new app install.

rompo

@girish
Unfortunately exactly the same with the test installation.
Attached is the log, all IP's URL's masked of course.

Divided into two posts:

Mar 06 19:29:32box:services teardownAddons: Tearing down addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"}
Mar 06 19:29:32box:services Tearing down OpenID connect
Mar 06 19:29:32box:services teardownAddons: Tearing down addon postgresql with options {}
Mar 06 19:29:32box:services Tearing down sendmail
Mar 06 19:29:32box:services teardownAddons: Tearing down addon sendmail with options {"supportsDisplayName":true}
Mar 06 19:29:32box:services Tearing down TURN
Mar 06 19:29:32box:services teardownAddons: Tearing down addon turn with options {"optional":true}
Mar 06 19:29:32box:tasks update 1889: {"percent":20,"message":"Downloading icon"}
Mar 06 19:29:32box:apptask downloadIcon: Downloading icon of org.matrix.synapse@1.89.0
Mar 06 19:29:33box:tasks update 1889: {"percent":30,"message":"Registering subdomains"}
Mar 06 19:29:33box:dns registerLocations: Will register [{"subdomain":"test","domain":"myurl.cloud"}] with options {"overwriteDns":false}
Mar 06 19:29:33box:network/generic getIPv4: querying ipv4.api.cloudron.io to get server IPv4
Mar 06 19:29:33box:tasks update 1889: {"message":"Registering location test.myurl.cloud"}
Mar 06 19:29:33box:dns upsertDNSRecord: location test on domain myurl.cloud of type A with values ["IP.xxx.xxx.xxx"]
Mar 06 19:29:33box:dns/manual upsert: test for zone myurl.cloud of type A with values ["IP.xxx.xxx.xxx"]
Mar 06 19:29:33box:tasks update 1889: {"percent":40,"message":"Downloading image"}
Mar 06 19:29:33box:docker downloadImage cloudron/org.matrix.synapse:20240306-082100-8745d1299
Mar 06 19:29:33box:tasks update 1889: {"percent":50,"message":"Creating app data directory"}
Mar 06 19:29:33box:tasks update 1889: {"percent":60,"message":"Setting up addons"}
Mar 06 19:29:33box:services setupAddons: Setting up ["localstorage","oidc","postgresql","sendmail","turn"]
Mar 06 19:29:33box:services setupAddons: setting up addon localstorage with options {}
Mar 06 19:29:33box:services setupLocalStorage
Mar 06 19:29:33box:shell createVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/322db221-a8da-40cc-b611-c51d1c445afe/data
Mar 06 19:29:33box:services Setting up OpenID connect
Mar 06 19:29:33box:services setupAddons: setting up addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"}
Mar 06 19:29:33box:services Setting up postgresql
Mar 06 19:29:33box:services setupAddons: setting up addon postgresql with options {}
Mar 06 19:29:34box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://user322db221a8da40ccb611c51d1c445afe:6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8@postgresql/db322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"user322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"db322db221a8da40ccb611c51d1c445afe"}]
Mar 06 19:29:34box:services Setting up SendMail
Mar 06 19:29:34box:services setupAddons: setting up addon sendmail with options {"supportsDisplayName":true}
Mar 06 19:29:34box:services Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_STARTTLS_PORT","value":"2587"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"20a582fdce400a37916df0ed7067ca0401ca70548fbc9f53"},{"name":"CLOUDRON_MAIL_FROM","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"myurl.cloud"},{"name":"CLOUDRON_MAIL_FROM_DISPLAY_NAME","value":""}]
Mar 06 19:29:34box:services setupAddons: setting up addon turn with options {"optional":true}
Mar 06 19:29:34box:services Setting up TURN
Mar 06 19:29:34box:tasks update 1889: {"percent":80,"message":"Creating container"}
Mar 06 19:29:34box:apptask createContainer: creating container
Mar 06 19:29:34box:shell addLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add 322db221-a8da-40cc-b611-c51d1c445afe /tmp/322db221-a8da-40cc-b611-c51d1c445afe.logrotate
Mar 06 19:29:34box:apptask startApp: starting container
Mar 06 19:29:35box:tasks update 1889: {"percent":85,"message":"Waiting for DNS propagation"}
Mar 06 19:29:35box:tasks update 1889: {"message":"Waiting for propagation of test.myurl.cloud"}
Mar 06 19:29:35box:dns/waitfordns waitForDns: waiting for test.myurl.cloud to be IP.xxx.xxx.xxx in zone myurl.cloud
Mar 06 19:29:35box:dns/waitfordns waitForDns: nameservers are ["ns.dns.ch","ns3.dns.ch","ns2.dns.ch"]
Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.51.254
Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns.dns.ch (IP.xxx.xxx.xxx). Expecting IP.xxx.xxx.xxx. Match true
Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns.dns.ch: done
Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.48.126
Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns3.dns.ch (IPx.26.48.126). Expecting IP.xxx.xxx.xxx. Match true
Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns3.dns.ch: done
Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.53.254
Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns2.dns.ch (IPx.26.53.254). Expecting IP.xxx.xxx.xxx. Match true
Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns2.dns.ch: done
Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud has propagated
Mar 06 19:29:35box:tasks update 1889: {"percent":95,"message":"Configuring reverse proxy"}
Mar 06 19:29:35box:reverseproxy ensureCertificate: test.myurl.cloud needs acme cert
Mar 06 19:29:35box:cert/acme2 getCertificate: for fqdn test.myurl.cloud and domain myurl.cloud
Mar 06 19:29:35box:cert/acme2 Acme2: will get cert for fqdn: test.myurl.cloud cn: test.myurl.cloud certName: test.myurl.cloud wildcard: false http: true
Mar 06 19:29:35box:cert/acme2 getCertificate: start acme flow for test.myurl.cloud from https://acme-v02.api.letsencrypt.org/directory
Mar 06 19:29:35box:cert/acme2 ensureAccount: registering user
Mar 06 19:29:36box:cert/acme2 sendSignedRequest: using nonce cotaLR5qvMBxfQA4SNbk8Ko-erxbPi2VvPiXYAughl8bcKsSuC4 for url https://acme-v02.api.letsencrypt.org/acme/new-acct
Mar 06 19:29:36box:cert/acme2 ensureAccount: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017
Mar 06 19:29:36box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 email: mail@myurl.ch
Mar 06 19:29:37box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYhhH0xfnDNAo1Bl2f9U44ClLZXbX7XQaaKxGJ5PkhcBk for url https://acme-v02.api.letsencrypt.org/acme/acct/1184548017
Mar 06 19:29:37box:cert/acme2 newOrder: ["test.myurl.cloud"]
Mar 06 19:29:37box:cert/acme2 updateContact: contact of user updated to mail@myurl.ch
Mar 06 19:29:38box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYQ6a2L3Vg6YSqGyctudVGLjufbho5IkucmNqhJjNl-GA for url https://acme-v02.api.letsencrypt.org/acme/new-order
Mar 06 19:29:38box:cert/acme2 acmeFlow: authorizing https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207
Mar 06 19:29:38box:cert/acme2 newOrder: created order test.myurl.cloud order: {"status":"pending","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377"} orderUrl: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377
Mar 06 19:29:39box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYzapghMJmdXLUVDmdU1VsbkyXNyYf1UJu5gX81FGgcdU for url https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207
Mar 06 19:29:39box:cert/acme2 prepareChallenge: http: true cn: test.myurl.cloud authorization: {"identifier":{"type":"dns","value":"test.myurl.cloud"},"status":"pending","expires":"2024-03-13T18:29:38Z","challenges":[{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"dns-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/RFIFGA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"tls-alpn-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/utwyeg","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}]}
Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: preparing for challenge {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}
Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: writing MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8.SBfGjSqlq1SOBwqKKsOm145-XCcexjlawaxzVxmM4Lo to /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8
Mar 06 19:29:39box:cert/acme2 notifyChallengeReady: https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA was met
Mar 06 19:29:40box:cert/acme2 sendSignedRequest: using nonce aSlagK2nxWbiFYMEzdjOOs67JiUTSdkQa5c0KsdYyxflSh6YodA for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA
Mar 06 19:29:40box:cert/acme2 waitingForChallenge: getting status
Mar 06 19:29:40box:cert/acme2 waitingForChallenge: {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}
Mar 06 19:29:41box:cert/acme2 sendSignedRequest: using nonce aSlagK2nvzbvDsCfWxmLYAmz7iGa41i00W7LrHJNoBZ4yrZb_ug for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA
Mar 06 19:29:41box:cert/acme2 Attempt 1 failed. Will retry: Challenge is in pending state
Mar 06 19:29:41box:cert/acme2 waitForChallenge: status is "pending" "{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}"
Mar 06 19:30:01box:cert/acme2 waitingForChallenge: getting status
Mar 06 19:30:02box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYCP-hlG5T8aUkkVuEBo1RqoIMZTVo4X0tutfWn0eQ1_4 for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA
Mar 06 19:30:02box:cert/acme2 cleanupChallenge: http: true
Mar 06 19:30:02box:cert/acme2 cleanupHttpChallenge: unlinking /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8
Mar 06 19:30:02box:cert/acme2 ensureKey: generating new key for test.myurl.cloud
Mar 06 19:30:02box:cert/acme2 createCsr: csr file created for test.myurl.cloud
Mar 06 19:30:02box:cert/acme2 signCertificate: sending sign request to https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377
Mar 06 19:30:03box:cert/acme2 sendSignedRequest: using nonce Jssy1fJY6FWDnG_-WUTKfJGV0bd6QTMF5WM5RPVFNyQmoy91P9Q for url https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377
Mar 06 19:30:04box:cert/acme2 waitForOrder: getting status
Mar 06 19:30:04box:cert/acme2 waitForOrder: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377
Mar 06 19:30:05box:cert/acme2 sendSignedRequest: using nonce iCRP5AvKyEZrYffHFYMppftm1B6_FDstPRwqnwL3ryx8JoMYPOQ for url https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377
Mar 06 19:30:05box:cert/acme2 waitForOrder: status is "valid {"status":"valid","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377","certificate":"https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8"}
Mar 06 19:30:05box:cert/acme2 downloadCertificate: downloading certificate of test.myurl.cloud
Mar 06 19:30:06box:cert/acme2 sendSignedRequest: using nonce aSlagK2n4Sf3NPZUaGuPwvD9SO2mGRmzQrn_YEMPk-J1t8OgKvk for url https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8
Mar 06 19:30:06box:cert/acme2 getCertificate: acme flow completed for test.myurl.cloud
Mar 06 19:30:06box:reverseproxy ensureCertificate: error: null
Mar 06 19:30:06box:reverseproxy writeAppLocationNginxConfig: writing config for "test.myurl.cloud" to /home/yellowtent/platformdata/nginx/applications/322db221-a8da-40cc-b611-c51d1c445afe/test.myurl.cloud.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"test.myurl.cloud","hasIPv6":true,"ip":"172.18.17.56","port":8008,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"id":"322db221-a8da-40cc-b611-c51d1c445afe","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Mar 06 19:30:06box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Mar 06 19:30:06box:tasks update 1889: {"percent":100,"message":"Done"}
Mar 06 19:30:06box:tasks setCompleted - 1889: {"result":null,"error":null}
Mar 06 19:30:06box:tasks update 1889: {"percent":100,"result":null,"error":null}
Mar 06 19:30:06box:taskworker Task took 34.899 seconds
Mar 06 19:32:45
Mar 06 19:32:45
Mar 06 19:32:45
Mar 06 19:32:45
Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database
Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1
Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.server - 348 - INFO - main - Finished setting up.
Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared
Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True
Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 54 - INFO - main - defined email pusher type
Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update'
Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile'
Mar 06 19:32:452024-03-06 18:32:45,918 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence'
Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing'
Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory'
Mar 06 19:32:452024-03-06 18:32:45,920 - synapse.app.homeserver - 179 - INFO - sentinel - Running
Mar 06 19:32:452024-03-06 18:32:45,921 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576
Mar 06 19:32:452024-03-06 18:32:45,924 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter
Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup
Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Mar 06 19:32:45Error during startup:
Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator

girish

@rompo said in Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0:

Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server

Thanks, the above is the root cause. The app is not able to CURL itself for some reason.

Are you behind Cloudflare?
Open a web terminal (of any app) and then curl https://my.myurl.cloud/openid/.well-known/openid-configuration . We need to find out why this is not working.

rompo

Mar 06 19:32:45The above exception was the direct cause of the following exception:
Mar 06 19:32:45The above exception was the direct cause of the following exception:
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45Traceback (most recent call last):
Mar 06 19:32:45await cb(*args, **kwargs)
Mar 06 19:32:45await cb(*args, **kwargs)
Mar 06 19:32:45await oidc.load_metadata()
Mar 06 19:32:45await oidc.load_metadata()
Mar 06 19:32:45await p.load_metadata()
Mar 06 19:32:45await p.load_metadata()
Mar 06 19:32:45body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:32:45body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:32:45current.result = callback( # type: ignore[misc]
Mar 06 19:32:45current.result = callback( # type: ignore[misc]
Mar 06 19:32:45metadata_response = await self._http_client.get_json(url)
Mar 06 19:32:45metadata_response = await self._http_client.get_json(url)
Mar 06 19:32:45raise Exception(
Mar 06 19:32:45raise Exception(
Mar 06 19:32:45raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:32:45raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:32:45raise self.value.with_traceback(self.tb)
Mar 06 19:32:45raise self.value.with_traceback(self.tb)
Mar 06 19:32:45response = await make_deferred_yieldable(request_deferred)
Mar 06 19:32:45response = await make_deferred_yieldable(request_deferred)
Mar 06 19:32:45response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:32:45response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:32:45result = context.run(
Mar 06 19:32:45result = context.run(
Mar 06 19:32:45return await f()
Mar 06 19:32:45return await f()
Mar 06 19:32:45return await self._cachedcall.get()
Mar 06 19:32:45return await self._cachedcall.get()
Mar 06 19:32:45return await self._provider_metadata.get()
Mar 06 19:32:45return await self._provider_metadata.get()
Mar 06 19:32:45return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:32:45return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:32:45self._result.raiseException()
Mar 06 19:32:45self._result.raiseException()
Mar 06 19:32:45synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:32:45synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:32:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 19:33:03=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:13=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:23=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:43=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:33:46==> Ensure we log to console
Mar 06 19:33:46==> Configuring synapse
Mar 06 19:33:46==> Configuring OIDC auth
Mar 06 19:33:46==> Fixing permissions
Mar 06 19:33:46==> Starting synapse
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 355 - WARNING - main - ***** STARTING SERVER *****
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 356 - WARNING - main - Server /app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py version 1.102.0
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 365 - INFO - main - Server hostname: myurl.cloud
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 366 - INFO - main - Instance name: master
Mar 06 19:33:482024-03-06 18:33:48,048 - root - 367 - INFO - main - Twisted reactor: EPollReactor
Mar 06 19:33:482024-03-06 18:33:48,048 - synapse.app.homeserver - 356 - INFO - main - Setting up server
Mar 06 19:33:482024-03-06 18:33:48,049 - synapse.server - 345 - INFO - main - Setting up.
Mar 06 19:33:482024-03-06 18:33:48,075 - synapse.storage.databases - 73 - INFO - main - [database config 'master']: Checking database server
Mar 06 19:33:482024-03-06 18:33:48,076 - synapse.storage.databases - 76 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state']
Mar 06 19:33:482024-03-06 18:33:48,076 - synapse.storage.prepare_database - 135 - INFO - main - ['main', 'state']: Checking existing schema version
Mar 06 19:33:482024-03-06 18:33:48,078 - synapse.storage.prepare_database - 139 - INFO - main - ['main', 'state']: Existing schema is 84 (+3 deltas)
Mar 06 19:33:482024-03-06 18:33:48,078 - synapse.storage.databases.main - 388 - INFO - main - Checking database for consistency with configuration...
Mar 06 19:33:482024-03-06 18:33:48,079 - synapse.storage.prepare_database - 433 - INFO - main - Applying schema deltas for v84
Mar 06 19:33:482024-03-06 18:33:48,080 - synapse.storage.prepare_database - 568 - INFO - main - Schema now up to date
Mar 06 19:33:482024-03-06 18:33:48,081 - synapse.storage.databases - 91 - INFO - main - [database config 'master']: Starting 'main' database
Mar 06 19:33:482024-03-06 18:33:48,093 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for pushers(id): 1
Mar 06 19:33:482024-03-06 18:33:48,094 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for deleted_pushers(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,094 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_stream(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for e2e_cross_signing_keys(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.databases.main.event_push_actions - 1167 - INFO - main - Searching for stream ordering 1 month ago
Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.databases.main.event_push_actions - 1171 - INFO - main - Found stream ordering 1 month ago: it's 0
Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.databases.main.event_push_actions - 1174 - INFO - main - Searching for stream ordering 1 day ago
Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.databases.main.event_push_actions - 1178 - INFO - main - Found stream ordering 1 day ago: it's 0
Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules(id): 1
Mar 06 19:33:482024-03-06 18:33:48,099 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_enable(id): 1
Mar 06 19:33:482024-03-06 18:33:48,100 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for access_tokens(id): 1
Mar 06 19:33:482024-03-06 18:33:48,101 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for refresh_tokens(id): 1
Mar 06 19:33:482024-03-06 18:33:48,102 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for event_reports(id): 1
Mar 06 19:33:482024-03-06 18:33:48,102 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_stream(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for user_signature_stream(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_outbound_pokes(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_in_room(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,104 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_remote_pending(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,104 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1
Mar 06 19:33:482024-03-06 18:33:48,105 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database
Mar 06 19:33:482024-03-06 18:33:48,106 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared
Mar 06 19:33:482024-03-06 18:33:48,106 - synapse.server - 348 - INFO - main - Finished setting up.
Mar 06 19:33:482024-03-06 18:33:48,124 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True
Mar 06 19:33:482024-03-06 18:33:48,124 - synapse.push.pusher - 54 - INFO - main - defined email pusher type
Mar 06 19:33:482024-03-06 18:33:48,125 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update'
Mar 06 19:33:482024-03-06 18:33:48,125 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile'
Mar 06 19:33:482024-03-06 18:33:48,126 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence'
Mar 06 19:33:482024-03-06 18:33:48,127 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing'
Mar 06 19:33:482024-03-06 18:33:48,127 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory'
Mar 06 19:33:482024-03-06 18:33:48,128 - synapse.app.homeserver - 179 - INFO - sentinel - Running
Mar 06 19:33:482024-03-06 18:33:48,129 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576
Mar 06 19:33:482024-03-06 18:33:48,133 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter
Mar 06 19:33:482024-03-06 18:33:48,158 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Mar 06 19:33:482024-03-06 18:33:48,158 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 19:33:48await p.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 19:33:48return await self._provider_metadata.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 19:33:48return await self._cachedcall.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 19:33:48self._result.raiseException()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 19:33:48raise self.value.with_traceback(self.tb)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 19:33:48result = context.run(
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 19:33:48return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 19:33:48return await f()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 19:33:48metadata_response = await self._http_client.get_json(url)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 19:33:48body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 19:33:48response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 19:33:48response = await make_deferred_yieldable(request_deferred)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 19:33:48current.result = callback( # type: ignore[misc]
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 19:33:48raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:33:48synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:33:48
Mar 06 19:33:48The above exception was the direct cause of the following exception:
Mar 06 19:33:48
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 19:33:48await cb(*args, **kwargs)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 19:33:48await oidc.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 19:33:48raise Exception(
Mar 06 19:33:48Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 19:33:48Error during startup:
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata
Mar 06 19:33:48await p.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata
Mar 06 19:33:48return await self._provider_metadata.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get
Mar 06 19:33:48return await self._cachedcall.get()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get
Mar 06 19:33:48self._result.raiseException()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException
Mar 06 19:33:48raise self.value.with_traceback(self.tb)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
Mar 06 19:33:48result = context.run(
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
Mar 06 19:33:48return g.throw(self.value.with_traceback(self.tb))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper
Mar 06 19:33:48return await f()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata
Mar 06 19:33:48metadata_response = await self._http_client.get_json(url)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json
Mar 06 19:33:48body = await self.get_raw(uri, args, headers=actual_headers)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw
Mar 06 19:33:48response = await self.request("GET", uri, headers=Headers(actual_headers))
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request
Mar 06 19:33:48response = await make_deferred_yieldable(request_deferred)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks
Mar 06 19:33:48current.result = callback( # type: ignore[misc]
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error
Mar 06 19:33:48raise RequestTimedOutError("Timeout waiting for response from remote server")
Mar 06 19:33:48synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server
Mar 06 19:33:48
Mar 06 19:33:48The above exception was the direct cause of the following exception:
Mar 06 19:33:48
Mar 06 19:33:48Traceback (most recent call last):
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper
Mar 06 19:33:48await cb(*args, **kwargs)
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start
Mar 06 19:33:48await oidc.load_metadata()
Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata
Mar 06 19:33:48raise Exception(
Mar 06 19:33:48Exception: Error while initialising OIDC provider 'oidc-cloudron'
Mar 06 19:33:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Mar 06 19:34:03=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:13=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:23=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:43=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008
Mar 06 19:34:48==> Ensure we log to console
Mar 06 19:34:48==> Configuring synapse
Mar 06 19:34:48==> Configuring OIDC auth
Mar 06 19:34:48==> Fixing permissions
Mar 06 19:34:48==> Starting synapse

rompo

@girish

No, I'm not behind Cloudflaire.

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

girish

@rompo you to check why that curl command does not work on your system. If the certificate is valid then it's a DNS related issue. Please check what the DNS of my.myurl.cloud is resolving to.

rompo

This post is deleted!

rompo

@girish

The DNS resolves correctly from my point of view.
my.myurl.cloud is also accessible from the Internet as well as from the internal network and I can log in to it.

Unreachable ports cannot be a problem either.

However, if I want to access the web interface via the internal IP, I get the following message: You are seeing this page because the DNS record of 172.27.212.20 is set to this server's IP but Cloudron has no app configured for this domain.

girish

@rompo you cannot access Cloudron pages by IP address, it has to be by domain name entirely.

Can you try to resolve host my.myurl.cloud from the web terminal? Does it give internal IP or external IP ? If it is the external IP, does your network support hairpinning - the ability to access internal Cloudron when using external IP ? See https://docs.cloudron.io/troubleshooting/#hairpin-nat

rompo

@girish
I receive the public IP as output.

With pfSense, there is a "host override" for all Cloudron URLs, which actually resolves correctly (internal 172 addresses within the network).

However, I have now activated NAT Reflection with everything that goes with it as a test, which does not result in any change for Cloudron. Within the 172 network, however, publicIP is now resolved and routed correctly.

However, it is not entirely clear to me why Cloudron outputs the public IP in the original configuration with the "Host Override" entries, because the DNS server address is actually assigned via DHCP.
How can I check this? I can't get to the required interface with ifconfig.

girish

@rompo Looks like we get closer. Cloudron has it's own internal recursive DNS resolver. This means that it might be bypassing pfSense DNS. Please see https://docs.cloudron.io/networking/#internal-dns-server to make Cloudron use the internal DNS. This way it will resolve DNS to internal network values .

rompo

@girish
Thanks for the tip, I have entered pfSense as DNS, now it works again.

At the end of last week I set up a test server to play with which is connected to a different firewall. I had exactly the same problem with this fresh installation.

Another problem with the new login procedure is that mobile devices with the policy of not allowing cookies can no longer log in to the Element app.
An alternative solution must now be found, possibly deactivating openID if this is possible, but that is another topic.

Many thanks for your help and have a good week!

For all those who need a short summary:

Log in via SSH
Create the file "custom.conf":

sudo touch /etc/unbound/unbound.conf.d/custom.conf

Edit file:

sudo nano /etc/unbound/unbound.conf.d/custom.conf

Insert content into the file:

# this disables DNSSEC
server:
  val-permissive-mode: yes

# forward all queries to the internal DNS
forward-zone:
  name: "."
  forward-addr: 172.xxx.xxx.xxx

Restart:

sudo systemctl restart unbound

Check:

sudo systemctl status unbound

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0