Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0
-
Hello everyone
As the title already describes, Matrix can no longer be started completely since an automatic update was carried out.
A complete restart of everything did not help either.
In the logs I have the feeling that it could fail in this part. In any case, it starts to reload immediately after the "Error: connect EHOSTUNREACH 172.18.19.117:8008". What kind of IP this is is also a mystery to me.
Does anyone have any ideas?
Dec 05 23:06:24Exception: Error while initialising OIDC provider 'oidc-cloudron'
Dec 05 23:06:37=> Healtheck error: Error: Timeout of 7000ms exceeded
Dec 05 23:06:47=> Healtheck error: Error: Timeout of 7000ms exceeded
Dec 05 23:06:57=> Healtheck error: Error: Timeout of 7000ms exceeded
Dec 05 23:07:00=> Healtheck error: Error: connect EHOSTUNREACH 172.18.19.117:8008 -
So , per https://github.com/matrix-org/synapse/issues/9662 , it seems there might be a cert issue. Do you have valid certs for the Cloudron dashboard ?
@girish
Thanks for your feedback and sorry for the late feedback, I didn't have the capacity to deal with the problem.Yes, the certificates are up to date and we don't have a problem with the certificates in the dashboard or other apps.
I'll add the log of an entire cycle below.
Mar 06 18:42:38 Mar 06 18:42:38 Mar 06 18:42:38 Mar 06 18:42:38 Mar 06 18:42:382024-03-06 17:42:38,090 - synapse.server - 348 - INFO - main - Finished setting up. Mar 06 18:42:382024-03-06 17:42:38,090 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared Mar 06 18:42:382024-03-06 17:42:38,109 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True Mar 06 18:42:382024-03-06 17:42:38,110 - synapse.push.pusher - 54 - INFO - main - defined email pusher type Mar 06 18:42:382024-03-06 17:42:38,111 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update' Mar 06 18:42:382024-03-06 17:42:38,111 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile' Mar 06 18:42:382024-03-06 17:42:38,112 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence' Mar 06 18:42:382024-03-06 17:42:38,112 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing' Mar 06 18:42:382024-03-06 17:42:38,113 - synapse.app.homeserver - 179 - INFO - sentinel - Running Mar 06 18:42:382024-03-06 17:42:38,113 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory' Mar 06 18:42:382024-03-06 17:42:38,115 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576 Mar 06 18:42:382024-03-06 17:42:38,118 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter Mar 06 18:42:382024-03-06 17:42:38,191 - synapse.handlers.deactivate_account - 250 - INFO - user_parter_loop-0 - User parter finished: stopping Mar 06 18:42:382024-03-06 17:42:38,317 - synapse.metrics._gc - 127 - INFO - sentinel - Collecting gc 1 Mar 06 18:42:382024-03-06 17:42:38,325 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server Mar 06 18:42:382024-03-06 17:42:38,326 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup Mar 06 18:42:38Error during startup: Mar 06 18:42:38Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 18:42:38Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 18:42:38File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 18:42:38The above exception was the direct cause of the following exception: Mar 06 18:42:38The above exception was the direct cause of the following exception: Mar 06 18:42:38Traceback (most recent call last): Mar 06 18:42:38Traceback (most recent call last): Mar 06 18:42:38Traceback (most recent call last): Mar 06 18:42:38Traceback (most recent call last): Mar 06 18:42:38await cb(*args, **kwargs) Mar 06 18:42:38await cb(*args, **kwargs) Mar 06 18:42:38await oidc.load_metadata() Mar 06 18:42:38await oidc.load_metadata() Mar 06 18:42:38await p.load_metadata() Mar 06 18:42:38await p.load_metadata() Mar 06 18:42:38body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 18:42:38body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 18:42:38current.result = callback( # type: ignore[misc] Mar 06 18:42:38current.result = callback( # type: ignore[misc] Mar 06 18:42:38metadata_response = await self._http_client.get_json(url) Mar 06 18:42:38metadata_response = await self._http_client.get_json(url) Mar 06 18:42:38raise Exception( Mar 06 18:42:38raise Exception( Mar 06 18:42:38raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 18:42:38raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 18:42:38raise self.value.with_traceback(self.tb) Mar 06 18:42:38raise self.value.with_traceback(self.tb) Mar 06 18:42:38response = await make_deferred_yieldable(request_deferred) Mar 06 18:42:38response = await make_deferred_yieldable(request_deferred) Mar 06 18:42:38response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 18:42:38response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 18:42:38result = context.run( Mar 06 18:42:38result = context.run( Mar 06 18:42:38return await f() Mar 06 18:42:38return await f() Mar 06 18:42:38return await self._cachedcall.get() Mar 06 18:42:38return await self._cachedcall.get() Mar 06 18:42:38return await self._provider_metadata.get() Mar 06 18:42:38return await self._provider_metadata.get() Mar 06 18:42:38return g.throw(self.value.with_traceback(self.tb)) Mar 06 18:42:38return g.throw(self.value.with_traceback(self.tb)) Mar 06 18:42:38self._result.raiseException() Mar 06 18:42:38self._result.raiseException() Mar 06 18:42:38synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 18:42:38synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 18:42:47=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:42:57=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:43:07=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:43:17=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:43:27=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:43:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.19.117:8008 Mar 06 18:43:38==> Ensure we log to console Mar 06 18:43:38==> Configuring synapse Mar 06 18:43:38==> Configuring OIDC auth Mar 06 18:43:39==> Fixing permissions Mar 06 18:43:39==> Starting synapse Mar 06 18:43:40=> Healtheck error: Error: connect ECONNREFUSED 172.18.19.117:8008 Mar 06 18:43:402024-03-06 17:43:40,465 - root - 355 - WARNING - main - ***** STARTING SERVER ***** Mar 06 18:43:402024-03-06 17:43:40,465 - root - 356 - WARNING - main - Server /app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py version 1.102.0 Mar 06 18:43:402024-03-06 17:43:40,465 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc Mar 06 18:43:402024-03-06 17:43:40,465 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse Mar 06 18:43:402024-03-06 17:43:40,465 - root - 365 - INFO - main - Server hostname: myurl.cloud Mar 06 18:43:402024-03-06 17:43:40,465 - root - 366 - INFO - main - Instance name: master Mar 06 18:43:402024-03-06 17:43:40,465 - root - 367 - INFO - main - Twisted reactor: EPollReactor Mar 06 18:43:402024-03-06 17:43:40,466 - synapse.app.homeserver - 356 - INFO - main - Setting up server Mar 06 18:43:402024-03-06 17:43:40,466 - synapse.server - 345 - INFO - main - Setting up. Mar 06 18:43:402024-03-06 17:43:40,495 - synapse.storage.databases - 73 - INFO - main - [database config 'master']: Checking database server Mar 06 18:43:402024-03-06 17:43:40,497 - synapse.storage.databases - 76 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state'] Mar 06 18:43:402024-03-06 17:43:40,497 - synapse.storage.prepare_database - 135 - INFO - main - ['main', 'state']: Checking existing schema version Mar 06 18:43:402024-03-06 17:43:40,500 - synapse.storage.prepare_database - 139 - INFO - main - ['main', 'state']: Existing schema is 84 (+3 deltas) Mar 06 18:43:402024-03-06 17:43:40,500 - synapse.storage.databases.main - 388 - INFO - main - Checking database for consistency with configuration... Mar 06 18:43:402024-03-06 17:43:40,501 - synapse.storage.prepare_database - 433 - INFO - main - Applying schema deltas for v84 Mar 06 18:43:402024-03-06 17:43:40,502 - synapse.storage.prepare_database - 568 - INFO - main - Schema now up to date Mar 06 18:43:402024-03-06 17:43:40,503 - synapse.storage.databases - 91 - INFO - main - [database config 'master']: Starting 'main' database Mar 06 18:43:402024-03-06 17:43:40,523 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for pushers(id): 80 Mar 06 18:43:402024-03-06 17:43:40,523 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for deleted_pushers(stream_id): 1 Mar 06 18:43:402024-03-06 17:43:40,524 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_stream(stream_id): 1 Mar 06 18:43:402024-03-06 17:43:40,530 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for e2e_cross_signing_keys(stream_id): 7 Mar 06 18:43:402024-03-06 17:43:40,530 - synapse.storage.databases.main.event_push_actions - 1167 - INFO - main - Searching for stream ordering 1 month ago Mar 06 18:43:402024-03-06 17:43:40,533 - synapse.storage.databases.main.event_push_actions - 1171 - INFO - main - Found stream ordering 1 month ago: it's 106 Mar 06 18:43:402024-03-06 17:43:40,533 - synapse.storage.databases.main.event_push_actions - 1174 - INFO - main - Searching for stream ordering 1 day ago Mar 06 18:43:402024-03-06 17:43:40,535 - synapse.storage.databases.main.event_push_actions - 1178 - INFO - main - Found stream ordering 1 day ago: it's 106 Mar 06 18:43:402024-03-06 17:43:40,536 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules(id): 1 Mar 06 18:43:402024-03-06 17:43:40,537 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_enable(id): 1 Mar 06 18:43:402024-03-06 17:43:40,540 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for access_tokens(id): 8 Mar 06 18:43:402024-03-06 17:43:40,540 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for refresh_tokens(id): 1 Mar 06 18:43:402024-03-06 17:43:40,543 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for event_reports(id): 1 Mar 06 18:43:402024-03-06 17:43:40,543 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_stream(stream_id): 26 Mar 06 18:43:402024-03-06 17:43:40,544 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for user_signature_stream(stream_id): 19 Mar 06 18:43:402024-03-06 17:43:40,544 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_outbound_pokes(stream_id): 1 Mar 06 18:43:402024-03-06 17:43:40,545 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_in_room(stream_id): 26 Mar 06 18:43:402024-03-06 17:43:40,545 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_remote_pending(stream_id): 1 Mar 06 18:43:402024-03-06 17:43:40,546 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 26 Mar 06 18:43:402024-03-06 17:43:40,547 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database Mar 06 18:43:402024-03-06 17:43:40,548 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared Mar 06 18:43:402024-03-06 17:43:40,548 - synapse.server - 348 - INFO - main - Finished setting up. Mar 06 18:43:402024-03-06 17:43:40,568 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True Mar 06 18:43:402024-03-06 17:43:40,568 - synapse.push.pusher - 54 - INFO - main - defined email pusher type Mar 06 18:43:402024-03-06 17:43:40,569 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update' Mar 06 18:43:402024-03-06 17:43:40,569 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile' Mar 06 18:43:402024-03-06 17:43:40,570 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence' Mar 06 18:43:402024-03-06 17:43:40,571 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing' Mar 06 18:43:402024-03-06 17:43:40,571 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory' Mar 06 18:43:402024-03-06 17:43:40,572 - synapse.app.homeserver - 179 - INFO - sentinel - Running Mar 06 18:43:402024-03-06 17:43:40,573 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576 Mar 06 18:43:402024-03-06 17:43:40,576 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter Mar 06 18:43:402024-03-06 17:43:40,602 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server Mar 06 18:43:402024-03-06 17:43:40,602 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup Mar 06 18:43:40Traceback (most recent call last): Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 18:43:40await p.load_metadata() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 18:43:40return await self._provider_metadata.get() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 18:43:40return await self._cachedcall.get() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 18:43:40self._result.raiseException() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 18:43:40raise self.value.with_traceback(self.tb) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 18:43:40result = context.run( Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 18:43:40return g.throw(self.value.with_traceback(self.tb)) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 18:43:40return await f() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 18:43:40metadata_response = await self._http_client.get_json(url) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 18:43:40body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 18:43:40response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 18:43:40response = await make_deferred_yieldable(request_deferred) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 18:43:40current.result = callback( # type: ignore[misc] Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 18:43:40raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 18:43:40synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 18:43:40 Mar 06 18:43:40The above exception was the direct cause of the following exception: Mar 06 18:43:40 Mar 06 18:43:40Traceback (most recent call last): Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 18:43:40await cb(*args, **kwargs) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 18:43:40await oidc.load_metadata() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 18:43:40raise Exception( Mar 06 18:43:40Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 18:43:40Error during startup: Mar 06 18:43:40Traceback (most recent call last): Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 18:43:40await p.load_metadata() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 18:43:40return await self._provider_metadata.get() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 18:43:40return await self._cachedcall.get() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 18:43:40self._result.raiseException() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 18:43:40raise self.value.with_traceback(self.tb) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 18:43:40result = context.run( Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 18:43:40return g.throw(self.value.with_traceback(self.tb)) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 18:43:40return await f() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 18:43:40metadata_response = await self._http_client.get_json(url) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 18:43:40body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 18:43:40response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 18:43:40response = await make_deferred_yieldable(request_deferred) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 18:43:40current.result = callback( # type: ignore[misc] Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 18:43:40raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 18:43:40synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 18:43:40 Mar 06 18:43:40The above exception was the direct cause of the following exception: Mar 06 18:43:40 Mar 06 18:43:40Traceback (most recent call last): Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 18:43:40await cb(*args, **kwargs) Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 18:43:40await oidc.load_metadata() Mar 06 18:43:40File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 18:43:40raise Exception( Mar 06 18:43:40Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 18:43:57=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:44:07=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:44:17=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:44:27=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 18:44:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.19.117:8008 Mar 06 18:44:41=> Healtheck error: Error: connect ECONNREFUSED 172.18.19.117:8008 Mar 06 18:44:41==> Ensure we log to console Mar 06 18:44:41==> Configuring synapse Mar 06 18:44:41==> Configuring OIDC auth Mar 06 18:44:41==> Fixing permissions Mar 06 18:44:41==> Starting synapseThank you very much for your support.
-
@rompo one idea is to install a test synapse instance with cloudron authentication and see if it starts. Just want to check if this is a migration issue or a general issue with even new app install.
@girish
Unfortunately exactly the same with the test installation.
Attached is the log, all IP's URL's masked of course.Divided into two posts:
Mar 06 19:29:32box:services teardownAddons: Tearing down addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"} Mar 06 19:29:32box:services Tearing down OpenID connect Mar 06 19:29:32box:services teardownAddons: Tearing down addon postgresql with options {} Mar 06 19:29:32box:services Tearing down sendmail Mar 06 19:29:32box:services teardownAddons: Tearing down addon sendmail with options {"supportsDisplayName":true} Mar 06 19:29:32box:services Tearing down TURN Mar 06 19:29:32box:services teardownAddons: Tearing down addon turn with options {"optional":true} Mar 06 19:29:32box:tasks update 1889: {"percent":20,"message":"Downloading icon"} Mar 06 19:29:32box:apptask downloadIcon: Downloading icon of org.matrix.synapse@1.89.0 Mar 06 19:29:33box:tasks update 1889: {"percent":30,"message":"Registering subdomains"} Mar 06 19:29:33box:dns registerLocations: Will register [{"subdomain":"test","domain":"myurl.cloud"}] with options {"overwriteDns":false} Mar 06 19:29:33box:network/generic getIPv4: querying ipv4.api.cloudron.io to get server IPv4 Mar 06 19:29:33box:tasks update 1889: {"message":"Registering location test.myurl.cloud"} Mar 06 19:29:33box:dns upsertDNSRecord: location test on domain myurl.cloud of type A with values ["IP.xxx.xxx.xxx"] Mar 06 19:29:33box:dns/manual upsert: test for zone myurl.cloud of type A with values ["IP.xxx.xxx.xxx"] Mar 06 19:29:33box:tasks update 1889: {"percent":40,"message":"Downloading image"} Mar 06 19:29:33box:docker downloadImage cloudron/org.matrix.synapse:20240306-082100-8745d1299 Mar 06 19:29:33box:tasks update 1889: {"percent":50,"message":"Creating app data directory"} Mar 06 19:29:33box:tasks update 1889: {"percent":60,"message":"Setting up addons"} Mar 06 19:29:33box:services setupAddons: Setting up ["localstorage","oidc","postgresql","sendmail","turn"] Mar 06 19:29:33box:services setupAddons: setting up addon localstorage with options {} Mar 06 19:29:33box:services setupLocalStorage Mar 06 19:29:33box:shell createVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/322db221-a8da-40cc-b611-c51d1c445afe/data Mar 06 19:29:33box:services Setting up OpenID connect Mar 06 19:29:33box:services setupAddons: setting up addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"} Mar 06 19:29:33box:services Setting up postgresql Mar 06 19:29:33box:services setupAddons: setting up addon postgresql with options {} Mar 06 19:29:34box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://user322db221a8da40ccb611c51d1c445afe:6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8@postgresql/db322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"user322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"db322db221a8da40ccb611c51d1c445afe"}] Mar 06 19:29:34box:services Setting up SendMail Mar 06 19:29:34box:services setupAddons: setting up addon sendmail with options {"supportsDisplayName":true} Mar 06 19:29:34box:services Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_STARTTLS_PORT","value":"2587"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"20a582fdce400a37916df0ed7067ca0401ca70548fbc9f53"},{"name":"CLOUDRON_MAIL_FROM","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"myurl.cloud"},{"name":"CLOUDRON_MAIL_FROM_DISPLAY_NAME","value":""}] Mar 06 19:29:34box:services setupAddons: setting up addon turn with options {"optional":true} Mar 06 19:29:34box:services Setting up TURN Mar 06 19:29:34box:tasks update 1889: {"percent":80,"message":"Creating container"} Mar 06 19:29:34box:apptask createContainer: creating container Mar 06 19:29:34box:shell addLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add 322db221-a8da-40cc-b611-c51d1c445afe /tmp/322db221-a8da-40cc-b611-c51d1c445afe.logrotate Mar 06 19:29:34box:apptask startApp: starting container Mar 06 19:29:35box:tasks update 1889: {"percent":85,"message":"Waiting for DNS propagation"} Mar 06 19:29:35box:tasks update 1889: {"message":"Waiting for propagation of test.myurl.cloud"} Mar 06 19:29:35box:dns/waitfordns waitForDns: waiting for test.myurl.cloud to be IP.xxx.xxx.xxx in zone myurl.cloud Mar 06 19:29:35box:dns/waitfordns waitForDns: nameservers are ["ns.dns.ch","ns3.dns.ch","ns2.dns.ch"] Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.51.254 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns.dns.ch (IP.xxx.xxx.xxx). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns.dns.ch: done Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.48.126 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns3.dns.ch (IPx.26.48.126). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns3.dns.ch: done Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.53.254 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns2.dns.ch (IPx.26.53.254). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns2.dns.ch: done Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud has propagated Mar 06 19:29:35box:tasks update 1889: {"percent":95,"message":"Configuring reverse proxy"} Mar 06 19:29:35box:reverseproxy ensureCertificate: test.myurl.cloud needs acme cert Mar 06 19:29:35box:cert/acme2 getCertificate: for fqdn test.myurl.cloud and domain myurl.cloud Mar 06 19:29:35box:cert/acme2 Acme2: will get cert for fqdn: test.myurl.cloud cn: test.myurl.cloud certName: test.myurl.cloud wildcard: false http: true Mar 06 19:29:35box:cert/acme2 getCertificate: start acme flow for test.myurl.cloud from https://acme-v02.api.letsencrypt.org/directory Mar 06 19:29:35box:cert/acme2 ensureAccount: registering user Mar 06 19:29:36box:cert/acme2 sendSignedRequest: using nonce cotaLR5qvMBxfQA4SNbk8Ko-erxbPi2VvPiXYAughl8bcKsSuC4 for url https://acme-v02.api.letsencrypt.org/acme/new-acct Mar 06 19:29:36box:cert/acme2 ensureAccount: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 Mar 06 19:29:36box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 email: mail@myurl.ch Mar 06 19:29:37box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYhhH0xfnDNAo1Bl2f9U44ClLZXbX7XQaaKxGJ5PkhcBk for url https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 Mar 06 19:29:37box:cert/acme2 newOrder: ["test.myurl.cloud"] Mar 06 19:29:37box:cert/acme2 updateContact: contact of user updated to mail@myurl.ch Mar 06 19:29:38box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYQ6a2L3Vg6YSqGyctudVGLjufbho5IkucmNqhJjNl-GA for url https://acme-v02.api.letsencrypt.org/acme/new-order Mar 06 19:29:38box:cert/acme2 acmeFlow: authorizing https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207 Mar 06 19:29:38box:cert/acme2 newOrder: created order test.myurl.cloud order: {"status":"pending","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377"} orderUrl: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:29:39box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYzapghMJmdXLUVDmdU1VsbkyXNyYf1UJu5gX81FGgcdU for url https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207 Mar 06 19:29:39box:cert/acme2 prepareChallenge: http: true cn: test.myurl.cloud authorization: {"identifier":{"type":"dns","value":"test.myurl.cloud"},"status":"pending","expires":"2024-03-13T18:29:38Z","challenges":[{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"dns-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/RFIFGA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"tls-alpn-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/utwyeg","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}]} Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: preparing for challenge {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"} Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: writing MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8.SBfGjSqlq1SOBwqKKsOm145-XCcexjlawaxzVxmM4Lo to /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8 Mar 06 19:29:39box:cert/acme2 notifyChallengeReady: https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA was met Mar 06 19:29:40box:cert/acme2 sendSignedRequest: using nonce aSlagK2nxWbiFYMEzdjOOs67JiUTSdkQa5c0KsdYyxflSh6YodA for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:29:40box:cert/acme2 waitingForChallenge: getting status Mar 06 19:29:40box:cert/acme2 waitingForChallenge: {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"} Mar 06 19:29:41box:cert/acme2 sendSignedRequest: using nonce aSlagK2nvzbvDsCfWxmLYAmz7iGa41i00W7LrHJNoBZ4yrZb_ug for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:29:41box:cert/acme2 Attempt 1 failed. Will retry: Challenge is in pending state Mar 06 19:29:41box:cert/acme2 waitForChallenge: status is "pending" "{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}" Mar 06 19:30:01box:cert/acme2 waitingForChallenge: getting status Mar 06 19:30:02box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYCP-hlG5T8aUkkVuEBo1RqoIMZTVo4X0tutfWn0eQ1_4 for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:30:02box:cert/acme2 cleanupChallenge: http: true Mar 06 19:30:02box:cert/acme2 cleanupHttpChallenge: unlinking /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8 Mar 06 19:30:02box:cert/acme2 ensureKey: generating new key for test.myurl.cloud Mar 06 19:30:02box:cert/acme2 createCsr: csr file created for test.myurl.cloud Mar 06 19:30:02box:cert/acme2 signCertificate: sending sign request to https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377 Mar 06 19:30:03box:cert/acme2 sendSignedRequest: using nonce Jssy1fJY6FWDnG_-WUTKfJGV0bd6QTMF5WM5RPVFNyQmoy91P9Q for url https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377 Mar 06 19:30:04box:cert/acme2 waitForOrder: getting status Mar 06 19:30:04box:cert/acme2 waitForOrder: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:30:05box:cert/acme2 sendSignedRequest: using nonce iCRP5AvKyEZrYffHFYMppftm1B6_FDstPRwqnwL3ryx8JoMYPOQ for url https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:30:05box:cert/acme2 waitForOrder: status is "valid {"status":"valid","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377","certificate":"https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8"} Mar 06 19:30:05box:cert/acme2 downloadCertificate: downloading certificate of test.myurl.cloud Mar 06 19:30:06box:cert/acme2 sendSignedRequest: using nonce aSlagK2n4Sf3NPZUaGuPwvD9SO2mGRmzQrn_YEMPk-J1t8OgKvk for url https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8 Mar 06 19:30:06box:cert/acme2 getCertificate: acme flow completed for test.myurl.cloud Mar 06 19:30:06box:reverseproxy ensureCertificate: error: null Mar 06 19:30:06box:reverseproxy writeAppLocationNginxConfig: writing config for "test.myurl.cloud" to /home/yellowtent/platformdata/nginx/applications/322db221-a8da-40cc-b611-c51d1c445afe/test.myurl.cloud.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"test.myurl.cloud","hasIPv6":true,"ip":"172.18.17.56","port":8008,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"id":"322db221-a8da-40cc-b611-c51d1c445afe","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false} Mar 06 19:30:06box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx Mar 06 19:30:06box:tasks update 1889: {"percent":100,"message":"Done"} Mar 06 19:30:06box:tasks setCompleted - 1889: {"result":null,"error":null} Mar 06 19:30:06box:tasks update 1889: {"percent":100,"result":null,"error":null} Mar 06 19:30:06box:taskworker Task took 34.899 seconds Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1 Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.server - 348 - INFO - main - Finished setting up. Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 54 - INFO - main - defined email pusher type Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update' Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile' Mar 06 19:32:452024-03-06 18:32:45,918 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence' Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing' Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory' Mar 06 19:32:452024-03-06 18:32:45,920 - synapse.app.homeserver - 179 - INFO - sentinel - Running Mar 06 19:32:452024-03-06 18:32:45,921 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576 Mar 06 19:32:452024-03-06 18:32:45,924 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server Mar 06 19:32:45Error during startup: Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator -
@girish
Unfortunately exactly the same with the test installation.
Attached is the log, all IP's URL's masked of course.Divided into two posts:
Mar 06 19:29:32box:services teardownAddons: Tearing down addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"} Mar 06 19:29:32box:services Tearing down OpenID connect Mar 06 19:29:32box:services teardownAddons: Tearing down addon postgresql with options {} Mar 06 19:29:32box:services Tearing down sendmail Mar 06 19:29:32box:services teardownAddons: Tearing down addon sendmail with options {"supportsDisplayName":true} Mar 06 19:29:32box:services Tearing down TURN Mar 06 19:29:32box:services teardownAddons: Tearing down addon turn with options {"optional":true} Mar 06 19:29:32box:tasks update 1889: {"percent":20,"message":"Downloading icon"} Mar 06 19:29:32box:apptask downloadIcon: Downloading icon of org.matrix.synapse@1.89.0 Mar 06 19:29:33box:tasks update 1889: {"percent":30,"message":"Registering subdomains"} Mar 06 19:29:33box:dns registerLocations: Will register [{"subdomain":"test","domain":"myurl.cloud"}] with options {"overwriteDns":false} Mar 06 19:29:33box:network/generic getIPv4: querying ipv4.api.cloudron.io to get server IPv4 Mar 06 19:29:33box:tasks update 1889: {"message":"Registering location test.myurl.cloud"} Mar 06 19:29:33box:dns upsertDNSRecord: location test on domain myurl.cloud of type A with values ["IP.xxx.xxx.xxx"] Mar 06 19:29:33box:dns/manual upsert: test for zone myurl.cloud of type A with values ["IP.xxx.xxx.xxx"] Mar 06 19:29:33box:tasks update 1889: {"percent":40,"message":"Downloading image"} Mar 06 19:29:33box:docker downloadImage cloudron/org.matrix.synapse:20240306-082100-8745d1299 Mar 06 19:29:33box:tasks update 1889: {"percent":50,"message":"Creating app data directory"} Mar 06 19:29:33box:tasks update 1889: {"percent":60,"message":"Setting up addons"} Mar 06 19:29:33box:services setupAddons: Setting up ["localstorage","oidc","postgresql","sendmail","turn"] Mar 06 19:29:33box:services setupAddons: setting up addon localstorage with options {} Mar 06 19:29:33box:services setupLocalStorage Mar 06 19:29:33box:shell createVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/322db221-a8da-40cc-b611-c51d1c445afe/data Mar 06 19:29:33box:services Setting up OpenID connect Mar 06 19:29:33box:services setupAddons: setting up addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"} Mar 06 19:29:33box:services Setting up postgresql Mar 06 19:29:33box:services setupAddons: setting up addon postgresql with options {} Mar 06 19:29:34box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://user322db221a8da40ccb611c51d1c445afe:6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8@postgresql/db322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"user322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"db322db221a8da40ccb611c51d1c445afe"}] Mar 06 19:29:34box:services Setting up SendMail Mar 06 19:29:34box:services setupAddons: setting up addon sendmail with options {"supportsDisplayName":true} Mar 06 19:29:34box:services Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_STARTTLS_PORT","value":"2587"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"20a582fdce400a37916df0ed7067ca0401ca70548fbc9f53"},{"name":"CLOUDRON_MAIL_FROM","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"myurl.cloud"},{"name":"CLOUDRON_MAIL_FROM_DISPLAY_NAME","value":""}] Mar 06 19:29:34box:services setupAddons: setting up addon turn with options {"optional":true} Mar 06 19:29:34box:services Setting up TURN Mar 06 19:29:34box:tasks update 1889: {"percent":80,"message":"Creating container"} Mar 06 19:29:34box:apptask createContainer: creating container Mar 06 19:29:34box:shell addLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add 322db221-a8da-40cc-b611-c51d1c445afe /tmp/322db221-a8da-40cc-b611-c51d1c445afe.logrotate Mar 06 19:29:34box:apptask startApp: starting container Mar 06 19:29:35box:tasks update 1889: {"percent":85,"message":"Waiting for DNS propagation"} Mar 06 19:29:35box:tasks update 1889: {"message":"Waiting for propagation of test.myurl.cloud"} Mar 06 19:29:35box:dns/waitfordns waitForDns: waiting for test.myurl.cloud to be IP.xxx.xxx.xxx in zone myurl.cloud Mar 06 19:29:35box:dns/waitfordns waitForDns: nameservers are ["ns.dns.ch","ns3.dns.ch","ns2.dns.ch"] Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.51.254 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns.dns.ch (IP.xxx.xxx.xxx). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns.dns.ch: done Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.48.126 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns3.dns.ch (IPx.26.48.126). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns3.dns.ch: done Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.53.254 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns2.dns.ch (IPx.26.53.254). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns2.dns.ch: done Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud has propagated Mar 06 19:29:35box:tasks update 1889: {"percent":95,"message":"Configuring reverse proxy"} Mar 06 19:29:35box:reverseproxy ensureCertificate: test.myurl.cloud needs acme cert Mar 06 19:29:35box:cert/acme2 getCertificate: for fqdn test.myurl.cloud and domain myurl.cloud Mar 06 19:29:35box:cert/acme2 Acme2: will get cert for fqdn: test.myurl.cloud cn: test.myurl.cloud certName: test.myurl.cloud wildcard: false http: true Mar 06 19:29:35box:cert/acme2 getCertificate: start acme flow for test.myurl.cloud from https://acme-v02.api.letsencrypt.org/directory Mar 06 19:29:35box:cert/acme2 ensureAccount: registering user Mar 06 19:29:36box:cert/acme2 sendSignedRequest: using nonce cotaLR5qvMBxfQA4SNbk8Ko-erxbPi2VvPiXYAughl8bcKsSuC4 for url https://acme-v02.api.letsencrypt.org/acme/new-acct Mar 06 19:29:36box:cert/acme2 ensureAccount: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 Mar 06 19:29:36box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 email: mail@myurl.ch Mar 06 19:29:37box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYhhH0xfnDNAo1Bl2f9U44ClLZXbX7XQaaKxGJ5PkhcBk for url https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 Mar 06 19:29:37box:cert/acme2 newOrder: ["test.myurl.cloud"] Mar 06 19:29:37box:cert/acme2 updateContact: contact of user updated to mail@myurl.ch Mar 06 19:29:38box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYQ6a2L3Vg6YSqGyctudVGLjufbho5IkucmNqhJjNl-GA for url https://acme-v02.api.letsencrypt.org/acme/new-order Mar 06 19:29:38box:cert/acme2 acmeFlow: authorizing https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207 Mar 06 19:29:38box:cert/acme2 newOrder: created order test.myurl.cloud order: {"status":"pending","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377"} orderUrl: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:29:39box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYzapghMJmdXLUVDmdU1VsbkyXNyYf1UJu5gX81FGgcdU for url https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207 Mar 06 19:29:39box:cert/acme2 prepareChallenge: http: true cn: test.myurl.cloud authorization: {"identifier":{"type":"dns","value":"test.myurl.cloud"},"status":"pending","expires":"2024-03-13T18:29:38Z","challenges":[{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"dns-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/RFIFGA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"tls-alpn-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/utwyeg","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}]} Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: preparing for challenge {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"} Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: writing MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8.SBfGjSqlq1SOBwqKKsOm145-XCcexjlawaxzVxmM4Lo to /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8 Mar 06 19:29:39box:cert/acme2 notifyChallengeReady: https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA was met Mar 06 19:29:40box:cert/acme2 sendSignedRequest: using nonce aSlagK2nxWbiFYMEzdjOOs67JiUTSdkQa5c0KsdYyxflSh6YodA for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:29:40box:cert/acme2 waitingForChallenge: getting status Mar 06 19:29:40box:cert/acme2 waitingForChallenge: {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"} Mar 06 19:29:41box:cert/acme2 sendSignedRequest: using nonce aSlagK2nvzbvDsCfWxmLYAmz7iGa41i00W7LrHJNoBZ4yrZb_ug for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:29:41box:cert/acme2 Attempt 1 failed. Will retry: Challenge is in pending state Mar 06 19:29:41box:cert/acme2 waitForChallenge: status is "pending" "{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}" Mar 06 19:30:01box:cert/acme2 waitingForChallenge: getting status Mar 06 19:30:02box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYCP-hlG5T8aUkkVuEBo1RqoIMZTVo4X0tutfWn0eQ1_4 for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:30:02box:cert/acme2 cleanupChallenge: http: true Mar 06 19:30:02box:cert/acme2 cleanupHttpChallenge: unlinking /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8 Mar 06 19:30:02box:cert/acme2 ensureKey: generating new key for test.myurl.cloud Mar 06 19:30:02box:cert/acme2 createCsr: csr file created for test.myurl.cloud Mar 06 19:30:02box:cert/acme2 signCertificate: sending sign request to https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377 Mar 06 19:30:03box:cert/acme2 sendSignedRequest: using nonce Jssy1fJY6FWDnG_-WUTKfJGV0bd6QTMF5WM5RPVFNyQmoy91P9Q for url https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377 Mar 06 19:30:04box:cert/acme2 waitForOrder: getting status Mar 06 19:30:04box:cert/acme2 waitForOrder: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:30:05box:cert/acme2 sendSignedRequest: using nonce iCRP5AvKyEZrYffHFYMppftm1B6_FDstPRwqnwL3ryx8JoMYPOQ for url https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:30:05box:cert/acme2 waitForOrder: status is "valid {"status":"valid","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377","certificate":"https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8"} Mar 06 19:30:05box:cert/acme2 downloadCertificate: downloading certificate of test.myurl.cloud Mar 06 19:30:06box:cert/acme2 sendSignedRequest: using nonce aSlagK2n4Sf3NPZUaGuPwvD9SO2mGRmzQrn_YEMPk-J1t8OgKvk for url https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8 Mar 06 19:30:06box:cert/acme2 getCertificate: acme flow completed for test.myurl.cloud Mar 06 19:30:06box:reverseproxy ensureCertificate: error: null Mar 06 19:30:06box:reverseproxy writeAppLocationNginxConfig: writing config for "test.myurl.cloud" to /home/yellowtent/platformdata/nginx/applications/322db221-a8da-40cc-b611-c51d1c445afe/test.myurl.cloud.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"test.myurl.cloud","hasIPv6":true,"ip":"172.18.17.56","port":8008,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"id":"322db221-a8da-40cc-b611-c51d1c445afe","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false} Mar 06 19:30:06box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx Mar 06 19:30:06box:tasks update 1889: {"percent":100,"message":"Done"} Mar 06 19:30:06box:tasks setCompleted - 1889: {"result":null,"error":null} Mar 06 19:30:06box:tasks update 1889: {"percent":100,"result":null,"error":null} Mar 06 19:30:06box:taskworker Task took 34.899 seconds Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1 Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.server - 348 - INFO - main - Finished setting up. Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 54 - INFO - main - defined email pusher type Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update' Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile' Mar 06 19:32:452024-03-06 18:32:45,918 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence' Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing' Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory' Mar 06 19:32:452024-03-06 18:32:45,920 - synapse.app.homeserver - 179 - INFO - sentinel - Running Mar 06 19:32:452024-03-06 18:32:45,921 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576 Mar 06 19:32:452024-03-06 18:32:45,924 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server Mar 06 19:32:45Error during startup: Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator@rompo said in Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0:
Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Thanks, the above is the root cause. The app is not able to CURL itself for some reason.
- Are you behind Cloudflare?
- Open a web terminal (of any app) and then
curl https://my.myurl.cloud/openid/.well-known/openid-configuration. We need to find out why this is not working.
-
@girish
Unfortunately exactly the same with the test installation.
Attached is the log, all IP's URL's masked of course.Divided into two posts:
Mar 06 19:29:32box:services teardownAddons: Tearing down addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"} Mar 06 19:29:32box:services Tearing down OpenID connect Mar 06 19:29:32box:services teardownAddons: Tearing down addon postgresql with options {} Mar 06 19:29:32box:services Tearing down sendmail Mar 06 19:29:32box:services teardownAddons: Tearing down addon sendmail with options {"supportsDisplayName":true} Mar 06 19:29:32box:services Tearing down TURN Mar 06 19:29:32box:services teardownAddons: Tearing down addon turn with options {"optional":true} Mar 06 19:29:32box:tasks update 1889: {"percent":20,"message":"Downloading icon"} Mar 06 19:29:32box:apptask downloadIcon: Downloading icon of org.matrix.synapse@1.89.0 Mar 06 19:29:33box:tasks update 1889: {"percent":30,"message":"Registering subdomains"} Mar 06 19:29:33box:dns registerLocations: Will register [{"subdomain":"test","domain":"myurl.cloud"}] with options {"overwriteDns":false} Mar 06 19:29:33box:network/generic getIPv4: querying ipv4.api.cloudron.io to get server IPv4 Mar 06 19:29:33box:tasks update 1889: {"message":"Registering location test.myurl.cloud"} Mar 06 19:29:33box:dns upsertDNSRecord: location test on domain myurl.cloud of type A with values ["IP.xxx.xxx.xxx"] Mar 06 19:29:33box:dns/manual upsert: test for zone myurl.cloud of type A with values ["IP.xxx.xxx.xxx"] Mar 06 19:29:33box:tasks update 1889: {"percent":40,"message":"Downloading image"} Mar 06 19:29:33box:docker downloadImage cloudron/org.matrix.synapse:20240306-082100-8745d1299 Mar 06 19:29:33box:tasks update 1889: {"percent":50,"message":"Creating app data directory"} Mar 06 19:29:33box:tasks update 1889: {"percent":60,"message":"Setting up addons"} Mar 06 19:29:33box:services setupAddons: Setting up ["localstorage","oidc","postgresql","sendmail","turn"] Mar 06 19:29:33box:services setupAddons: setting up addon localstorage with options {} Mar 06 19:29:33box:services setupLocalStorage Mar 06 19:29:33box:shell createVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/322db221-a8da-40cc-b611-c51d1c445afe/data Mar 06 19:29:33box:services Setting up OpenID connect Mar 06 19:29:33box:services setupAddons: setting up addon oidc with options {"loginRedirectUri":"/_synapse/client/oidc/callback"} Mar 06 19:29:33box:services Setting up postgresql Mar 06 19:29:33box:services setupAddons: setting up addon postgresql with options {} Mar 06 19:29:34box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://user322db221a8da40ccb611c51d1c445afe:6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8@postgresql/db322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"user322db221a8da40ccb611c51d1c445afe"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"6bdb880682dcf68a5b23b9a6003608628409e34522cb0e65d278a7e0b3770a4ccd0366adcfed31bf9d1bd196045b367c1a970472cc5542943425177a09bf63d8"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"db322db221a8da40ccb611c51d1c445afe"}] Mar 06 19:29:34box:services Setting up SendMail Mar 06 19:29:34box:services setupAddons: setting up addon sendmail with options {"supportsDisplayName":true} Mar 06 19:29:34box:services Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_STARTTLS_PORT","value":"2587"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"20a582fdce400a37916df0ed7067ca0401ca70548fbc9f53"},{"name":"CLOUDRON_MAIL_FROM","value":"test.app@myurl.cloud"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"myurl.cloud"},{"name":"CLOUDRON_MAIL_FROM_DISPLAY_NAME","value":""}] Mar 06 19:29:34box:services setupAddons: setting up addon turn with options {"optional":true} Mar 06 19:29:34box:services Setting up TURN Mar 06 19:29:34box:tasks update 1889: {"percent":80,"message":"Creating container"} Mar 06 19:29:34box:apptask createContainer: creating container Mar 06 19:29:34box:shell addLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add 322db221-a8da-40cc-b611-c51d1c445afe /tmp/322db221-a8da-40cc-b611-c51d1c445afe.logrotate Mar 06 19:29:34box:apptask startApp: starting container Mar 06 19:29:35box:tasks update 1889: {"percent":85,"message":"Waiting for DNS propagation"} Mar 06 19:29:35box:tasks update 1889: {"message":"Waiting for propagation of test.myurl.cloud"} Mar 06 19:29:35box:dns/waitfordns waitForDns: waiting for test.myurl.cloud to be IP.xxx.xxx.xxx in zone myurl.cloud Mar 06 19:29:35box:dns/waitfordns waitForDns: nameservers are ["ns.dns.ch","ns3.dns.ch","ns2.dns.ch"] Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.51.254 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns.dns.ch (IP.xxx.xxx.xxx). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns.dns.ch: done Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.48.126 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns3.dns.ch (IPx.26.48.126). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns3.dns.ch: done Mar 06 19:29:35box:dns/waitfordns resolveIp: Checking if test.myurl.cloud has A record at IPx.26.53.254 Mar 06 19:29:35box:dns/waitfordns isChangeSynced: test.myurl.cloud (A) was resolved to IP.xxx.xxx.xxx at NS ns2.dns.ch (IPx.26.53.254). Expecting IP.xxx.xxx.xxx. Match true Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud at ns ns2.dns.ch: done Mar 06 19:29:35box:dns/waitfordns waitForDns: test.myurl.cloud has propagated Mar 06 19:29:35box:tasks update 1889: {"percent":95,"message":"Configuring reverse proxy"} Mar 06 19:29:35box:reverseproxy ensureCertificate: test.myurl.cloud needs acme cert Mar 06 19:29:35box:cert/acme2 getCertificate: for fqdn test.myurl.cloud and domain myurl.cloud Mar 06 19:29:35box:cert/acme2 Acme2: will get cert for fqdn: test.myurl.cloud cn: test.myurl.cloud certName: test.myurl.cloud wildcard: false http: true Mar 06 19:29:35box:cert/acme2 getCertificate: start acme flow for test.myurl.cloud from https://acme-v02.api.letsencrypt.org/directory Mar 06 19:29:35box:cert/acme2 ensureAccount: registering user Mar 06 19:29:36box:cert/acme2 sendSignedRequest: using nonce cotaLR5qvMBxfQA4SNbk8Ko-erxbPi2VvPiXYAughl8bcKsSuC4 for url https://acme-v02.api.letsencrypt.org/acme/new-acct Mar 06 19:29:36box:cert/acme2 ensureAccount: user registered keyid: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 Mar 06 19:29:36box:cert/acme2 updateContact: registrationUri: https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 email: mail@myurl.ch Mar 06 19:29:37box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYhhH0xfnDNAo1Bl2f9U44ClLZXbX7XQaaKxGJ5PkhcBk for url https://acme-v02.api.letsencrypt.org/acme/acct/1184548017 Mar 06 19:29:37box:cert/acme2 newOrder: ["test.myurl.cloud"] Mar 06 19:29:37box:cert/acme2 updateContact: contact of user updated to mail@myurl.ch Mar 06 19:29:38box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYQ6a2L3Vg6YSqGyctudVGLjufbho5IkucmNqhJjNl-GA for url https://acme-v02.api.letsencrypt.org/acme/new-order Mar 06 19:29:38box:cert/acme2 acmeFlow: authorizing https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207 Mar 06 19:29:38box:cert/acme2 newOrder: created order test.myurl.cloud order: {"status":"pending","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377"} orderUrl: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:29:39box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYzapghMJmdXLUVDmdU1VsbkyXNyYf1UJu5gX81FGgcdU for url https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207 Mar 06 19:29:39box:cert/acme2 prepareChallenge: http: true cn: test.myurl.cloud authorization: {"identifier":{"type":"dns","value":"test.myurl.cloud"},"status":"pending","expires":"2024-03-13T18:29:38Z","challenges":[{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"dns-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/RFIFGA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"},{"type":"tls-alpn-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/utwyeg","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}]} Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: preparing for challenge {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"} Mar 06 19:29:39box:cert/acme2 prepareHttpChallenge: writing MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8.SBfGjSqlq1SOBwqKKsOm145-XCcexjlawaxzVxmM4Lo to /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8 Mar 06 19:29:39box:cert/acme2 notifyChallengeReady: https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA was met Mar 06 19:29:40box:cert/acme2 sendSignedRequest: using nonce aSlagK2nxWbiFYMEzdjOOs67JiUTSdkQa5c0KsdYyxflSh6YodA for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:29:40box:cert/acme2 waitingForChallenge: getting status Mar 06 19:29:40box:cert/acme2 waitingForChallenge: {"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"} Mar 06 19:29:41box:cert/acme2 sendSignedRequest: using nonce aSlagK2nvzbvDsCfWxmLYAmz7iGa41i00W7LrHJNoBZ4yrZb_ug for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:29:41box:cert/acme2 Attempt 1 failed. Will retry: Challenge is in pending state Mar 06 19:29:41box:cert/acme2 waitForChallenge: status is "pending" "{"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA","token":"MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8"}" Mar 06 19:30:01box:cert/acme2 waitingForChallenge: getting status Mar 06 19:30:02box:cert/acme2 sendSignedRequest: using nonce Jssy1fJYCP-hlG5T8aUkkVuEBo1RqoIMZTVo4X0tutfWn0eQ1_4 for url https://acme-v02.api.letsencrypt.org/acme/chall-v3/323203906207/VhkOwA Mar 06 19:30:02box:cert/acme2 cleanupChallenge: http: true Mar 06 19:30:02box:cert/acme2 cleanupHttpChallenge: unlinking /home/yellowtent/platformdata/acme/MCBGA69eEwTYpAD0JvM5rrXrlHZuzXG1OBWO8LFTzN8 Mar 06 19:30:02box:cert/acme2 ensureKey: generating new key for test.myurl.cloud Mar 06 19:30:02box:cert/acme2 createCsr: csr file created for test.myurl.cloud Mar 06 19:30:02box:cert/acme2 signCertificate: sending sign request to https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377 Mar 06 19:30:03box:cert/acme2 sendSignedRequest: using nonce Jssy1fJY6FWDnG_-WUTKfJGV0bd6QTMF5WM5RPVFNyQmoy91P9Q for url https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377 Mar 06 19:30:04box:cert/acme2 waitForOrder: getting status Mar 06 19:30:04box:cert/acme2 waitForOrder: https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:30:05box:cert/acme2 sendSignedRequest: using nonce iCRP5AvKyEZrYffHFYMppftm1B6_FDstPRwqnwL3ryx8JoMYPOQ for url https://acme-v02.api.letsencrypt.org/acme/order/1184548017/250131616377 Mar 06 19:30:05box:cert/acme2 waitForOrder: status is "valid {"status":"valid","expires":"2024-03-13T18:29:38Z","identifiers":[{"type":"dns","value":"test.myurl.cloud"}],"authorizations":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/323203906207"],"finalize":"https://acme-v02.api.letsencrypt.org/acme/finalize/1184548017/250131616377","certificate":"https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8"} Mar 06 19:30:05box:cert/acme2 downloadCertificate: downloading certificate of test.myurl.cloud Mar 06 19:30:06box:cert/acme2 sendSignedRequest: using nonce aSlagK2n4Sf3NPZUaGuPwvD9SO2mGRmzQrn_YEMPk-J1t8OgKvk for url https://acme-v02.api.letsencrypt.org/acme/cert/04a0ae9b552611803f87f86482757f50abc8 Mar 06 19:30:06box:cert/acme2 getCertificate: acme flow completed for test.myurl.cloud Mar 06 19:30:06box:reverseproxy ensureCertificate: error: null Mar 06 19:30:06box:reverseproxy writeAppLocationNginxConfig: writing config for "test.myurl.cloud" to /home/yellowtent/platformdata/nginx/applications/322db221-a8da-40cc-b611-c51d1c445afe/test.myurl.cloud.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"test.myurl.cloud","hasIPv6":true,"ip":"172.18.17.56","port":8008,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/test.myurl.cloud.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"id":"322db221-a8da-40cc-b611-c51d1c445afe","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false} Mar 06 19:30:06box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx Mar 06 19:30:06box:tasks update 1889: {"percent":100,"message":"Done"} Mar 06 19:30:06box:tasks setCompleted - 1889: {"result":null,"error":null} Mar 06 19:30:06box:tasks update 1889: {"percent":100,"result":null,"error":null} Mar 06 19:30:06box:taskworker Task took 34.899 seconds Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:45 Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database Mar 06 19:32:452024-03-06 18:32:45,897 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1 Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.server - 348 - INFO - main - Finished setting up. Mar 06 19:32:452024-03-06 18:32:45,898 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True Mar 06 19:32:452024-03-06 18:32:45,916 - synapse.push.pusher - 54 - INFO - main - defined email pusher type Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update' Mar 06 19:32:452024-03-06 18:32:45,917 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile' Mar 06 19:32:452024-03-06 18:32:45,918 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence' Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing' Mar 06 19:32:452024-03-06 18:32:45,919 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory' Mar 06 19:32:452024-03-06 18:32:45,920 - synapse.app.homeserver - 179 - INFO - sentinel - Running Mar 06 19:32:452024-03-06 18:32:45,921 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576 Mar 06 19:32:452024-03-06 18:32:45,924 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server Mar 06 19:32:45Error during startup: Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:32:45Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 19:32:45File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGeneratorMar 06 19:32:45The above exception was the direct cause of the following exception: Mar 06 19:32:45The above exception was the direct cause of the following exception: Mar 06 19:32:45Traceback (most recent call last): Mar 06 19:32:45Traceback (most recent call last): Mar 06 19:32:45Traceback (most recent call last): Mar 06 19:32:45Traceback (most recent call last): Mar 06 19:32:45await cb(*args, **kwargs) Mar 06 19:32:45await cb(*args, **kwargs) Mar 06 19:32:45await oidc.load_metadata() Mar 06 19:32:45await oidc.load_metadata() Mar 06 19:32:45await p.load_metadata() Mar 06 19:32:45await p.load_metadata() Mar 06 19:32:45body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 19:32:45body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 19:32:45current.result = callback( # type: ignore[misc] Mar 06 19:32:45current.result = callback( # type: ignore[misc] Mar 06 19:32:45metadata_response = await self._http_client.get_json(url) Mar 06 19:32:45metadata_response = await self._http_client.get_json(url) Mar 06 19:32:45raise Exception( Mar 06 19:32:45raise Exception( Mar 06 19:32:45raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 19:32:45raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 19:32:45raise self.value.with_traceback(self.tb) Mar 06 19:32:45raise self.value.with_traceback(self.tb) Mar 06 19:32:45response = await make_deferred_yieldable(request_deferred) Mar 06 19:32:45response = await make_deferred_yieldable(request_deferred) Mar 06 19:32:45response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 19:32:45response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 19:32:45result = context.run( Mar 06 19:32:45result = context.run( Mar 06 19:32:45return await f() Mar 06 19:32:45return await f() Mar 06 19:32:45return await self._cachedcall.get() Mar 06 19:32:45return await self._cachedcall.get() Mar 06 19:32:45return await self._provider_metadata.get() Mar 06 19:32:45return await self._provider_metadata.get() Mar 06 19:32:45return g.throw(self.value.with_traceback(self.tb)) Mar 06 19:32:45return g.throw(self.value.with_traceback(self.tb)) Mar 06 19:32:45self._result.raiseException() Mar 06 19:32:45self._result.raiseException() Mar 06 19:32:45synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 19:32:45synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 19:32:57=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 19:33:03=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:33:13=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:33:23=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:33:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:33:43=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:33:46==> Ensure we log to console Mar 06 19:33:46==> Configuring synapse Mar 06 19:33:46==> Configuring OIDC auth Mar 06 19:33:46==> Fixing permissions Mar 06 19:33:46==> Starting synapse Mar 06 19:33:482024-03-06 18:33:48,048 - root - 355 - WARNING - main - ***** STARTING SERVER ***** Mar 06 19:33:482024-03-06 18:33:48,048 - root - 356 - WARNING - main - Server /app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py version 1.102.0 Mar 06 19:33:482024-03-06 18:33:48,048 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc Mar 06 19:33:482024-03-06 18:33:48,048 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse Mar 06 19:33:482024-03-06 18:33:48,048 - root - 365 - INFO - main - Server hostname: myurl.cloud Mar 06 19:33:482024-03-06 18:33:48,048 - root - 366 - INFO - main - Instance name: master Mar 06 19:33:482024-03-06 18:33:48,048 - root - 367 - INFO - main - Twisted reactor: EPollReactor Mar 06 19:33:482024-03-06 18:33:48,048 - synapse.app.homeserver - 356 - INFO - main - Setting up server Mar 06 19:33:482024-03-06 18:33:48,049 - synapse.server - 345 - INFO - main - Setting up. Mar 06 19:33:482024-03-06 18:33:48,075 - synapse.storage.databases - 73 - INFO - main - [database config 'master']: Checking database server Mar 06 19:33:482024-03-06 18:33:48,076 - synapse.storage.databases - 76 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state'] Mar 06 19:33:482024-03-06 18:33:48,076 - synapse.storage.prepare_database - 135 - INFO - main - ['main', 'state']: Checking existing schema version Mar 06 19:33:482024-03-06 18:33:48,078 - synapse.storage.prepare_database - 139 - INFO - main - ['main', 'state']: Existing schema is 84 (+3 deltas) Mar 06 19:33:482024-03-06 18:33:48,078 - synapse.storage.databases.main - 388 - INFO - main - Checking database for consistency with configuration... Mar 06 19:33:482024-03-06 18:33:48,079 - synapse.storage.prepare_database - 433 - INFO - main - Applying schema deltas for v84 Mar 06 19:33:482024-03-06 18:33:48,080 - synapse.storage.prepare_database - 568 - INFO - main - Schema now up to date Mar 06 19:33:482024-03-06 18:33:48,081 - synapse.storage.databases - 91 - INFO - main - [database config 'master']: Starting 'main' database Mar 06 19:33:482024-03-06 18:33:48,093 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for pushers(id): 1 Mar 06 19:33:482024-03-06 18:33:48,094 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for deleted_pushers(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,094 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_stream(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for e2e_cross_signing_keys(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.databases.main.event_push_actions - 1167 - INFO - main - Searching for stream ordering 1 month ago Mar 06 19:33:482024-03-06 18:33:48,097 - synapse.storage.databases.main.event_push_actions - 1171 - INFO - main - Found stream ordering 1 month ago: it's 0 Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.databases.main.event_push_actions - 1174 - INFO - main - Searching for stream ordering 1 day ago Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.databases.main.event_push_actions - 1178 - INFO - main - Found stream ordering 1 day ago: it's 0 Mar 06 19:33:482024-03-06 18:33:48,098 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules(id): 1 Mar 06 19:33:482024-03-06 18:33:48,099 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for push_rules_enable(id): 1 Mar 06 19:33:482024-03-06 18:33:48,100 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for access_tokens(id): 1 Mar 06 19:33:482024-03-06 18:33:48,101 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for refresh_tokens(id): 1 Mar 06 19:33:482024-03-06 18:33:48,102 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for event_reports(id): 1 Mar 06 19:33:482024-03-06 18:33:48,102 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_stream(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for user_signature_stream(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_outbound_pokes(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,103 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_in_room(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,104 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_remote_pending(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,104 - synapse.storage.util.id_generators - 99 - INFO - main - Initialising stream generator for device_lists_changes_converted_stream_position(stream_id): 1 Mar 06 19:33:482024-03-06 18:33:48,105 - synapse.storage.databases - 108 - INFO - main - [database config 'master']: Starting 'state' database Mar 06 19:33:482024-03-06 18:33:48,106 - synapse.storage.databases - 123 - INFO - main - [database config 'master']: prepared Mar 06 19:33:482024-03-06 18:33:48,106 - synapse.server - 348 - INFO - main - Finished setting up. Mar 06 19:33:482024-03-06 18:33:48,124 - synapse.push.pusher - 45 - INFO - main - email enable notifs: True Mar 06 19:33:482024-03-06 18:33:48,124 - synapse.push.pusher - 54 - INFO - main - defined email pusher type Mar 06 19:33:482024-03-06 18:33:48,125 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.device_list_update' Mar 06 19:33:482024-03-06 18:33:48,125 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'profile' Mar 06 19:33:482024-03-06 18:33:48,126 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.presence' Mar 06 19:33:482024-03-06 18:33:48,127 - synapse.federation.federation_server - 1379 - INFO - main - Registering federation EDU handler for 'm.typing' Mar 06 19:33:482024-03-06 18:33:48,127 - synapse.federation.federation_server - 1399 - INFO - main - Registering federation query handler for 'directory' Mar 06 19:33:482024-03-06 18:33:48,128 - synapse.app.homeserver - 179 - INFO - sentinel - Running Mar 06 19:33:482024-03-06 18:33:48,129 - synapse.app.homeserver - 36 - INFO - sentinel - Set file limit to: 1048576 Mar 06 19:33:482024-03-06 18:33:48,133 - synapse.handlers.deactivate_account - 240 - INFO - user_parter_loop-0 - Starting user parter Mar 06 19:33:482024-03-06 18:33:48,158 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server Mar 06 19:33:482024-03-06 18:33:48,158 - synapse.app._base - 273 - CRITICAL - sentinel - Error during startup Mar 06 19:33:48Traceback (most recent call last): Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:33:48await p.load_metadata() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:33:48return await self._provider_metadata.get() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:33:48return await self._cachedcall.get() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:33:48self._result.raiseException() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:33:48raise self.value.with_traceback(self.tb) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:33:48result = context.run( Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 19:33:48return g.throw(self.value.with_traceback(self.tb)) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:33:48return await f() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:33:48metadata_response = await self._http_client.get_json(url) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:33:48body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:33:48response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:33:48response = await make_deferred_yieldable(request_deferred) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:33:48current.result = callback( # type: ignore[misc] Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:33:48raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 19:33:48synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 19:33:48 Mar 06 19:33:48The above exception was the direct cause of the following exception: Mar 06 19:33:48 Mar 06 19:33:48Traceback (most recent call last): Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:33:48await cb(*args, **kwargs) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:33:48await oidc.load_metadata() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:33:48raise Exception( Mar 06 19:33:48Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:33:48Error during startup: Mar 06 19:33:48Traceback (most recent call last): Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 147, in load_metadata Mar 06 19:33:48await p.load_metadata() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 595, in load_metadata Mar 06 19:33:48return await self._provider_metadata.get() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 143, in get Mar 06 19:33:48return await self._cachedcall.get() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 113, in get Mar 06 19:33:48self._result.raiseException() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 505, in raiseException Mar 06 19:33:48raise self.value.with_traceback(self.tb) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks Mar 06 19:33:48result = context.run( Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator Mar 06 19:33:48return g.throw(self.value.with_traceback(self.tb)) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 133, in _wrapper Mar 06 19:33:48return await f() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 607, in _load_metadata Mar 06 19:33:48metadata_response = await self._http_client.get_json(url) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 564, in get_json Mar 06 19:33:48body = await self.get_raw(uri, args, headers=actual_headers) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 649, in get_raw Mar 06 19:33:48response = await self.request("GET", uri, headers=Headers(actual_headers)) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 416, in request Mar 06 19:33:48response = await make_deferred_yieldable(request_deferred) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/twisted/internet/defer.py", line 1078, in _runCallbacks Mar 06 19:33:48current.result = callback( # type: ignore[misc] Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/http/client.py", line 957, in _timeout_to_request_timed_out_error Mar 06 19:33:48raise RequestTimedOutError("Timeout waiting for response from remote server") Mar 06 19:33:48synapse.http.RequestTimedOutError: 504: Timeout waiting for response from remote server Mar 06 19:33:48 Mar 06 19:33:48The above exception was the direct cause of the following exception: Mar 06 19:33:48 Mar 06 19:33:48Traceback (most recent call last): Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/_base.py", line 258, in wrapper Mar 06 19:33:48await cb(*args, **kwargs) Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/app/homeserver.py", line 368, in start Mar 06 19:33:48await oidc.load_metadata() Mar 06 19:33:48File "/app/code/env/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 151, in load_metadata Mar 06 19:33:48raise Exception( Mar 06 19:33:48Exception: Error while initialising OIDC provider 'oidc-cloudron' Mar 06 19:33:57=> Healtheck error: Error: Timeout of 7000ms exceeded Mar 06 19:34:03=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:34:13=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:34:23=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:34:33=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:34:43=> Healtheck error: Error: connect EHOSTUNREACH 172.18.17.56:8008 Mar 06 19:34:48==> Ensure we log to console Mar 06 19:34:48==> Configuring synapse Mar 06 19:34:48==> Configuring OIDC auth Mar 06 19:34:48==> Fixing permissions Mar 06 19:34:48==> Starting synapse -
@rompo said in Matrix "Not responding" after automatic Update from v1.82.0 to v1.83.0:
Mar 06 19:32:452024-03-06 18:32:45,963 - synapse.http.client - 428 - INFO - sentinel - Error sending request to GET https://my.myurl.cloud/openid/.well-known/openid-configuration: RequestTimedOutError 504: Timeout waiting for response from remote server
Thanks, the above is the root cause. The app is not able to CURL itself for some reason.
- Are you behind Cloudflare?
- Open a web terminal (of any app) and then
curl https://my.myurl.cloud/openid/.well-known/openid-configuration. We need to find out why this is not working.
No, I'm not behind Cloudflaire.
curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. -
@rompo you to check why that curl command does not work on your system. If the certificate is valid then it's a DNS related issue. Please check what the DNS of my.myurl.cloud is resolving to.
The DNS resolves correctly from my point of view.
my.myurl.cloud is also accessible from the Internet as well as from the internal network and I can log in to it.Unreachable ports cannot be a problem either.
However, if I want to access the web interface via the internal IP, I get the following message: You are seeing this page because the DNS record of 172.27.212.20 is set to this server's IP but Cloudron has no app configured for this domain.
-
@rompo you cannot access Cloudron pages by IP address, it has to be by domain name entirely.
Can you try to resolve
host my.myurl.cloudfrom the web terminal? Does it give internal IP or external IP ? If it is the external IP, does your network support hairpinning - the ability to access internal Cloudron when using external IP ? See https://docs.cloudron.io/troubleshooting/#hairpin-nat -
@rompo you cannot access Cloudron pages by IP address, it has to be by domain name entirely.
Can you try to resolve
host my.myurl.cloudfrom the web terminal? Does it give internal IP or external IP ? If it is the external IP, does your network support hairpinning - the ability to access internal Cloudron when using external IP ? See https://docs.cloudron.io/troubleshooting/#hairpin-nat@girish
I receive the public IP as output.With pfSense, there is a "host override" for all Cloudron URLs, which actually resolves correctly (internal 172 addresses within the network).
However, I have now activated NAT Reflection with everything that goes with it as a test, which does not result in any change for Cloudron. Within the 172 network, however, publicIP is now resolved and routed correctly.
However, it is not entirely clear to me why Cloudron outputs the public IP in the original configuration with the "Host Override" entries, because the DNS server address is actually assigned via DHCP.
How can I check this? I can't get to the required interface with ifconfig. -
@rompo Looks like we get closer. Cloudron has it's own internal recursive DNS resolver. This means that it might be bypassing pfSense DNS. Please see https://docs.cloudron.io/networking/#internal-dns-server to make Cloudron use the internal DNS. This way it will resolve DNS to internal network values .
-
@rompo Looks like we get closer. Cloudron has it's own internal recursive DNS resolver. This means that it might be bypassing pfSense DNS. Please see https://docs.cloudron.io/networking/#internal-dns-server to make Cloudron use the internal DNS. This way it will resolve DNS to internal network values .
@girish
Thanks for the tip, I have entered pfSense as DNS, now it works again.At the end of last week I set up a test server to play with which is connected to a different firewall. I had exactly the same problem with this fresh installation.
Another problem with the new login procedure is that mobile devices with the policy of not allowing cookies can no longer log in to the Element app.
An alternative solution must now be found, possibly deactivating openID if this is possible, but that is another topic.Many thanks for your help and have a good week!
For all those who need a short summary:
Log in via SSH
Create the file "custom.conf":sudo touch /etc/unbound/unbound.conf.d/custom.confEdit file:
sudo nano /etc/unbound/unbound.conf.d/custom.confInsert content into the file:
# this disables DNSSEC server: val-permissive-mode: yes # forward all queries to the internal DNS forward-zone: name: "." forward-addr: 172.xxx.xxx.xxxRestart:
sudo systemctl restart unboundCheck:
sudo systemctl status unbound
