Suddenly Matrix federation doesn't work anymore :(

luckow

Symptom is: no new messages in channels from other instances & no search results.

In the logfiles I found

synapse.http.matrixfederationclient - 503 - INFO - POST-6883 - {GET-O-74} [matrix.org] Got response headers: 401 Unauthorized

synapse.http.matrixfederationclient - 580 - WARNING - POST-6883 - {GET-O-74} [matrix.org] Request failed: GET matrix://matrix.org/_matrix/federation/v1/publicRooms?include_all_networks=false&limit=20: HttpResponseException('401: Unauthorized')

synapse.http.server - 86 - INFO - POST-6883 - <XForwardedForRequest at 0x7fe7244a3160 method='POST' uri='/_matrix/client/r0/publicRooms?server=matrix.org' clientproto='HTTP/1.1' site='8008'> SynapseError: 401 - Failed to find any key to satisfy VerifyJsonRequest(server=example.org, key_ids=['1234567:abcde'], min_valid=000000000)

I've tried the curl command from the Cloudron docs
$ curl https://example.com/.well-known/matrix/server
Instead of { "m.server": "matrix-homeserver.example.com:443" } I got the following response:

<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

BTW: same for the cloudron.io domain.

girish

@luckow I have fixed this in https://git.cloudron.io/cloudron/box/-/commit/2f58092af2344a257a05ab31f773ebbfb558eb4f

It's a small change if you want to fix it yourself. The file is /home/yellowtent/box/src/nginxconfig.ejs. Just have to add those two lines in the commit above. Then go to Location -> Save to regenerate the nginx config. No need to restart box code.

yusf

@luckow Does it work with a default homeserver.yaml? Depending on when you installed it some setting might've not been carried over to the new version. This is the first thing I try when updates break an app.

luckow

@yusf I've installed a fresh new matrix app to a new domain. Same behavior. Maybe something is broken in the nginx configuration. (because of the wrong result in curl)

nebulon

Can you verify that the settings for the domain are correct: https://docs.cloudron.io/domains/#matrix-server-location

luckow

@nebulon settings are correct. I've rebooted the whole server instance. Nothing changed
The feedback of the federation tester (https://federationtester.matrix.org) is

Connection Errors
Get "https://IP.EXAMPLE.ORG:8448/_matrix/key/v2/server": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

And with curl https://example.com/.well-known/matrix/server I've got the same answer from the cloudron.io domain that my domain gives me in return. But the feedback of the federation tester is different if i use cloudron.io as the test domain.

Any other ideas?

nebulon

Ok, so I've did some testing, is it possible that you only have a wildcard DNS record for your domain? Apparently matrix federation or at least the tester needs a distinct DNS record to work.
If that is the case, setup an A record for example.com pointing to your IP and after DNS propagation the federation tester should succeed.

luckow

@nebulon good guess. You're right. Normally I use *.example.org for redirecting everything to my Cloudron. Ok. I've added an exclusive sudomain for my matrix-domain. Now I'm waiting for the DNS propagation. Let's see.

girish

@luckow said in Suddenly Matrix federation doesn't work anymore :

BTW: same for the cloudron.io domain.

$ curl -L https://cloudron.io/.well-known/matrix/server
{ "m.server": "matrix.cloudron.io:443" }

-L follows redirects.

girish

@luckow *.example.com does not cover example.com in DNS. It only covers the subdomains . Also, you need to have an app installed on example.com or alternately add it as a redirect of an existing app.

luckow

@girish Ok. got it. From my side it was a misunderstanding. It's not a topic about "*.example.org is not enough" and to make federation work it needs to have an exclusive matrix-homeserver.example.org (btw: that makes no sense in my poor expertise about dns stuff). It the hint, that it needs a dns entry for example.org to get federation working. But: yes I have such DNS record.

luckow

I found the problem for my problem.

example.org is handled by the surfer app. I've switched to www.example.org this week and redirected example.org to www.example.org. That was the moment matrix federation stops working. Switching back to example.org solves the problem.

Maybe there is a bug in the surfer app @nebulon ?
Goal is to have www.example.org because of https://www.yes-www.org/why-use-www/

luckow

@girish should we add the -L option to the docs? https://docs.cloudron.io/apps/synapse/#step-23-verify-setup

girish

@luckow Looks like there is a bug when the bare domain is a redirect. Investigating.

girish

@luckow I have fixed this in https://git.cloudron.io/cloudron/box/-/commit/2f58092af2344a257a05ab31f773ebbfb558eb4f

It's a small change if you want to fix it yourself. The file is /home/yellowtent/box/src/nginxconfig.ejs. Just have to add those two lines in the commit above. Then go to Location -> Save to regenerate the nginx config. No need to restart box code.

luckow

@girish top!

Got 1 connection report.
Homeserver version: Synapse 1.27.0

atridad

Sorry to revive this but I had a question regarding this. It seems like Element can no longer connect when I do this. Also, I noticed the same sort of setup is here for Mastodon: https://docs.cloudron.io/domains/#matrix-server-location

So is it then possible to have the same base domain for both? I want to have matrix.domain.com and social.domain.com. Just a little confused.

@girish

nebulon

@atrilahiji so the apps can be installed on any domain really but for the user/channel handles to work in federation, the base domain (in your example domain.com) needs to provide information where to find the backend servers. That information is stored in a well known location.

We've just added those cases in the domain configuration directly, to avoid users having to edit text files in specific URL paths, which can be error prone.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Suddenly Matrix federation doesn't work anymore :(