SOLVED matrix.org (communication)

nebulon

Run a matrix home server in the cloudron, that also can join in a federated decentralized network of many servers. Can be bridged with a lot of stuff, including IRC. Text/audio/video chats, groups, etc.

doodlemania2

Initial attempt has been started:
https://git.cloudron.io/msbt/matrix-riot-app
Have deployed locally (not app store) - works great. Just need to package an additional app to do the front end (Riot).

msbt

Just made another update, a few things weren't working properly. Everything should run a bit smoother now, but it's still a work in progress (I want it to be a dedicated identity server as well).

Not sure if it's necessary to pack the client itself, but if you want, it was already included in the version that I forked my app from (https://git.cloudron.io/joey/matrix-riot-app), maybe you could just grab the riot part and make a new app out of it.

Cheers, M

yusf

@msbt Your version works great so far. Do you plan to add the Riot client as well? I’m happy with just the server part but I believe I read somewhere that Cloudron Web Store apps are required a proper front-end to be published.

msbt

hey @yusf, glad to hear that, although I should be updating more often, there has been quite a few updates since I packed it. They say it's not good security-wise to put the client in the same place as the server, that's why I actually removed it from the initial setup that I forked. But I just updated the riot-app itself, you can check it out here: https://git.cloudron.io/msbt/riot-app

It will try to use $APP_DOMAIN as a host, which is exactly what you're not supposed to do (and won't work with the cloudron setup), so you need to edit the config file after the installation.

yusf

Sorry for eons time until replying. You’re right, I’ve heard about that vulnerability myself now that I think on it.

Have you considered adding mxisd? The project seem to add complete LDAP support without too much hassle which would make the package an ideal Cloudron app. https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/ldap.md

msbt

I think I did check it out at some point, but the package as it is now already supports ldap login. I also wanted to add an identity server, but ran out of time to test it back when I built it.

yusf

already supports ldap login

Oh, that's great! I'm trying to install the latest app version with cloudron build and cloudron install but now I get:

ERROR No image found, please run `cloudron build` first or specify a `dockerImage` in the CloudronManifest [ /usr/local/lib/node_modules/cloudron/src/helper.js:68:29 ]

girish

Looks like cloudron build was not run. Can you also make sure you are using the latest cloudron CLI? cloudron --version will say 1.7.0.

yusf

@girish But I did run build successfully:

Build succeeded
Success

Version is 1.7.0, yes.

girish

Can you try deleting ~/.cloudron.json and try again? There was a format change about 2-3 months ago and maybe the json file is in the old format/corrupt.

yusf

That solved my problem, thank you.

yusf

This post is deleted!

yusf

@msbt Care to update the Riot package?

msbt

hey @yusf, sure thing, will check it out tomorrow. I haven't used it in a while because I changed to the electron desktop client, but I'll see if it's an easy update and let you know.

msbt

ok this was a quick fix, had a few minutes to do it, here you go:
https://git.cloudron.io/msbt/riot-app
please compare your config to the example one, not that you miss anything, but I think it's good to go like that.

yusf

Thanks! There are some bugs with the package however. Users have to login twice to get inside.

Also, the Security & Privacy section won't open at all.

I've carefully compared my riot_config.json with the new one and adjusted accordingly (almost no diff).

msbt

which version of matrix are you using?

msbt

@yusf I can't confirm that. I deployed it, replaced all occurences of the riot domain to the matrix/synapse one and it works right out of the box (using username to login). Are you on the latest matrix build? Might push another update in a bit as they released 0.99.2rc1 earlier today, this means the 0.99.2 should be out soon.

yusf

@msbt Dang, I forgot to push an update of matrix, sorry. After I did, things started working more proper.

Security & Privacy section of room settings still won't open but I’ve pinned that issue down to a specific room, the premade ”discuss” room, so let’s put that blame on Riot.

Thank you for your help and advice. Is matrix/riot coming to the app store some day?

msbt

that I don't know, it would require a lot more testing since I've just forked and updated someone else's repo for my own needs. 0.99.2 is out, maybe I'll do a rebuild when 1.0 is ready and add all features that are required/available.

yusf

I'd be interested in an integrations server app for Cloudron, built on dimension.

yusf

@msbt Would you bump the version please? v1.0.5 is the latest release.

msbt

sure thing @yusf, there you go: https://git.cloudron.io/msbt/riot-app

yusf

What’s the process of getting this to the App Store?

msbt

well, riot alone won't do much good. as for matrix: I would want to wait for v1.0 and redo the repo (v0.99.3rc1 got released today).

msbt

@yusf before you ask, I've just pushed the 1.0.6 update

yusf

@msbt Lol, I was just going to!

msbt

Matrix is also updated to v0.99.3: https://git.cloudron.io/msbt/matrix-riot-app

yusf

Btw, there's an inofficial Matrix channel at #cloudron:matrix.org.

yusf

So since matrix.org’s development infrastructure got megaowned today i figured I’d put this here:

After Matrix has restored its major services, they noticed that the GPG keys used for signing packages where compromised.

The key IDs are:

AD0592FE47F0DF61 (synapse)
E019645248E8F4A1 (Riot/Web)

Please make sure to no longer use those keys.

yusf

@msbt Errors installing Matrix app anew:

sed: can't read homeserver.yaml: No such file or directory

When looking in /app/data only nginx and synapse folders show, and they are both empty.

Please look into it.

msbt

hey @yusf
I've pushed another update for the latest riot and matrix apps. I did not set up email and ignored the turn settings too, since quite a few things changed since they're heading towards v1.0 and I don't have too much time on my hands. But I tested it and login should work for Ldap users. Thanks for the info, didn't realize it wasn't working on fresh setups.

yusf

Thanks so much, will try soon.

yusf

Works good so far. Email is not important in this case but TURN is, so I’m eagerly awaiting that feature.

Thank you for fixing it.

I just reread an earlier post of yours. From this version of synapse an onwards, will there be forward compatibility? It’s my bad I didn’t read it properly and of course my fault for installing unfinished packages but since I’m pretty screwed if I can’t upgrade this package later I want to inquire if that’s the case and possibly beg you to keep future versions compatible with this version. I’d help with testing if that

yusf

Idea: Put index.html in /app/data so that it's editable.

yusf

Building app I get deprecation warnings:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Is that on app or box level?

msbt

This message is on app-level but valid nonetheless. In a week (10th of June) there should be the release of v1.0 and I'll do a rebuild of the container with python 3 and hopefully audio/video support. I'll keep you posted.

yusf

That is great, thank you!

msbt

fyi, v1.0.0 got released earlier and I'm working on it. Everything is looking good so far, the only things that aren't working at the moment are videocalls (audio is working) and the integration server, not sure why. Will keep you posted and eventually push the "final" version so people can test it.

yusf

That’s exciting. Your work is much appreciated, @msbt .

msbt

alright alright alright, v1.0.0 is (mostly) working, as far as I can see only the videochat feature is missing, I'll further investigate when I have the time. Other than that it seems to be looking good, changed to python3, registration including email & activation is working, url preview (which wasn't always before), introduced a new healthcheck page.

Grab it from here and let me know if you encounter any other issues. Not sure if upgrading from older versions work flawlessly, since quite a few config items got introduced and some are required. So if it doesn't work from scratch, install a fresh one on another domain and compare with your current one before upgrading.

Possible things that might need changing if you want the features (depending on the the version you first installed it) in homeserver.yaml:

enable_notifs: true
comment #template_dir
require_transport_security: false
comment #riot_base_url
add public_base_url: https://yourmatrixserver.com
change case url_preview_enabled: true
comment '172.18.0.0/12' in url_preview_ip_range_blacklist

The Riot app also got pushed to 1.2.2, available here as usual.

msbt

ok this is embarrassing, videochat was working all along, only my strict windows settings prohibited me from using it

yusf

Nice work! From a glance I can't see anything breaking when upgrading the Matrix package.

Is federation working for you though?

Log.

technotame

Thanks for working on this! I've tested it out and it works great so far, both Matrix and Riot.

One issue I see, and this may just be me not being very familiar with Matrix, is that when I go to search the room directory for Matrix.org, I get Riot failed to get the public room list. Internal server error. I don't know if I am supposed to be able to search and connect to Matrix.org rooms, but I thought I should be able to. Is this a bug or just me?

Thanks again!

msbt

Ah nice catch, I haven't tested federation because it used to work. This could be for a number of reasons, either new regular homeserver settings, nginx config or cors related issues. I'll try to narrow it down, thanks for reporting!

murgero

@msbt I can test if there is a need for that.

murgero

@msbt I noticed this app does not actually have Riot front end? Also is the identity server implemented?

Edit: for the federation errors, here is my relevant log line(s):

2019-06-21 19:25:33,552 - synapse.http.matrixfederationclient - 433 - INFO - POST-317 - {GET-O-4} [matrix.org] Got response headers: 401 Unauthorized
2019-06-21 19:25:33,553 - synapse.http.matrixfederationclient - 517 - WARNING - POST-317 - {GET-O-4} [matrix.org] Request failed: GET matrix://matrix.org/_matrix/federation/v1/publicRooms?include_all_networks=true&limit=20: HttpResponseException("401: b'Unauthorized'",)
2019-06-21 19:25:33,554 - synapse.http.server - 112 - ERROR - POST-317 - Failed handle request via 'PublicRoomListRestServlet': <XForwardedForRequest at 0x7f13e8464ba8 method='POST' uri='/_matrix/client/r0/publicRooms?server=matrix.org' clientproto='HTTP/1.0' site=8008>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/synapse/http/server.py", line 81, in wrapped_request_handler
    yield h(self, request)
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/synapse/http/server.py", line 316, in _async_render
    callback_return = yield callback(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/synapse/rest/client/v1/room.py", line 387, in on_POST
    third_party_instance_id=third_party_instance_id,
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/synapse/handlers/room_list.py", line 467, in get_remote_public_room_list
    third_party_instance_id=third_party_instance_id,
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/synapse/federation/transport/client.py", line 348, in get_public_rooms
    ignore_backoff=True,
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/synapse/http/matrixfederationclient.py", line 760, in get_json
    timeout=timeout,
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/dist-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/dist-packages/synapse/http/matrixfederationclient.py", line 248, in _send_request_with_optional_trailing_slash
    request, **send_request_args
  File "/usr/local/lib/python3.6/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.6/dist-packages/synapse/http/matrixfederationclient.py", line 472, in _send_request
    raise e
synapse.api.errors.HttpResponseException: 401: b'Unauthorized'

msbt

@murgero yes, the riot frontend is here: https://git.cloudron.io/msbt/riot-app - the devs suggested not to run both on the same machine, splitting the apps was the way to go. The identity server is also not implemented (yet), so the registration uses vector.im at the moment.

Thanks for your input, I'll review the changes and try to figure out the point where the federation stopped working.

yusf

Also check my log records in https://forum.cloudron.io/post/3817

murgero

Any news on federation?

~~Edit: I know the devs for Cloudron recommended that these apps be separated, however I believe they would be better together (maybe a 3rd app that includes both?) because:~~

~~Users can get confused on the domains (logging into riot.example.com, but user is matrix.example.com)~~
~~Having them in the same place can allow for better troubleshooting, and in some cases, is just more convenient.~~
~~Easier to update both at the same time, then to have to rebuild the app twice.~~

msbt

This is no recommendation from cloudron, but from the riot-devs themselves: https://github.com/vector-im/riot-web#important-security-note

I didn't have time to look further into the federation issue, maybe I'll find some time this weekend, sorry for the delay.

murgero

@msbt Ah my bad on misinterpreting that. and No worries. This is awesome work and worth the wait!

yusf

Recently learned that federation is needed for integrations to work so it's important for any integrations as well.

msbt

Weirdly enough, integrations are working, but federation is not... Is it possible, that the requests are being denied by cloudron and not the app @nebulon @girish? I remember we had the embedding feature which got removed and will eventually be replaced with CSP. I've tried several things, but I don't really know how to fix it... I'll jump on the matrix network and ask there if they have a clue.

girish

@msbt Is there anything in the browser console? CSP/X-Frame related issue will be printed in browser console.

msbt

federation is working, thanks to the help of the synapse admins and community! Please grab the latest version from here and let me know if it also works for you. You might need to adjust the homeserver.yaml again, probably best if you install a fresh one and compare the config. There might be some finetuning required for preview and such, but since I'm on vacation, that's a topic for another day

murgero

@msbt That's weird, I installed a fresh Matrix server, still getting error 500 XHR requests when trying to federate to another room off-server. (though I can list rooms now???)

murgero

Alright so federation is working but some rooms don't (maybe they time out because a bridge is down??)

yusf

@msbt Many thanks! Can’t wait to try it.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

SOLVED matrix.org (communication)