Solved Meet Kopano (unstable version) - a few questions & problems :)

[fbartels]

Hi @avatar1024,

happy to answer your questions.

1. that is actually the default in WebRTC communications. Connections are then always peer to peer and also end to end encrypted. What this often weakens in other projects is the usage of SFU or MCU services to lessen the upload for the individual. but this also means that then there will be a central instance that needs to receive and decrypt the user connections (the same can be said to all services offering server side call recording or sip gateways, these features simply cannot work with e2ee).
2. WebRTC is best supported in Chromium browsers. Firefox is mostly fine as well. Safari is the modern day Internet Explorer and Apple is slow to pick up and fix their browser for modern technology that enables users to escape the walled garden.
3. There is no hard limit on Meet or its components itself. Since calls are peer to peer there is hardly an impact on the server for an additional connection (just another websocket connections). The only server side limitation could be turn (for the case where it needs to relay connections) and this limitation is mostly bandwidth. Backend services are done in golang and are able to scale to thousands of connections on ordinary hardware.
4. like I said, no real requirements server side. but since its all peer to peer and every participants needs to be connected to every other will likely run into user side limitations (encoding all streams, connection limits, bandwidth) on the clients if you really do calls with 10+ users.
5. video is vp8 (because it supported on all browsers and easy on the cpu) and audio is opus. screensharing is vp9 since it requires less bandwidth for needed higher resolution (compared to caller video).

[avatar1024]

@fbartels Many thanks for your reply. That's very useful and makes it clear very.

Since what you are describing is a full mesh p2p network typology, is there any optimisation implemented by Kopano Meet (compared to others) to limit uplink bandwidth requirement on each peer to improve the performance and scalability?
As far as I understand it is one of the key limitation of doing things this way (compared to going through and SFU) as a trade off for being able to implement proper E2EE.
You mention VP8 as being easy on the CPU (but also more bandwidth). Do you therefore think that limitation on the user is likely to come more from CPU rather than bandwidth (comparing increase CPU power vs increase in broadband/mobile bandwidth)?

[fbartels]

@avatar1024 said in Meet Kopano (unstable version) - a few questions & problems :

is there any optimisation implemented by Kopano Meet (compared to others) to limit uplink bandwidth requirement on each peer

You cannot generate more available bandwidth through software

Generally speaking connections are limited to 1Mbit each (for video and audio). If there is not enough bandwidth available the browser itself will limit the data transmitted (reduce video quality) until the point where video will be deactivated completely.

@avatar1024 said in Meet Kopano (unstable version) - a few questions & problems :

You mention VP8 as being easy on the CPU (but also more bandwidth).

On of the reasons its so light on cpu is because hardware acceleration does widely exist for it, btw. But yes I think this is a fair statement. In the end you could run into situations where client would be unable to handle all the incoming and outgoing streams. There is a special "audio only" mode in Meet that could kind of help in these situations, as it turn off all outgoing and incoming video (incl. screensharing) and therefore would lessen the load of encoding the video.

[scooke]

This post is deleted!

[yusf]

Since #1 was resolved the guest access feature should work now, right? I've edited `kwmserver.cfg

Enable guest access by default

[yusf]

Enable guest access by default 🎉

But when is v1.2.0 coming?

[nebulon]

The update package with guest support is released now

[luckow]

The new update works fine. I had my first guest meeting

[luckow]

@luckow If you want to have a meeting with iOS users, tell them that they (must) use Safari. There is no app for meet kopano. Don't use Firefox or other browsers on iOS. Safari is the tool of choice. But this is not a meet kopano issue.

[yusf]

I seem to be doing something wrong with guest-enabled rooms. Guests are still prompted to sign in. Are there any gotchas I'm missing?

My fault, I shared my own room URL rather than using the "Share Group link" feature. Maybe a shared room URL should redirect to the guest onboarding as well, since that still contains a link to the login flow?

[jdaviescoates]

@luckow doesn't seem to work very well at all for me

Have tried multiple times now and so far failed to have a successful call.

I've mostly been trying on Firefox on Ubuntu, but have also tried Chromium with very similar results. Shall have to investigate further... (where shall I look for error logs, I wonder...)

[fbartels]

@jdaviescoates said in Meet Kopano (unstable version) - a few questions & problems :

where shall I look for error logs, I wonder..

Yes, that is indeed the drawback with mostly client side applications, server side logs will not really help you. Hints on what could be going wrong can instead be found the the browser console. chrome://webrtc-internals/ can be interesting as well (depending on the concrete error).

What is the exact behaviour you experience? The Cloudron turn is running on port 3478, could that port be blocked for either you or the other user in your call?

[yusf]

I've also performed a realistic test scenario now and it's very spotty. With three people on different networks in different locations (though geographically pretty close) I managed to get video from one other participant but not audio, and neither video nor audio from the other one.

I tried both Firefox (desktop) and Mobile Safari. Calls are pretty flawless with the same setup on the same LAN (though not the exact same peer devices).

[fbartels]

@yusf which version of iOS are you on?

Video but no audio is strange, since both are transmitted in the same connection.

[yusf]

@fbartels Latest stable, 13.3.1.

[fbartels]

@yusf the latest is actually 13.4.1

[jdaviescoates]

@fbartels said in Meet Kopano (unstable version) - a few questions & problems :

What is the exact behaviour you experience?

I've done a few tests.

First of all I tried having a call with myself using my laptop on wifi (Lenovo T510 running Ubuntu 18.04 and Firefox 75.0) and my phone on 4g (Wileyfox Swift 2 Plus running Android 8.1.0 and Firefox 68.7.0) but it didn't work at all. No video nor audio.

Not only that but after a while on my laptop Firefox seemed to suddenly be unable to connect to my camera and mic and then would fail to reconnect (this doesn't happen when testing Nextcloud Talk). And then when I clicked the screenshare button (just to see what it was) my whole laptop froze and I had to power off.

Although saying all that, I just replicated the same test and this time it appeared to work quite well. I even got my wife to join in on her phone too and it kinda sorta worked (she lost connection at one point and mostly her video was frozen).

Another time I did a test with my laptop, my phone, and my Mum on her laptop (in a different city). I can't remember exactly what happened, but I think I could see and her my Mum on my phone, but not on my laptop. But she could see me on my laptop but not on my phone, or something like that.

So all in all, in the all tests I've done to date, mostly not really working (although the test I just did now went OK until there were 3 of us on the call)

I've had much better experience using Nextcloud Talk so far, although I have to say I much prefer that way Kopano Meet gets guests to set their name and get their camera and mic working before joining the call. On Nextcloud Talk it's common to have multiple people join and for them to all be called Guest, which can get a bit confusing. I think I'll add an issue to Nextcloud Talk suggesting they make Guests add their name before joining too...

[fbartels]

Hi @jdaviescoates,

that all in all sounds strange. Especially freezing when trying to screenshare is nothing I have seen here so far.

[nebulon]

I remember @fbartels mentioning some browser session debugging tool to collect relevant information in such a case, maybe that can help sort our some of those cases. Overall we had generally a good experience so far with the app, but given all the various browsers and their various states of webrtc it seems to be a hard issue to tackle.

[oj]

Hi @fbartels,

Just had my first few one-on-one video calls with registered users and guests... Iridium browser at my end and Chrome at the other end. Worked flawlessly...even with 4G at my end. Excellent user experience at both ends. Thanks for bringing Kopano Meet to Cloudron!

[NCKNE]

Hello @fbartels,

I just installed Kopano Meet to give it a try and encountered the following error after installation on cloudron v5.1.4 with Kopano Meet version 2.1.0 (Package v1.2.1):

Apr 15 17:16:33 2020/04/15 15:16:33 Waiting for https://meet.xxx.xx/.well-known/openid-configuration: unexpected HTTP status code: 404.
Apr 15 17:16:33 2020/04/15 15:16:33 Failed to wait: timed out: https://meet.xxx.xx/.well-known/openid-configuration.
Apr 15 17:16:33 time="2020-04-15T15:16:33Z" level=info msg="serve start"
Apr 15 17:16:33 time="2020-04-15T15:16:33Z" level=info msg="using external TURN service: https://turnauth.kopano.com/turnserverauth/"
Apr 15 17:16:33 time="2020-04-15T15:16:33Z" level=info msg="serve started"
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=warning msg="failed to initialize OIDC provider" error="Timeout (:0x105)" iss="https://meet.xxx.xx"
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=warning msg="admin: using random admin tokens singing key - API endpoint admin disabled"
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=info msg="pattern ^group/public/.* public guest rooms enabled" manager=guest
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=info msg="guest: API endpoint enabled"
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=info msg="rtm: API endpoint enabled"
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=info msg="starting http listener" listenAddr="127.0.0.1:8778"
Apr 15 17:16:43 time="2020-04-15T15:16:43Z" level=info msg="ready to handle requests"

The app starts, but no login is possible (the login button on the Kopano Meet page does not do anything upon clicking).

On the client side, the following error comes up:

oidc-client.min.js:718 GET https://meet.xxx.xx/.well-known/openid-configuration 404
(anonymous) @ oidc-client.min.js:718
(anonymous) @ oidc-client.min.js:693
(anonymous) @ oidc-client.min.js:289
value @ usermanager.js:317
(anonymous) @ actions.js:667
w @ runtime.js:62
(anonymous) @ runtime.js:288
(anonymous) @ runtime.js:114
w @ runtime.js:62
e @ runtime.js:152
(anonymous) @ runtime.js:187
o @ runtime.js:186
(anonymous) @ runtime.js:209
(anonymous) @ runtime.js:114
(anonymous) @ runtime.js:233
(anonymous) @ actions.js:658
Promise.then (async)
(anonymous) @ actions.js:656
(anonymous) @ index.js:8
dispatch @ applyMiddleware.js:46
(anonymous) @ actions.js:318
w @ runtime.js:62
(anonymous) @ runtime.js:288
(anonymous) @ runtime.js:114
w @ runtime.js:62
e @ runtime.js:152
(anonymous) @ runtime.js:187
o @ runtime.js:186
(anonymous) @ runtime.js:209
(anonymous) @ runtime.js:114
(anonymous) @ runtime.js:233
(anonymous) @ actions.js:309
(anonymous) @ index.js:8
dispatch @ applyMiddleware.js:46
(anonymous) @ actions.js:356
w @ runtime.js:62
(anonymous) @ runtime.js:288
(anonymous) @ runtime.js:114
w @ runtime.js:62
e @ runtime.js:152
(anonymous) @ runtime.js:162
Promise.then (async)
e @ runtime.js:161
(anonymous) @ runtime.js:187
o @ runtime.js:186
(anonymous) @ runtime.js:209
(anonymous) @ runtime.js:114
(anonymous) @ runtime.js:233
(anonymous) @ actions.js:316
(anonymous) @ index.js:8
dispatch @ applyMiddleware.js:46
n @ actions.js:216
(anonymous) @ actions.js:232
w @ runtime.js:62
(anonymous) @ runtime.js:288
(anonymous) @ runtime.js:114
w @ runtime.js:62
e @ runtime.js:152
(anonymous) @ runtime.js:187
o @ runtime.js:186
(anonymous) @ runtime.js:209
(anonymous) @ runtime.js:114
(anonymous) @ runtime.js:233
(anonymous) @ actions.js:224
(anonymous) @ AuthenticatedRoute.js:22
Promise.then (async)
render @ AuthenticatedRoute.js:20
(anonymous) @ react-router.js:444
Li @ react-dom.production.min.js:4074
qa @ react-dom.production.min.js:5514
Ka @ react-dom.production.min.js:5536
Ds @ react-dom.production.min.js:5958
Ms @ react-dom.production.min.js:5925
Es @ react-dom.production.min.js:5860
es @ react-dom.production.min.js:5787
enqueueSetState @ react-dom.production.min.js:2790
(anonymous) @ react.production.min.js:72
(anonymous) @ Main.js:56
u @ runtime.js:45
(anonymous) @ runtime.js:264
(anonymous) @ runtime.js:98
r @ asyncToGenerator.js:3
s @ asyncToGenerator.js:25
Promise.then (async)
r @ asyncToGenerator.js:13
s @ asyncToGenerator.js:25
(anonymous) @ asyncToGenerator.js:32
(anonymous) @ asyncToGenerator.js:21
Promise.then (async)
value @ Main.js:53
Ba @ react-dom.production.min.js:4980
Ga @ react-dom.production.min.js:5123
(anonymous) @ react-dom.production.min.js:5975
(anonymous) @ scheduler.production.min.js:274
Ls @ react-dom.production.min.js:5974
Ds @ react-dom.production.min.js:5958
Ms @ react-dom.production.min.js:5925
Es @ react-dom.production.min.js:5860
es @ react-dom.production.min.js:5787
enqueueSetState @ react-dom.production.min.js:2790
(anonymous) @ react.production.min.js:72
t @ index.js:240
(anonymous) @ index.js:250
Promise.then (async)
(anonymous) @ index.js:249
(anonymous) @ index.js:197
lo @ react-dom.production.min.js:2854
Ri @ react-dom.production.min.js:3759
Li @ react-dom.production.min.js:3960
qa @ react-dom.production.min.js:5514
Ka @ react-dom.production.min.js:5536
Ds @ react-dom.production.min.js:5958
Ms @ react-dom.production.min.js:5925
Es @ react-dom.production.min.js:5860
es @ react-dom.production.min.js:5787
enqueueSetState @ react-dom.production.min.js:2790
(anonymous) @ react.production.min.js:72
(anonymous) @ IntlContainer.js:281
Promise.then (async)
value @ IntlContainer.js:280
Ba @ react-dom.production.min.js:4978
Ga @ react-dom.production.min.js:5123
(anonymous) @ react-dom.production.min.js:5975
(anonymous) @ scheduler.production.min.js:274
Ls @ react-dom.production.min.js:5974
Ds @ react-dom.production.min.js:5958
Ms @ react-dom.production.min.js:5925
Es @ react-dom.production.min.js:5860
es @ react-dom.production.min.js:5787
Us @ react-dom.production.min.js:6077
Ys @ react-dom.production.min.js:6085
(anonymous) @ react-dom.production.min.js:6276
(anonymous) @ react-dom.production.min.js:6360
Is @ react-dom.production.min.js:6007
Vs @ react-dom.production.min.js:6359
render @ react-dom.production.min.js:6390
831 @ app.js:51
i @ bootstrap:89
Promise.then (async)
(anonymous) @ index.js:18
Promise.then (async)
184 @ index.js:14
i @ bootstrap:89
107 @ main.758e3f7e.chunk.js?__WB_REVISION__=ebc8e0d9096114f5d646:1
i @ bootstrap:89
r @ bootstrap:45
t @ bootstrap:32
(anonymous) @ main.758e3f7e.chunk.js?__WB_REVISION__=ebc8e0d9096114f5d646:1
actions.js:251 failed to continue config initialization Error:  (404)
    at XMLHttpRequest.<anonymous> (oidc-client.min.js:715)

Any idea what could be the problem?

[fbartels]

Hi @NCKNE,

thanks for your feedback. The immediate cause for your error is the 404 when getting the discovery document from the .well-known directory. Can you check if Konnect is running (the program that provides the document)? You can do so when opening a terminal inside of the app and then running supervisorctl status kopano-konnectd.

What I find especially weird is that you got a 404 (file not found), in case Konnect isn't running this should normally be a 503 (service unavailable). Which version of Cloudron are you on? Where is that machine running (behind dynamic dns)? How do you manage dns (could it be that meet.xxx.xx still points somewhere else)?

Edit: ah, this seems to be some general breakage with the latest version of Cloudron. https://forum.cloudron.io/post/7396 reports something similar and @girish said a new version is already rolling out.

[girish]

@fbartels Sorry, yes, my bad. In 5.1.4 I broke apps that serve up .well-known docs of their own

On a side note, we use Kopano meet for our weeklies and we haven't had any issues.

[NCKNE]

Thanks for the update @girish! The 5.1.5 update of cloudron fixes the error and Kopano Meet is working now.

[luckow]

Let us find out how many participants are possible with a Kopano Meet app on Cloudron (v5.15).

Invitation
When: Monday 20. April 21:00 (Europe/Berlin)
Where: https://meet.luckow.org/meet/r/join/group/public/cloudron-meetup

Please: use chromium/chrome & Headset

Specs: netcup vm, 6 CPU, 32 RAM, HDD

[fbartels]

@luckow thats a nice idea. I have created a reminder for myself.

We once had a call with around 30 participants, but that was not really stable and just a quick test.

We are currently working on a sfu for some customers, if there is interest we could do a comparing test afterwards with that kind of a setup.

[scooke]

@luckow I'll join in, thanks.

[stantropics]

@jdaviescoates Have you investigated further? I have also tried to a few meetings with 3 to 5 people with several problems. Sometimes people cannot connect at all, sometimes they are not getting any sound or their sound is not transmitted to other participants. Very hard to analyze ...

[jdaviescoates]

@stantropics no not had a chance (nor really sure how to), although I think I read somewhere that with full p2p WebRTC apps like Kopano Meet all users need to have enough bandwidth for all streams or something, is that right @fbartels?

Either way it seems most people have so far had a better experience than us... So I'm keen to try more. Will attempt to join the group call..

[girish]

@luckow I will join if my home internet starts working again.

[fbartels]

@jdaviescoates said in Meet Kopano (unstable version) - a few questions & problems :

all users need to have enough bandwidth for all streams or something

Yes, with a peer to peer connection all users need to connect to each other participant. Connections in Meet are limited to max 1 Mbit per participant, if a user does not have sufficient bandwidth for that the browser will automatically reduce quality down to a level where video would be deactivated completely.

Some more usage questions are answered in https://kopano.com/blog/top-10-things-to-ask-about-kopano-meet/

[avatar1024]

@jdaviescoates said in Meet Kopano (unstable version) - a few questions & problems :

with full p2p WebRTC apps like Kopano Meet all users need to have enough bandwidth for all streams or something

Here is a nice explanation of the pros and cons of the three main typologies used for video conference: p2p / MCU / SFU

[avatar1024]

Also might be of interest:

An article which test a methodology to compare SFUs, and then test a few of them. Results probably do not reflect real case scenario though.

[scooke]

Sorry for not participating, had some connectivity issues here. How did the call go?

[msbt]

you didn't miss it, it's this evening

[scooke]

@msbt Ah, I had the already-converted time in my head, "9", and then misremembered it as 'am'. Phew.

[girish]

That went pretty well! Thanks for joining and @luckow for organizing.

[jdaviescoates]

@girish @luckow how many people did it get up to? I wanted to join but alas was having a nightmare of a children's bedtime instead!

[scooke]

@jdaviescoates Seven. One left their video off. We had three reconnections but those were momentary and the reconnection initially resulted in clearer video images each time.

[yusf]

How resource-intensive was the session? Was the beefy spec justified?

[imc67]

Hi all, today I introduced Kopane Meet for the first time in our (volunteers) organization for a video meeting with 1 account and 5 guests.

It was a disaster! After half an hour trying and frustration we stopped and 4 of them started a WhatsApp groupvideo (Whatsapp has a max of 4?!).

What went right?
The first 2 persons (account and 1 guest) succeeded in having audio and video.

What went wrong?
All other guests could join (we could see their names) but they didn't see any video or heared audio and the first two didn't see/hear them. They could see themselves in their browser.

Time to uninstall the app and hopefully Jitsi Meet will be here soon.

[yusf]

Sounds like my experience with Kopano. Maybe you can describe device types, browsers and network topology used? I suspect some of the problems derives from part network restrictions but also weak devices and faulty browsers.

[stantropics]

@imc67 It also sounds exactly like my experience with Kopano meet. I'm trying to analyze whether the problem is somewhere on my side - however, it appears to be a real challenge.

[imc67]

@yusf of some I know their devices/browsers:

1. account: MacOS+Safari = all fine (could see/hear 2.)
2. guest: MacOS+Safari = all fine (could see/hear account, BTW on same network as account)
3. guest: MacOS+Safari and iPad+Safari = connected, could see himself, but no audio/video to/from others
4. guest: iPad+Safari = connected, could see himself, but no audio/video to/from others
5. guest: Samsung tablet with Chrome = connected, could see himself, but no audio/video to/from others
6. guest: ? = connected, could see himself, but no audio/video to/from others

The WhatsApp groupvideo after half an hour strugling with four (2./3./4./5.) went perfect, so it's not a matter of bandwith I guess.

Cloudron server load or bandwith usage during the "call" was normal, so no increase in CPU resources or network (I use ZABBIX agent on the Cloudron server so can see all stats live and all history is stored).

[necrevistonnezr]

off-topic: I think one lesson to learn here is that to do video conferencing properly is quite hard. You can hate on Zoom and Teams all you want but if you actually have your work depend on it (e.g. meeting with a client, like I do on a daily basis), it's quite a different story than just having fun with friends.

[fbartels]

Hi @imc67,

browser versions are equally as important as the browser itself. and sadly Safari isn't the best when it comes to supporting modern standard such as PWAs and WebRTC. The general recommendation would be to use Chrome or Chromium based browsers.

@imc67 said in Meet Kopano (unstable version) - a few questions & problems :

Cloudron server load or bandwith usage during the "call" was normal

Yes, that is to be expected when using WebRTC, the majority of traffic is between individual callers. There are still some connections to the server, like for example a websocket connection to kwmserver for notifications and then of course the connection to the turn server.

[luckow]

Charming participants at the first (virtual) Cloudron meetup

[oj]

@luckow We should do this again when @fbartels packages "Kopano Meet Boost" on Cloudron!

"Kopano Meet Boost is a server that we offer to our enterprise customers to meet this challenge. It is a Selective Forwarding Unit (SFU) that receives only one data stream from the sender and forwards it to all other participants. This drastically reduces the required computing power at the client and saves bandwidth. Thus, video meetings are also possible with 50 and more participants."

https://github.com/Kopano-dev/kwmbridge

[fbartels]

@oj said in Meet Kopano (unstable version) - a few questions & problems :

packages "Kopano Meet Boost" on Cloudron

thats unfortunately a bit impractical to do. kwmbridge requires a large range of udp ports so running it in a container with forwarded ports is quite inefficient (because there is also quite a memory overhead in Docker in this case).

But it should be easy to make the app configurable enough to have kwmbridge running on a dedicated host and simply hook up to kwmserver inside of the app. having (multiple) external kwmbridge endpoints is kind of the setup we are expecting anyways.