Cloudron with Nextcloud Talk can't connect calls after v20 upgrade

[robi]

v19 used to work fine, calls connect.
v20 voice/video calls ring, but just wait to connect forever.

No other configuration changed, and we still have port 3478 enabled in the ISP router firewall as the NC is on NAT behind it.

The only error/failure in the logs is this snippet:

2020-11-26T23:30:37.000Z [Thu Nov 26 23:30:37.764174 2020] [php7:notice] [pid 544] [client 172.18.0.1:43104] {"reqId":"9yitCKVWq90cnsnMuJA7","level":3,"time":"2020-11-26T23:30:37+00:00","remoteAddr":"x.x.x.x","user":"rob","app":"PHP","method":"GET","url":"/ocs/v1.php/cloud/user?format=json","message":"fopen(/dev/stderr): failed to open stream: No such file or directory at /app/code/lib/private/Log/File.php#85","userAgent":"Mozilla/5.0 (iOS) Nextcloud-Talk v10.1.0","version":"20.0.1.1"}

Hence not sure if this has to do with any port changes on the NC side or changes in call routing?

• We have tested with a fresh install, no change, no call.
• We have tested with a fresh clone, no change, no call.
• We installed Kopano, no change, no calls connect.
• We have tested with another Cloudron NC:T (VPS, no NAT) and the same user from the same NAT network, and we could connect a call.

Looking at the logs from the other Cloudron NC:T instance, there are no such log snippets or call error logs. (probably because it just worked)

Kopano install has logs that point to other internal warnings and errors, but nothing useful to help debug this and looks to be using an external TURN service by default:

2020-11-27T18:21:49.000Z time="2020-11-27T18:21:49Z" level=info msg="using external TURN service: https://turnauth.kopano.com/turnserverauth/"

So this seems either Cloudron TURN server related or new port opening requirement for NC:T for which we don't have any info.

How do we get this working again?

[girish]

@robi Thanks for the extensive testing! Can you also test with a fresh install of NC 19? You can do this by clicking NC on the app store and then change the package version in the URL bar as 4.6.5 (it will say 4.8.1 now). That will help us isolate it to most likely something network related.

[robi]

Thanks to LDAP we didn't have to redo user accounts, but no luck connecting with v19 now either.

Odd.

Nothing interesting in the app logs again.

[girish]

@robi said in Cloudron with Nextcloud Talk can't connect calls after v20 upgrade:

We have tested with another Cloudron NC:T (VPS, no NAT) and the same user from the same NAT network, and we could connect a call.

This seems then something TURN and NAT related. I think non firefox, you can go to about:webrtc to see what is happenning

[robi]

@girish this just leads to a search or invalid URL error.

tried in chrome based browsers
about:webrtc
chrome:webrtc
vivaldi:webrtc

nothing.

Tried in FF, and that brings up the WebRTC internals page.

[robi]

@girish you have an email with logs.

[girish]

@robi thanks, @nebulon and I decided to make a test call tomorrow on our instance to see if this is a general problem or something to do you with your install/network.

[girish]

@robi Johannes and I tried a whole bunch of calls. Calls fail now and then. But if we refresh browser connect/discoonect, it connects eventually. I could connect reliably in chrome but not in firefox. If I start a call in firefox, then johannes can never join it. It fails with "WebRTC: ICE failed, see about:webrtc for more details" for me in the console.

Not sure, what the final outcome of all this is. It's basically, "your mileage may vary". Maybe should be reported upstream?

[robi]

@girish Thanks for testing.

It's disappointing that what once worked so well and is the basis of the self-hosted install is now unusable.

Furthermore, there don't appear to be sufficient logs figure out what could possible be adjusted. What about the TURN server logs? TCP/UDP ports?

Even stranger is that depending on who initiates the connection, an audio stream manages to connect, but only in one direction.

Since VPS Cloudrons work well, I'm leaning towards the router port forwarding, but which ports will make a difference??

[robi]

@girish Do you have a better idea of what to report upstream vs 'now it doesn't work' ?

[nebulon]

@robi it more looks like some state handling issue in Nextcloud talk and not so much an issue in the TURN server or port setup, if it works sometimes and sometimes not.

I guess for your setup to gain some more insight would be to check if you are able to make calls at all even if only sometimes and also test kopano meet to check if this is even similar to what we have seen.

Generally when reporting something here upstream it will surely help to send a dump of the browser console logs alongside, Nextcloud talk outputs a lot of debugging info, I assume this is there for a reason...

[girish]

@robi I don't know what to exactly "report". (TBH, If I did, I would have reported it upstream myself ). I didn't investigate much but I couldn't figure how to get meaningful logs for the nextcloud devs.

[robi]

@girish it seems to be a difference between single port usage and multi port usage via TURN server. The former used to work fine but may have been deprecated?

My gut instinct was telling me it's also port related.

Doing some more testing we enabled UDP ports in the 50,000 range and began to have better call connection results.

Final tweak was realizing that one of the clients had a mobile VPN enabled which only dropped incoming audio connections but not outgoing. :

So if you can validate your testing with @nebulon that adding the UDP ports to the NAT Firewall lets you connect, that would give enough data for an upstream report.

[girish]

@robi said in Cloudron with Nextcloud Talk can't connect calls after v20 upgrade:

So if you can validate your testing with @nebulon that adding the UDP ports to the NAT Firewall lets you connect, that would give enough data for an upstream report.

FWIW, we tested with a Cloudron on a VPS, so there is cloud/NAT firewall. Indeed, if you have a NAT firewall those ports have to be opened up. I will add it to https://docs.cloudron.io/security/#cloud-firewall

[robi]

@girish right, yet this all used to work fine with a single port until the latest changes. Not sure if that was planned or a regression.

[jdaviescoates]

...and I'm not sure if this issue effects me/ everyone or not? Is there someone extra we now all have to do to get Nextcloud Talk working after a v20 upgrade?

[robi]

@jdaviescoates only if you're behind a NATing firewall.

VPS Cloudrons are not affected.
Home and Office Cloudrons would be and need to have the appropriate port forwarding set up if expecting calls with other users outside the local network to connect.

[atrilahiji]

I'd like to revive this. Looks like its still an issue for me. For context I have even tried making my server the DMZ on my router to rule out DNS issues. Here are my ports I've forwarded. It just loads for me and anyone I try and invite.

Any ideas? @girish @robi

EDIT: Just tried local connections and it works... O_O

[atrilahiji]

So one issue I noticed is that when I use my Adguard DNS to talk to someone with nextcloud talk it works. When I use the default DNS server I have issues. Odd...

EDIT: I have some TURN logs I sent in via a support ticket btw

EDIT 2: Nevermind the adguard trick only works on my phone... unsure whats going on.

EDIT 3: Nevermind again. Seems like this issue is happening for me in the browser??? works fine afaik on the iOS/Android app.

[atrilahiji]

Incredibly confusing and frustrating. I guess I'll look at it more in the morning...