Unusably Slow Playback on Peertube??

[tshirt-chihuahu]

@jdaviescoates thanks for this. I actually re-uploaded the same video to my instance to use it as a benchmark.

On your instance I can see the download speed spike to a more reasonable speed to get the video buffered as shown on the peertube interface:

This doesn't happen with mine.

So unless there was some sort of configuration that you improved (i haven't been able to find anything in the Peertube Admin), so at least thanks to your help we've ruled out that it's something to do with the basic function of Peertube.

[tshirt-chihuahu]

@girish I'll DM/email you. I'm out of ideas.

Only bottlenecks I can think of are:

1. memory( i would think this would show up in the usage graphs)
2. cpu
3. disk i/o ( Vultr advertises SSD)
4. network connection ( seems unlikely)

CPU seems the most likely culprit. Server has "4 vCPUs", CPU usage chart shows a spike from 18% to just under 30%:

Outside of my expertise here, but I believe by default Docker containers are single-threaded. On Cloudron I've got processor resource availability set to 100%, but perhaps it's possible this gets "under-reported" in the Vultr CPU usage chart because it's actually maxing out 1 of the 4 "vCPUs", but the PeerTube process is nonetheless maxed out......that's my current remaining feeble idea.

I'll open a ticket with Vultr as well.

[subven]

@tshirt-chihuahu how about you answer the question first? Upload a test clip --> make it public --> try to play it with a different device and/or from another network or link it here (if this is an option).

If your app crashes because of memory (OOM), Cloudron would notify you. If something is going wrong with your Peertube instance, the apps logs would show it. We try to think methodically and look for answers at the right spots other then guessing. Many times, the answer is blank obvious and most of the times just user error.

[tshirt-chihuahu]

@subven Been working on it. Open to other hypotheses to better narrow in on the "right spots."

It's definitely not the device or the network—the clip kindly provided by @jdaviescoates downloads and plays just fine on same network/machine.

No sign of crash notifications. The app is responsive, the download speed of the video is just slow.

[subven]

@tshirt-chihuahu is your Peertubes RTMP port allowed in Vultrs firewall?

[tshirt-chihuahu]

@subven I have not explicitly opened that port on the Vultr server, so I'll assume not. However, I had been under the impression the RTMP was only relevant to livestreaming.....
I'll assume that's not the case now. Thanks for the lead.

[subven]

@tshirt-chihuahu yeah you might be right. Just thought it could be relevant because Peertube uses a lot of federation and even torrent like downloading of videos. Normal browser playback should just work fine.

I did something nasty...installed a Peertube on the Cloudron demo server and uploaded a video Download speed is around 51-61KB and it does not play play well at all (30 second 18mb file). Okay after some minutes passed I got 1mbit download speed with Chrome after a page refresh but this might be because of caching. You can check it out here until someone purges the instance ^^ I know this deployment is the worst case scenario but at least I have a fresh instance that somehow behaves the same like you described.

[subven]

@subven said in Unusably Slow Playback on Peertube??:

Download speed is around 51-61KB and it does not play play well at all (30 second 18mb file).

Got it working on the Cloudron demo instance! Go into file manager and edit production.yaml. This is what I changed:

- line 123 --> accept_from: 'nobody'
- line 201 --> enabled: false
- line 238 --> enabled: false

line 130:

tracker:
  # If you disable the tracker, you disable the P2P aspect of PeerTube
  enabled: false

After disabling the tracker (and peertubes P2P function), download speed was instant and every video played just fine. If this works for you, this might be the solution! Ressource usage was standard - app memory at 512mb and redis at 150mb. No other changes made.

[girish]

@subven Ah, good find. I had no idea P2P aspect could be disabled!

[subven]

@girish I think disabling P2P should be the standard for new Cloudron installations. Could save you lots of trouble and federation publication is off by default too.

[tshirt-chihuahu]

@subven it worked! My test video plays instantly and smoothly now!

face palm on my part for not looking directly for a config file to turn off federation....

Thanks for the lift @subven This came up big!

[tshirt-chihuahu]

@subven @girish An alternative suggestion to disabling by default: calling this setting out in the cloudron app documentation. ( I'm kind of surprised this hadn't come up in the forum yet.)

[jdaviescoates]

Hmm, interesting, I don't think I ever edited any config to turn any p2p or federation stuff off.

[girish]

Thing is we don't generally touch upstream defaults - https://github.com/Chocobozzz/PeerTube/blob/develop/config/default.yaml#L282 and https://github.com/Chocobozzz/PeerTube/blob/develop/config/production.yaml.example#L280 .

I will put this in our docs.

[girish]

Added a note in https://docs.cloudron.io/apps/peertube/#disable-p2p

[subven]

@girish said in Unusably Slow Playback on Peertube??:

Thing is we don't generally touch upstream defaults

Thats a fair position. Though, I have installed a fresh peertube on my private Cloudron server, gave it enough ressources and was able to reproduce the problem. Normal playback is not possible out of the box and the P2P Webtorrent functionality seems to be the culpid. P2P works within your browser and my public (client/browser) IP is displayed correctly but I still think it has something to do with the way Cloudrons setup works.

Since I as a user assume that every app in the Cloudron store works without further configuration, I see a problem here. I wonder why this wasn't noticed earlier, since the P2P defaults have been in peertube for over 3 1/2 years and the app is not new at Cloudron. To make my point: Just because it works upstream as a default, does not necessarily mean it works for us ^^

[jdaviescoates]

@subven said in Unusably Slow Playback on Peertube??:

@girish said in Unusably Slow Playback on Peertube??:

Thing is we don't generally touch upstream defaults

Thats a fair position. Though, I have installed a fresh peertube on my private Cloudron server, gave it enough ressources and was able to reproduce the problem. Normal playback is not possible out of the box and the P2P functionality seems to be the culpid. P2P works within your browser and my public (client/browser) IP is displayed correctly but I still think it has something to do with the way Cloudrons setup works.

Since I as a user assume that every app in the Cloudron store works without further configuration, I see a problem here. I wonder why this wasn't noticed earlier, since the P2P defaults have been in peertube for over 3 1/2 years and the app is not new at Cloudron. To make my point: Just because it works upstream as a default, does not necessarily mean it works for us ^^

I wonder why I haven't hit this issue then? Pretty sure I've not done anything to turn P2P stuff off on https://video.uniteddiversity.coop

[subven]

@jdaviescoates said in Unusably Slow Playback on Peertube??:

I wonder why I haven't hit this issue then?

P2P and Webtorrent is on by default. If I open up your video it plays nice even if it does not have direct/instant download speed but you seem to have Webtorrent disabled and use HLS instead. Try something with more bitrate like a 4K or FullHD file with 10-25mbit. Does it still work well? (I would assume yes)

There are different player modes depending on the P2P/Webtorrent/HLS settings (right click a playing video and choose stats for nerds to see player modes):

Player-Mode: HTTP
P2P: deactivated
--> Plays instant and nice but does not follow the approach of Peertube because of no P2P. This is what I considered as a workaround or "stable" default but it is not really desirable.

Player-Mode: p2p-media-loader
P2P: activated (or disabled in some cases?)
--> @jdaviescoates this is what your instance does. Seems to work and supports P2P so if there are no other peers, the video is downloaded just from your server. This should work for everyone and I would consider it as a good default option.

Player-Mode: webtorrent
P2P: activated
--> This is the default configuration and it does not work well. Slow download and unplayable. There is also a difference if you're logged in or not. I opened a video in 5 different browser windows --> got 5 peers --> video started to download/play nicer for "everyone" but still really slow. Without another peer connected, the download is terrible (20-150kbs) so the feature works but heavily relies on federation and multiple peers which is not suitable for starters or stand alone instances.

Explanation at the offical documentation about webtorrent/HLS streaming.

You can find the UI configuration for the features here:

I would recommend to turn webtorrent off unless you have a huge federation and a lot of peers. Not sure if it would be okay to enable HFS as default since it requires ffmpeg but it is recommended.

@girish I think the docs (Disable P2P) should be adjusted since we now know that it is webtorrent and not P2P (also possible through WebRTC) in general. You can turn off webtorrent within the UI but you need to restart the app afterwards (or at least it did not switch instantly for me). The biggest problem with webtorrent on as default is that files that were uploaded with "webtorrent on / HLS off" still are "unplayable" after switching the settings. They need to be reencoded and I'm not sure if this triggers automacially or at all. I tested the behavior with two files and different settings enabled.

Another hurdle: From this point, all the following keys can be overridden by the web interface ~~ the modification of the following keys will have no effect starts at L360 in the example yaml. However in my production.yaml (Cloudron app) it is at L162 so you could end up doing configuration that gets ignored.

[tshirt-chihuahu]

@subven thanks a lot for helping us get to the bottom of this.

The webtorrent selection UI in the settings is a bit odd (and the peertube documentation has no search functionality), so your writeup should hopefully help any future travelers figure it out.

I was able to find the the relevant section in the Peertube Docs now that explains these options a bit. HLS is actually the recommended setting by them as well:

We recommend you to enable HLS (and disable WebTorrent if you don't want to store 2 different versions of the same video resolution) because video playback is really better

https://docs.joinpeertube.org/admin-configuration?id=webtorrent-transcoding-or-hls-transcoding

[girish]

@subven Oh, I forgot to act on your suggestion. I will go ahead and disable webtorrent in the next package release.