Peertube and S3/Minio Objectstorage

[scooke]

@girish I'm not sure if what you shared applies to my situation since my Minio is self-hosted on another non-Cloudron VPS. But I'm reading it anyway a few times as all these concepts slowly get understood by me.

[girish]

@scooke aha.. ignore my response Mine was for a minio hosted on cloudron.

[scooke]

Sadly, this is STILL not working for me. I've tried EVERY combo but it keeps failing at the point of uploading the video to my Minio. Here is the error with an explanation:

peertube.example.com - this is the domain of Peertube instance on Cloudron
peertube-miniobucket- this is the name of the bucket on my Minio I'm using
minio.myserver.com - this is the domain of my Minio instance elsewhere

[peertube.example.com:443] 2022-06-22 16:36:33.539 error: Cannot move video https://peertube.example.com/videos/watch/a8be4cf6-0004-4522-b38d-bc67dfb8d0c1 to object storage. {
Jun 22 18:36:33 "err": {
Jun 22 18:36:33 "stack": "Error: getaddrinfo ENOTFOUND peertube-minobucket.minio.myserver.com\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)",
Jun 22 18:36:33 "message": "getaddrinfo ENOTFOUND peertube-miniobucket.minio.myserver.com",
Jun 22 18:36:33 "errno": -3008,
Jun 22 18:36:33 "code": "ENOTFOUND",
Jun 22 18:36:33 "syscall": "getaddrinfo",
Jun 22 18:36:33 "hostname": "peertube-miniobucket.minio.myserver.com",
Jun 22 18:36:33 "$metadata": {
Jun 22 18:36:33 "attempts": 1,
Jun 22 18:36:33 "totalRetryDelay": 0
Jun 22 18:36:33 }
Jun 22 18:36:33 }
Jun 22 18:36:33 }

For some bizarre reason it seems to be PREPENDING the bucket name to the Minio domain. I've even made a new DNS entry for peertube-miniobucket.minio.myserver.com, just in case. But that didn't make a difference.

Here is my production.yaml:

object_storage:
  enabled: true
  # Without protocol, will default to HTTPS
  endpoint: 'minio.myserver.com:9000' # 's3.amazonaws.com' or 's3.fr-par.scw.cloud' for example
  region: 'fr-rbx-5'
  # Set this ACL on each uploaded object
  upload_acl: 'public'
  credentials:
    # You can also use AWS_ACCESS_KEY_ID env variable
    access_key_id: 'acess_key'
    # You can also use AWS_SECRET_ACCESS_KEY env variable
    secret_access_key: 'secret_key'
  # Maximum amount to upload in one request to object storage
  max_upload_part: 2GB
  streaming_playlists:
    bucket_name: 'peertube-miniobucket'
    # Allows setting all buckets to the same value but with a different prefix
    prefix: 'streaming-playlists/'
    # Example: 'streaming-playlists:'
    # Base url for object URL generation, scheme and host will be replaced by this URL
    # Useful when you want to use a CDN/external proxy
    base_url: 'https://minio.myserver.com/buckets/'
  # Same settings but for webtorrent videos
  videos:
    bucket_name: 'peertube-miniobucket'
    prefix: 'videos/'
    base_url: 'https://minio.myserver.com/buckets/'

For base_url I've tried every combo of minio.myserver.com, minio.myserver.com:9000, with the /buckets, or /peertube-miniobucket, or even commenting it out.

For endpoint I've also tried with or without https:// and with and without the :9000.

I had to do alot of this kind of fiddling with my Minio and my Mastodon, but it only took about 10-15 minutes to get it working with the correct combo of details. But this. THIS is taking forever!

Can anyone help?

[girish]

@scooke said in Peertube and S3/Minio Objectstorage:

"stack": "Error: getaddrinfo ENOTFOUND peertube-minobucket.minio.myserver.com

It says minobucket and not miniobucket . Or is this a copy/paste error ?

[scooke]

@girish yeah, just a copy paste error here. Good eye!

[girish]

@scooke The ENOTFOUND means things are failing at DNS level. So, I am going to take a guess of what is happening here.

In S3 object storage, buckets are accessed as "bucket.minio.blah.com" . Notice how the bucket name is part of the domain name. This means that you have to add that in your DNS.

To give more context, S3 "protocol" has two ways to access buckets. The so called prefix style which is like minio.blah.com/bucket and so called subdomain style which bucket.minio.blah.com . The former prefix style was "deprecated" by AWS and most S3 compatible providers have moved to this subdomain style as well.

AFAIK, minio does not support the subdomain style. So, maybe peertube object storage does not even work with minio and only works with the "cloud" s3 providers. I was wrong about this. Please see https://docs.min.io/docs/minio-server-configuration-guide.html (the 'Domain' section) and that should most likely fix your problem.

[girish]

@scooke This offers a better explanation of this path style vs virtual host style - https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

[scooke]

Well, thank you @girish for your help. I don't know what to do. I've made an A record for bucketname.minio.myserver.com, and pinging it shows it is reachable. I've even used certbot to generate a certificate for it. I still get the same error.

So, this is something within Peercode then, right? Because I have not had near the same kinds of problems with using my Minio with either Wordpress sites or Mastodon. I just punch in the right deets and it works. Not so with Peertube.

So I wonder how @awnzky has theirs working?? Or did he say he doesn't have it working?

[girish]

@scooke an idea might be to first test with a cloud S3 provider.

[scooke]

@girish I got it. THANK YOU.

I'm going to rewrite the steps in the way that helped click for me. In a nutshell, I didn't realize that I needed to enter the bucket name within the base_url config ( virtual-hosted style). The error messages WERE telling me what the problem was, and you were pointing me in the right direction, I just didn't comprehend it.

IF
peertube.example.com - this is the domain of Peertube instance on Cloudron
peertube-miniobucket- this is the name of the bucket on my Minio
minio.myserver.com - this is the domain of my Minio instance elsewhere

THEN in production.yaml:

object_storage:
  enabled: true
  # Without protocol, will default to HTTPS
  endpoint: 'https://minio.myserver.com:9000' #I entered https:// anyway.
  region: 'fr-rbx-5' # This is my own made-up region entered in Minio
  # Set this ACL on each uploaded object
  upload_acl: 'public'
  credentials:
    # You can also use AWS_ACCESS_KEY_ID env variable
    access_key_id: 'acess_key'
    # You can also use AWS_SECRET_ACCESS_KEY env variable
    secret_access_key: 'secret_key'
  # Maximum amount to upload in one request to object storage
  max_upload_part: 2GB
  streaming_playlists:
    bucket_name: 'peertube-miniobucket'
    # Allows setting all buckets to the same value but with a different prefix
    prefix: 'streaming-playlists/'
    # Example: 'streaming-playlists:'
    # Base url for object URL generation, scheme and host will be replaced by this URL
    # Useful when you want to use a CDN/external proxy
    base_url: 'https://peertube-miniobucket.minio.myserver.com/'
  # Same settings but for webtorrent videos
  videos:
    bucket_name: 'peertube-miniobucket'
    prefix: 'videos/'
    base_url: 'https://peertube-miniobucket.minio.myserver.com/'

For this to work I did have to make a new A Record of peertube-miniobucket.minio.myserver.com.

Then, I had to regenerate the cert using sudo certbot certonly --standalone -d minio.myserver.com -d peertube-miniobucket.minio.myserver.com. The first time I did this I ran it only for the new subdomain. PLUS, since this is --standalone, I forgot to copy the certs to where Minio looks for them. So, having regenerated a new cert with the two domains in it, I could then do:

sudo cp /etc/letsencrypt/live/minio.myserver.com/fullchain.pem /etc/minio/certs/public.crt
and
sudo cp /etc/letsencrypt/live/minio.myserver.com/privkey.pem /etc/minio/certs/private.key (note that even thought the new cert had only the main domain in it's name, both domains were expanded into the new cert.)

Then,

sudo chown minio-user:minio-user /etc/minio/certs/public.crt
sudo chown minio-user:minio-user /etc/minio/certs/private.key

And finally, sudo systemctl restart minio.

**https://vegastack.com/tutorials/how-to-set-up-an-object-storage-server-using-minio-on-ubuntu-18-04/ was super helpful with the cert work.

What threw me the first time was thinking @awnzky's base_url: 'https://peertube.e4u6.la11.idrivee2-5.com' was their Minio domain. I didn't realize that the peertube was the ALSO the bucket name. I also thought that the software was putting the domain together by itself, somehow magically.

So, all this time spent simply because I didn't realize the bucket name has to be part of the base_url in the yaml file. Thank you @girish and @awnzky . Hopefully my write-up can help others with a mind like mine.

[scooke]

@scooke Well, I'm getting ahead of myself. I've managed to set it up to upload to my Minio... but now there is a problem with playback. As in, the video doesn't playback. When I try to download it, the initial link says https://peertube.example.com/download/videos/longfilename.mp4 but then starting the download results in a failed white screen saying that the following can't be reached, https://peertube-miniobucket.minio.myserver.com/peertube-miniobucket/videos/longfilename.mp4 .

The documentation I find online seems conflicting - some say Peertube doesn't work with S3, others say yes, and still a third says "yes but with a CDN directive". So I'm off to see what I can find out.

[girish]

@scooke Try unsetting base_url and see if that works. I think base_url is for CDN setups which requires additional configuration - https://docs.joinpeertube.org/admin-remote-storage?id=cache-server . In a CDN setup, the video is served to the browser straight from minio (as opposed to browser -> peertube -> minio).

[scooke]

@girish Holy smokes. That did it. I just # commented out those two base_url lines, restarted the app, and now the upload works, and the download.

Now, I thought I had reached a similar point before, but I think what I had missed was copying the new cert to the proper directory. Thus, it wasn't working.

Now, it is. You sir are a magician.

[jdaviescoates]

@awnzky said in Peertube and S3/Minio Objectstorage:

iam already try using contabo. it's does not work.

Wish I'd read/ seen this earlier. I just spent hours pulling my hair trying to get contabo object storage to work with peertube but I just couldn't get it to work either (whereas on Scaleway it Just Worked - and even when it didn't work, like when I hadn't set the region, it gave useful error messages telling me what was wrong and how to fix it )

However, even with the S3 connection to my Scaleway object buckets working fine, playback actually just didn't work

Playback with Object Storage disabled works fine.

I don't have any base_url set anywhere.

[robi]

@jdaviescoates playback may require the bucket to be https accessible, which is a setting.

[jdaviescoates]

@robi said in Peertube and S3/Minio Objectstorage:

@jdaviescoates playback may require the bucket to be https accessible, which is a setting.

You mean making the bucket public? Already is. Can't find any other settings related to https.

[scooke]

@jdaviescoates I wonder if the problem is the TYPE of bucket. In the comments above I was running a self-installed instance of Minio. That is, I didn't use Yunohost, or CapRover, or Cloudron, etc. I installed it by hand. Somehow (I guess I could figure it out by rereading the above) I set Minio up to use virtual-host, rather than path-style. And for whatever reason, Peertube (only) works with virtual-host style. Maybe this has something to do with it?

FWIW (I'm only getting my own head sort of around all this), the end_point in teh config has to be whatever your API enpoint is. Off the top of my head, I forget how to tell which is which, but one address is the home adress, and the other is the end point address. That is to say, when it is being installed on mino.example.com, a second API address is created, something like minio-s3.example.com. The API is the address needed.

[jdaviescoates]

@scooke said in Peertube and S3/Minio Objectstorage:

Maybe this has something to do with it?

That's why Contabo didn't work, I think.

But Scaleway does support that and did work in that the files get transferred to the bucket fine. It's just that then playback basically doesn't work at all.

[scooke]

@jdaviescoates Care to share your config (the pertinent part)?

[jdaviescoates]

@scooke

# Store all videos in one bucket on Scaleway
object_storage:
  enabled: true
  # Scaleway endpoint
  endpoint: 's3.fr-par.scw.cloud'
  region: 'fr-par'
  videos:
    bucket_name: 'bridport-tv'
    prefix: 'videos/'
  # Use the same bucket as for webtorrent videos but with a different prefix
  streaming_playlists:
    bucket_name: 'bridport-tv'
    prefix: 'streaming-playlists/'
  credentials:
    access_key_id: 'xxxxx'
    secret_access_key: 'xxxxx'

That's was working fine for getting stuff into the bucket, but then videos didn't actually play

Bucket is also public: