Amazing app to help with backups and saving space!

[scooke]

Here is what I have working for my Peertube:

storage:  #For some reason you don't need to remove anything here
  tmp: '/app/data/storage/tmp/' # Use to download data (imports etc), store uploaded files before processing...
  videos: '/app/data/storage/videos/'
  streaming_playlists: '/app/data/storage/streaming-playlists/'
  avatars: '/app/data/storage/avatars/'
  redundancy: '/app/data/storage/redundancy/'
  logs: '/app/data/storage/logs/'
  previews: '/app/data/storage/previews/'
  thumbnails: '/app/data/storage/thumbnails/'
  torrents: '/app/data/storage/torrents/'
  captions: '/app/data/storage/captions/'
  cache: '/app/data/storage/cache/'
  plugins: '/app/data/storage/plugins/'
  client_overrides: '/app/data/storage/client-overrides/'
  bin: /app/data/storage/bin/
# This is me trying to use my Minio for storage
object_storage:
  enabled: true
  # Without protocol, will default to HTTPS
  endpoint: 'https://minio.example.com:9000' # 's3.amazonaws.com' or 's3.fr-par.scw.cloud' for example
  region: 'fr-rbx-1'
  # Set this ACL on each uploaded object
  upload_acl: 'public'
  credentials:
    access_key_id: 'whateverbutatleast20characters'
    secret_access_key: 'whateverbutatleast40characters'
  # Maximum amount to upload in one request to object storage
  max_upload_part: 2GB
  streaming_playlists:
    bucket_name: 'peertube-bucket'
    # Allows setting all buckets to the same value but with a different prefix
    prefix: 'streaming-playlists/'
    # Example: 'streaming-playlists:'
    # Base url for object URL generation, scheme and host will be replaced by this URL
    # Useful when you want to use a CDN/external proxy
    # base_url: 'https://peertube-bucket.minio.example.com' #NOTE-the sub.subdomain is the name of the bucket.
  # Same settings but for webtorrent videos
  videos:
    bucket_name: 'peertube-bucket'
    prefix: 'videos/'
    # base_url: 'http://peertube-bucket.minio.example.com/'

I've left in some of the commented out lines just for reference. It didn't work for me with them, so I commented them out. The Bucket name is important, and just as important are the "prefix" entries because these become sub-folders in the Bucket. So my peertube-bucket has two folders, streaming-playlists and videos. As in the Mastodon config above, the Bucket needed to be Public, with the extra readwrite permission, plus an Identity/User with a readwrite Policy, to work.

Even though these Buckets are on the same Minio instance, and the Mastodon and Peertube Identities have readwrite Policies and thus could access the other buckets, I still opted to make separate Identities for each use... out of habit. And just in case it is possible to limit access using the arcane S3 cryptic policy terminology which I have not yet comprehended.

Like with the Mastodon setup, I didn't have to do anything in the GUI of Peertube, other than enter the info into the production.yaml file of the Peertube app.

[scooke]

Here is a bit more info:
For the Minio buckets to work with Mastodon, Peertube, XBackBone, and another restic-based backup solution I set up, you MUST enter new A records for the bucket and domain of your Minio setup. I don't know how it will work on Cloudron, but for my installed-by-hand Minio instance on my KS-1, I had buckets like peertube-bucket, restic-backup, mastodon-bucket, etc. My Minio instance domain is https://minio.example.com. So, I needed to make (new) A records like:

A   minio.example.com    ip123
A   peertube-bucket.minio.example.com   ip123
A   restic-backup.minio.example.com    ip123
A   mastodon-bucket.minio.example.com    ip123

After those were active, I then had to rerun sudo certbot certonly --standalone -d minio.example.com -d peertube-bucket.minio.example.com -d restic-backup.minio.example.com -d mastodon-bucket.minio.example.com -d and then copy the two new certs into the proper place (I imagine the Cloudron-based Minio will do all this automatically?)(Certbot calls this "Expanding" the certificate, and I actually added Expanded the two certs three times, rerunning the certbot certonly --standalone with all previous domains, plus whichever was the new one. It didn't work to make a new separate cert, even with it's own A Record, for, for example, resti-backup.minio.example.com plus the original minio,example.com cert. Again, I don't understand completely, but access to Minio depended on there being ONE cert with as many additional domains as necessary within it.) I tried to just use a wildcard entry for the certbot (*.minio.example.com) but it didn't work. I don't recall why, but I had to enter each sub-subdomain fully. Finally, I had to restart Minio.

If you read all my previous posts asking for help, you'll see how I achieved Mino-enlightenment bit by bit, with help from others.

One thing that confused me for awhile was that I initially got into Minio with a Media Cloud plugin for Wordpress. I didn't understand at the time what Path Style Endpoint meant, which is the default for the Wordpress plugin Media Cloud. Basically, it worked right away just entering the Bucket name, region, and Access and Secret Keys. So I didn't realize the need for A records and SSL certs for Minio to be accessible by the other method (whose name I forget!).

I should probably rewrite all of this, make it more succinct, but for now, voila!

[doodlemania2]

Everyone give it up for @scooke - that is a freaking AMAZING set of posts. I knew that each Minio instance was essentially "single user" unless you made bucket names goofy long and relied on that.

Will try with my PeerTube and Mastodon (I wonder if it'll migrate!) Thank you so much good sir!

[fbartels]

@scooke said in Amazing app to help with backups and saving space!:

you MUST enter new A records for the bucket and domain of your Minio setup

This is commonly referred to as "path style" vs. "domain style" buckets. Some applications can do both, some just one of the two.

When I was backing up my Synology NAS to my own Minio instance I had a wildcard vhost in Nginx and a wildcard certificate for this.

For domain style buckets to work with Cloudron, the Cloudron app would need to support domain aliases.

[jdaviescoates]

@fbartels said in Amazing app to help with backups and saving space!:

For domain style buckets to work with Cloudron, the Cloudron app would need to support domain aliases.

Sounds like perhaps @staff should add the aliases feature to the Minio app

[scooke]

@doodlemania2 Thanks
An update: I neglected to renew my KS-1 on which the Minio instance I was using in this post resided, so I lost it all. I then managed to get another KS-1 and used Caprover on it. By installing Minio via Caprover I have avoided hand-rolling all those certs - Caprover does it for me, I guess, in the background, with an approriately set up domain (*.example.com, not just example.com). Even though I enjoyed hand-installing it and managing it, using some software is actually nice.

The other difference, having installed visa Caprover, is that the HOSTNAME and ENDPOINT are the same (minio-s3.example.com). Not sure why. But the actual webapp of Minio is at mino.example.com. In my previous handrolled setup, the HOSTNAME was minio.example.com and the EndPoint was minio-s3.example.com. Like I said, I'm not sure why this is different. Maybe other software like Cloudron and Yunohost do the same as Caprover... all I know is it works.

Oh yes, another difference is that I used a region, eu-west-1, that doesn't necessarily correspond to my actual server location (Germany), but as long as I used that in the env file as well as the config for Minio, everything was fine.

One more difference is that there didn't seem to be a need to include the S3_PROTOCOL with my handrolled Minio, but it is needed with the Caprover install.

Here is the current env.production:
S3_ENABLED=true
S3_BUCKET=mastodon-bucket
AWS_ACCESS_KEY_ID=longlongkeyand numbers
AWS_SECRET_ACCESS_KEY=anotherlongkeywithevenmorenumbers
S3_REGION=eu-west-1
S3_PROTOCOL=https
S3_HOSTNAME=minio-s3.example.com
S3_ENDPOINT=https://minio-s3.example.com

Let me add that I am LOVING my little Mastodon instance; following and getting all kinds of super cool people and info, way more than I ever found on Twitter.

[LoudLemur]

@scooke There is a sale

[jdaviescoates]

I'm trying to follow the info in this thread to get an install of Mastodon to store media files on a Scaleway bucket, but so far I'm failing

I think it maybe something to do with needing to add DNS records to my bucket or something.

But I'm not sure how/ where/ what to do next

Currently I have this in my env.production:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud

But when I try to add any media I just get a 503 Service Unavailable

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

@scooke @fbartels @staff any ideas? Thanks!

Edit: also, it's be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain

Can anyone help? Thanks!

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

Doesn't seem like I did need to do that!

I just needed to add S3_ENDPOINT=https://s3.fr-par.scw.cloud

So now I have this and it seems to all be working:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud
S3_ENDPOINT=https://s3.fr-par.scw.cloud

@jdaviescoates said in Amazing app to help with backups and saving space!:

be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain
Can anyone help? Thanks!

Would still like to try that too if anyone can help?

[robi]

@jdaviescoates can't you just add a CNAME record that points to the S3_ALIAS_HOST ?

[jdaviescoates]

@robi thanks, I think it's something to do with that yes

(from what I've read on https://thomas-leister.de/en/mastodon-s3-media-storage/ and https://chrishubbs.com/2022/11/19/hosting-a-mastodon-instance-moving-asset-storage-to-s3/ and https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md )

In fact, I've done that. Here's the relevant DNS entry for safejust.space:

But that didn't seem to do the trick, I think because of cert issues which I'm not sure how to resolve.

But perhaps I'm doing something wrong?

[robi]

@jdaviescoates can you retrieve media manually via that CNAME?

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

But perhaps I'm doing something wrong?

Aha!

I am doing something wrong, the bucket name needs to be the same as the URL

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates can you retrieve media manually via that CNAME?

Nope, because I need to change my bucket name...

[robi]

@jdaviescoates yep.. media.s3...

[jdaviescoates]

@robi hmz, not sure this is actually properly possible with Scaleway because at the end of this guide:

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

It says:

Important:

SSL is not available when connecting to a bucket in this way.

Which I think means there is no way to resolve the certificates issue?

[scooke]

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

[robi]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

[jdaviescoates]

@scooke said in Amazing app to help with backups and saving space!:

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

Thanks, that makes sense in your case where you're using a self-hosted Minio bucket on your own domains which you can control.

I'm not using Minio, I'm just using a Scaleway Object Storage Bucket, and of course I have no control over the https://s3.fr-par.scw.cloud domain and so can't fix certificate issues with that.