Amazing app to help with backups and saving space!

[doodlemania2]

Everyone give it up for @scooke - that is a freaking AMAZING set of posts. I knew that each Minio instance was essentially "single user" unless you made bucket names goofy long and relied on that.

Will try with my PeerTube and Mastodon (I wonder if it'll migrate!) Thank you so much good sir!

[fbartels]

@scooke said in Amazing app to help with backups and saving space!:

you MUST enter new A records for the bucket and domain of your Minio setup

This is commonly referred to as "path style" vs. "domain style" buckets. Some applications can do both, some just one of the two.

When I was backing up my Synology NAS to my own Minio instance I had a wildcard vhost in Nginx and a wildcard certificate for this.

For domain style buckets to work with Cloudron, the Cloudron app would need to support domain aliases.

[jdaviescoates]

@fbartels said in Amazing app to help with backups and saving space!:

For domain style buckets to work with Cloudron, the Cloudron app would need to support domain aliases.

Sounds like perhaps @staff should add the aliases feature to the Minio app

[scooke]

@doodlemania2 Thanks
An update: I neglected to renew my KS-1 on which the Minio instance I was using in this post resided, so I lost it all. I then managed to get another KS-1 and used Caprover on it. By installing Minio via Caprover I have avoided hand-rolling all those certs - Caprover does it for me, I guess, in the background, with an approriately set up domain (*.example.com, not just example.com). Even though I enjoyed hand-installing it and managing it, using some software is actually nice.

The other difference, having installed visa Caprover, is that the HOSTNAME and ENDPOINT are the same (minio-s3.example.com). Not sure why. But the actual webapp of Minio is at mino.example.com. In my previous handrolled setup, the HOSTNAME was minio.example.com and the EndPoint was minio-s3.example.com. Like I said, I'm not sure why this is different. Maybe other software like Cloudron and Yunohost do the same as Caprover... all I know is it works.

Oh yes, another difference is that I used a region, eu-west-1, that doesn't necessarily correspond to my actual server location (Germany), but as long as I used that in the env file as well as the config for Minio, everything was fine.

One more difference is that there didn't seem to be a need to include the S3_PROTOCOL with my handrolled Minio, but it is needed with the Caprover install.

Here is the current env.production:
S3_ENABLED=true
S3_BUCKET=mastodon-bucket
AWS_ACCESS_KEY_ID=longlongkeyand numbers
AWS_SECRET_ACCESS_KEY=anotherlongkeywithevenmorenumbers
S3_REGION=eu-west-1
S3_PROTOCOL=https
S3_HOSTNAME=minio-s3.example.com
S3_ENDPOINT=https://minio-s3.example.com

Let me add that I am LOVING my little Mastodon instance; following and getting all kinds of super cool people and info, way more than I ever found on Twitter.

[LoudLemur]

@scooke There is a sale

[jdaviescoates]

I'm trying to follow the info in this thread to get an install of Mastodon to store media files on a Scaleway bucket, but so far I'm failing

I think it maybe something to do with needing to add DNS records to my bucket or something.

But I'm not sure how/ where/ what to do next

Currently I have this in my env.production:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud

But when I try to add any media I just get a 503 Service Unavailable

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

@scooke @fbartels @staff any ideas? Thanks!

Edit: also, it's be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain

Can anyone help? Thanks!

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

Doesn't seem like I did need to do that!

I just needed to add S3_ENDPOINT=https://s3.fr-par.scw.cloud

So now I have this and it seems to all be working:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud
S3_ENDPOINT=https://s3.fr-par.scw.cloud

@jdaviescoates said in Amazing app to help with backups and saving space!:

be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain
Can anyone help? Thanks!

Would still like to try that too if anyone can help?

[robi]

@jdaviescoates can't you just add a CNAME record that points to the S3_ALIAS_HOST ?

[jdaviescoates]

@robi thanks, I think it's something to do with that yes

(from what I've read on https://thomas-leister.de/en/mastodon-s3-media-storage/ and https://chrishubbs.com/2022/11/19/hosting-a-mastodon-instance-moving-asset-storage-to-s3/ and https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md )

In fact, I've done that. Here's the relevant DNS entry for safejust.space:

But that didn't seem to do the trick, I think because of cert issues which I'm not sure how to resolve.

But perhaps I'm doing something wrong?

[robi]

@jdaviescoates can you retrieve media manually via that CNAME?

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

But perhaps I'm doing something wrong?

Aha!

I am doing something wrong, the bucket name needs to be the same as the URL

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates can you retrieve media manually via that CNAME?

Nope, because I need to change my bucket name...

[robi]

@jdaviescoates yep.. media.s3...

[jdaviescoates]

@robi hmz, not sure this is actually properly possible with Scaleway because at the end of this guide:

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

It says:

Important:

SSL is not available when connecting to a bucket in this way.

Which I think means there is no way to resolve the certificates issue?

[scooke]

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

[robi]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

[jdaviescoates]

@scooke said in Amazing app to help with backups and saving space!:

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

Thanks, that makes sense in your case where you're using a self-hosted Minio bucket on your own domains which you can control.

I'm not using Minio, I'm just using a Scaleway Object Storage Bucket, and of course I have no control over the https://s3.fr-par.scw.cloud domain and so can't fix certificate issues with that.

[LoudLemur]

@jdaviescoates said in Amazing app to help with backups and saving space!:

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

https://bunny.net

is very well regarded and might be a fine alternative to censorious, centralizing, chokepoint cloudflare.

What do others here think?

[jdaviescoates]

@LoudLemur said in Amazing app to help with backups and saving space!:

What do others here think?

IMHO, irrelevant to most people, as most people are not running sites with very high levels of traffic.