Amazing app to help with backups and saving space!

[scooke]

@doodlemania2 Thanks
An update: I neglected to renew my KS-1 on which the Minio instance I was using in this post resided, so I lost it all. I then managed to get another KS-1 and used Caprover on it. By installing Minio via Caprover I have avoided hand-rolling all those certs - Caprover does it for me, I guess, in the background, with an approriately set up domain (*.example.com, not just example.com). Even though I enjoyed hand-installing it and managing it, using some software is actually nice.

The other difference, having installed visa Caprover, is that the HOSTNAME and ENDPOINT are the same (minio-s3.example.com). Not sure why. But the actual webapp of Minio is at mino.example.com. In my previous handrolled setup, the HOSTNAME was minio.example.com and the EndPoint was minio-s3.example.com. Like I said, I'm not sure why this is different. Maybe other software like Cloudron and Yunohost do the same as Caprover... all I know is it works.

Oh yes, another difference is that I used a region, eu-west-1, that doesn't necessarily correspond to my actual server location (Germany), but as long as I used that in the env file as well as the config for Minio, everything was fine.

One more difference is that there didn't seem to be a need to include the S3_PROTOCOL with my handrolled Minio, but it is needed with the Caprover install.

Here is the current env.production:
S3_ENABLED=true
S3_BUCKET=mastodon-bucket
AWS_ACCESS_KEY_ID=longlongkeyand numbers
AWS_SECRET_ACCESS_KEY=anotherlongkeywithevenmorenumbers
S3_REGION=eu-west-1
S3_PROTOCOL=https
S3_HOSTNAME=minio-s3.example.com
S3_ENDPOINT=https://minio-s3.example.com

Let me add that I am LOVING my little Mastodon instance; following and getting all kinds of super cool people and info, way more than I ever found on Twitter.

[LoudLemur]

@scooke There is a sale

[jdaviescoates]

I'm trying to follow the info in this thread to get an install of Mastodon to store media files on a Scaleway bucket, but so far I'm failing

I think it maybe something to do with needing to add DNS records to my bucket or something.

But I'm not sure how/ where/ what to do next

Currently I have this in my env.production:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud

But when I try to add any media I just get a 503 Service Unavailable

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

@scooke @fbartels @staff any ideas? Thanks!

Edit: also, it's be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain

Can anyone help? Thanks!

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

Doesn't seem like I did need to do that!

I just needed to add S3_ENDPOINT=https://s3.fr-par.scw.cloud

So now I have this and it seems to all be working:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud
S3_ENDPOINT=https://s3.fr-par.scw.cloud

@jdaviescoates said in Amazing app to help with backups and saving space!:

be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain
Can anyone help? Thanks!

Would still like to try that too if anyone can help?

[robi]

@jdaviescoates can't you just add a CNAME record that points to the S3_ALIAS_HOST ?

[jdaviescoates]

@robi thanks, I think it's something to do with that yes

(from what I've read on https://thomas-leister.de/en/mastodon-s3-media-storage/ and https://chrishubbs.com/2022/11/19/hosting-a-mastodon-instance-moving-asset-storage-to-s3/ and https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md )

In fact, I've done that. Here's the relevant DNS entry for safejust.space:

But that didn't seem to do the trick, I think because of cert issues which I'm not sure how to resolve.

But perhaps I'm doing something wrong?

[robi]

@jdaviescoates can you retrieve media manually via that CNAME?

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

But perhaps I'm doing something wrong?

Aha!

I am doing something wrong, the bucket name needs to be the same as the URL

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates can you retrieve media manually via that CNAME?

Nope, because I need to change my bucket name...

[robi]

@jdaviescoates yep.. media.s3...

[jdaviescoates]

@robi hmz, not sure this is actually properly possible with Scaleway because at the end of this guide:

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

It says:

Important:

SSL is not available when connecting to a bucket in this way.

Which I think means there is no way to resolve the certificates issue?

[scooke]

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

[robi]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

[jdaviescoates]

@scooke said in Amazing app to help with backups and saving space!:

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

Thanks, that makes sense in your case where you're using a self-hosted Minio bucket on your own domains which you can control.

I'm not using Minio, I'm just using a Scaleway Object Storage Bucket, and of course I have no control over the https://s3.fr-par.scw.cloud domain and so can't fix certificate issues with that.

[LoudLemur]

@jdaviescoates said in Amazing app to help with backups and saving space!:

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

https://bunny.net

is very well regarded and might be a fine alternative to censorious, centralizing, chokepoint cloudflare.

What do others here think?

[jdaviescoates]

@LoudLemur said in Amazing app to help with backups and saving space!:

What do others here think?

IMHO, irrelevant to most people, as most people are not running sites with very high levels of traffic.

[jdaviescoates]

@scooke said in Amazing app to help with backups and saving space!:

Here is a bit more info:
For the Minio buckets to work with Mastodon, Peertube, XBackBone, and another restic-based backup solution I set up, you MUST enter new A records for the bucket and domain of your Minio setup. I don't know how it will work on Cloudron, but for my installed-by-hand Minio instance on my KS-1, I had buckets like peertube-bucket, restic-backup, mastodon-bucket, etc. My Minio instance domain is https://minio.example.com. So, I needed to make (new) A records like:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

After those were active, I then had to rerun sudo certbot certonly --standalone -d minio.example.com -d peertube-bucket.minio.example.com -d restic-backup.minio.example.com -d mastodon-bucket.minio.example.com -d and then copy the two new certs into the proper place (I imagine the Cloudron-based Minio will do all this automatically?)(Certbot calls this "Expanding" the certificate, and I actually added Expanded the two certs three times, rerunning the certbot certonly --standalone with all previous domains, plus whichever was the new one. It didn't work to make a new separate cert, even with it's own A Record, for, for example, resti-backup.minio.example.com plus the original minio,example.com cert. Again, I don't understand completely, but access to Minio depended on there being ONE cert with as many additional domains as necessary within it.) I tried to just use a wildcard entry for the certbot (*.minio.example.com) but it didn't work.

I'm trying to use a Cloudron hosted instance of Minio to back-up some non-Cloudron hosted instances of WordPress that I help manage, and I think it's not working because of this issue.

I think perhaps the Cloudron Minio app needs to have the domain alias feature added like the WordPress (Developer) App has for use with multi-sites, so that when we have set our Minio API url to minio-api.example.coop and create a bucket called bucket-name we can also add an alias bucket-name.minio-api.example.coop.