Amazing app to help with backups and saving space!

[jdaviescoates]

I'm trying to follow the info in this thread to get an install of Mastodon to store media files on a Scaleway bucket, but so far I'm failing

I think it maybe something to do with needing to add DNS records to my bucket or something.

But I'm not sure how/ where/ what to do next

Currently I have this in my env.production:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud

But when I try to add any media I just get a 503 Service Unavailable

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

@scooke @fbartels @staff any ideas? Thanks!

Edit: also, it's be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain

Can anyone help? Thanks!

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Do I need to create a record in my DNS for safejust.space that relates to s3.fr-par.scw.cloud or something?

Doesn't seem like I did need to do that!

I just needed to add S3_ENDPOINT=https://s3.fr-par.scw.cloud

So now I have this and it seems to all be working:

# Trying to store data on Scaleway S3 object
S3_ENABLED=true
S3_BUCKET=safe-just-space
AWS_ACCESS_KEY_ID=<key_id>
AWS_SECRET_ACCESS_KEY=<secret_key>
S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud
S3_ENDPOINT=https://s3.fr-par.scw.cloud

@jdaviescoates said in Amazing app to help with backups and saving space!:

be nice if I could S3_ALIAS_HOST=<url> settings too so that media URLs would e.g. be media.safejust.space instead of a s3.fr-par.scw.cloud domain
Can anyone help? Thanks!

Would still like to try that too if anyone can help?

[robi]

@jdaviescoates can't you just add a CNAME record that points to the S3_ALIAS_HOST ?

[jdaviescoates]

@robi thanks, I think it's something to do with that yes

(from what I've read on https://thomas-leister.de/en/mastodon-s3-media-storage/ and https://chrishubbs.com/2022/11/19/hosting-a-mastodon-instance-moving-asset-storage-to-s3/ and https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md )

In fact, I've done that. Here's the relevant DNS entry for safejust.space:

But that didn't seem to do the trick, I think because of cert issues which I'm not sure how to resolve.

But perhaps I'm doing something wrong?

[robi]

@jdaviescoates can you retrieve media manually via that CNAME?

[jdaviescoates]

@jdaviescoates said in Amazing app to help with backups and saving space!:

But perhaps I'm doing something wrong?

Aha!

I am doing something wrong, the bucket name needs to be the same as the URL

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates can you retrieve media manually via that CNAME?

Nope, because I need to change my bucket name...

[robi]

@jdaviescoates yep.. media.s3...

[jdaviescoates]

@robi hmz, not sure this is actually properly possible with Scaleway because at the end of this guide:

https://www.scaleway.com/en/docs/tutorials/s3-customize-url-cname/

It says:

Important:

SSL is not available when connecting to a bucket in this way.

Which I think means there is no way to resolve the certificates issue?

[scooke]

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

[robi]

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

[jdaviescoates]

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

[jdaviescoates]

@scooke said in Amazing app to help with backups and saving space!:

@scooke said in Amazing app to help with backups and saving space!:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

I did make A records for my self-rolled Minio for the buckets to work. Without those, they were unreachable. I also had to add one more A record,
A *.minio.example.com ip123 in order for a certificate to be generated that was useful and applicable to ALL those buckets!

Thats said, with this new Caprover installation, there's been no problem, no need to tweak DNS beyond the initial Caprover setup. I'm also still using path-style.

Thanks, that makes sense in your case where you're using a self-hosted Minio bucket on your own domains which you can control.

I'm not using Minio, I'm just using a Scaleway Object Storage Bucket, and of course I have no control over the https://s3.fr-par.scw.cloud domain and so can't fix certificate issues with that.

[LoudLemur]

@jdaviescoates said in Amazing app to help with backups and saving space!:

@robi said in Amazing app to help with backups and saving space!:

@jdaviescoates said in Amazing app to help with backups and saving space!:

Which I think means there is no way to resolve the certificates issue?

you have Cloudflare right?

Nope.

I dislike the centralisation caused by Cloudflare and have never really got why so many people want to give their and their uses data to them.

I think sites are many times more likely to go down due to Cloudflare outages than by getting slash dotted.

https://bunny.net

is very well regarded and might be a fine alternative to censorious, centralizing, chokepoint cloudflare.

What do others here think?

[jdaviescoates]

@LoudLemur said in Amazing app to help with backups and saving space!:

What do others here think?

IMHO, irrelevant to most people, as most people are not running sites with very high levels of traffic.

[jdaviescoates]

@scooke said in Amazing app to help with backups and saving space!:

Here is a bit more info:
For the Minio buckets to work with Mastodon, Peertube, XBackBone, and another restic-based backup solution I set up, you MUST enter new A records for the bucket and domain of your Minio setup. I don't know how it will work on Cloudron, but for my installed-by-hand Minio instance on my KS-1, I had buckets like peertube-bucket, restic-backup, mastodon-bucket, etc. My Minio instance domain is https://minio.example.com. So, I needed to make (new) A records like:

A minio.example.com ip123
A peertube-bucket.minio.example.com ip123
A restic-backup.minio.example.com ip123
A mastodon-bucket.minio.example.com ip123

After those were active, I then had to rerun sudo certbot certonly --standalone -d minio.example.com -d peertube-bucket.minio.example.com -d restic-backup.minio.example.com -d mastodon-bucket.minio.example.com -d and then copy the two new certs into the proper place (I imagine the Cloudron-based Minio will do all this automatically?)(Certbot calls this "Expanding" the certificate, and I actually added Expanded the two certs three times, rerunning the certbot certonly --standalone with all previous domains, plus whichever was the new one. It didn't work to make a new separate cert, even with it's own A Record, for, for example, resti-backup.minio.example.com plus the original minio,example.com cert. Again, I don't understand completely, but access to Minio depended on there being ONE cert with as many additional domains as necessary within it.) I tried to just use a wildcard entry for the certbot (*.minio.example.com) but it didn't work.

I'm trying to use a Cloudron hosted instance of Minio to back-up some non-Cloudron hosted instances of WordPress that I help manage, and I think it's not working because of this issue.

I think perhaps the Cloudron Minio app needs to have the domain alias feature added like the WordPress (Developer) App has for use with multi-sites, so that when we have set our Minio API url to minio-api.example.coop and create a bucket called bucket-name we can also add an alias bucket-name.minio-api.example.coop.