Backing up media cache
-
Hi, I don't want to start a new thread. I need to connect an S3 bucket to the IDrive storage. I am trying to add the configuration from the bucket in the repository to the /app/data/env.production file. I have tried just about everything. Bucket private, public, set Cname record with subdomain pointing to the repository, but after restarting the app, uploading images ends with a 500 error, or the image uploads but can't be opened, it's like hidden. Has anyone solved a similar problem please. I must have tried everything and I don't know how to set the configuration file anymore. Thank you all very much for any advice.
@archos it could be a CORS issue?
Have you already done what is described here: https://www.idrive.com/object-storage-e2/faq-dashboard#cross_origin ?
-
Thank you for your reply. I tried it now, but the result is the same.
According to me, I think I'm making a mistake in the hostname. Please Bucket should be public or private? I've been playing with this all day, I really don't know anymore.
Do I need a Cname DNS record? -
Thank you for your reply. I tried it now, but the result is the same.
According to me, I think I'm making a mistake in the hostname. Please Bucket should be public or private? I've been playing with this all day, I really don't know anymore.
Do I need a Cname DNS record?@archos said in Backing up media cache:
Please Bucket should be public or private?
From my limited experience I think either should work but you probably want it to be private (I think public means the bucket itself can be publicly browsed)
@archos said in Backing up media cache:
Do I need a Cname DNS record?
I don't think so, unless you're trying to set-up an S3 hostname using one of your domains, but I've no idea if that's actually possible if you're not self-hosting the S3 buckets yourself. I tried a while ago but then gave up (although seems this might be possible with iDrive https://www.idrive.com/object-storage-e2/faq-dashboard#use-domain but that's just adding additional complexity, so I'd focus on just getting it working for now!)
@archos said in Backing up media cache:
According to me, I think I'm making a mistake in the hostname.
Quite possible.
The annoying thing is there doesn't seem to be any standard agreed upon way it all works and it seems different providers want different things.
On Scaleway S3 this is what my working settings looks like for Mastodon:
# Store media on Scaleway S3 object S3_ENABLED=true S3_BUCKET=example_bucknet_name AWS_ACCESS_KEY_ID=example_key_id AWS_SECRET_ACCESS_KEY=example_secret_access_key S3_REGION=fr-par S3_PROTOCOL=https S3_HOSTNAME=s3.fr-par.scw.cloud S3_ENDPOINT=https://s3.fr-par.scw.cloud -
@archos said in Backing up media cache:
Please Bucket should be public or private?
From my limited experience I think either should work but you probably want it to be private (I think public means the bucket itself can be publicly browsed)
@archos said in Backing up media cache:
Do I need a Cname DNS record?
I don't think so, unless you're trying to set-up an S3 hostname using one of your domains, but I've no idea if that's actually possible if you're not self-hosting the S3 buckets yourself. I tried a while ago but then gave up (although seems this might be possible with iDrive https://www.idrive.com/object-storage-e2/faq-dashboard#use-domain but that's just adding additional complexity, so I'd focus on just getting it working for now!)
@archos said in Backing up media cache:
According to me, I think I'm making a mistake in the hostname.
Quite possible.
The annoying thing is there doesn't seem to be any standard agreed upon way it all works and it seems different providers want different things.
On Scaleway S3 this is what my working settings looks like for Mastodon:
# Store media on Scaleway S3 object S3_ENABLED=true S3_BUCKET=example_bucknet_name AWS_ACCESS_KEY_ID=example_key_id AWS_SECRET_ACCESS_KEY=example_secret_access_key S3_REGION=fr-par S3_PROTOCOL=https S3_HOSTNAME=s3.fr-par.scw.cloud S3_ENDPOINT=https://s3.fr-par.scw.cloud@jdaviescoates Thank you very much, I will try to create a bucket in Scaleway, the price is almost the same, so hopefully it will work.
-
@jdaviescoates Thank you very much, I will try to create a bucket in Scaleway, the price is almost the same, so hopefully it will work.
@archos good luck!
-
@jdaviescoates Thank you very much, I will try to create a bucket in Scaleway, the price is almost the same, so hopefully it will work.
@archos Note, you also have to do the CORS stuff on Scaleway too, so it won't work without doing that as well,
see:
https://www.scaleway.com/en/docs/storage/object/api-cli/setting-cors-rules/
But before you can do that you need to set-up the AWS CLI stuff, see:
https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/
And before you can do that you need API Keys:
https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/
Good luck!
-
@archos Note, you also have to do the CORS stuff on Scaleway too, so it won't work without doing that as well,
see:
https://www.scaleway.com/en/docs/storage/object/api-cli/setting-cors-rules/
But before you can do that you need to set-up the AWS CLI stuff, see:
https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/
And before you can do that you need API Keys:
https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/
Good luck!
@jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.
Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):I tried to re-create the keys, but to no avail.
-
@jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.
Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):I tried to re-create the keys, but to no avail.
@archos said in Backing up media cache:
@jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.
Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):I tried to re-create the keys, but to no avail.
The logs where? Which provider?
-
@archos said in Backing up media cache:
@jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.
Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):I tried to re-create the keys, but to no avail.
The logs where? Which provider?
-
I followed the joinmastodon instructions, on the Scaleway repository. Now I'm getting this error in the logs in the application.
Aws::S3::Errors::AuthorizationHeaderMalformed (The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'):I chose Paris as the region for the bucket and in env.production. So I really don't know what I'm doing wrong :-(.
# Store data using S3 object S3_ENABLED=true S3_BUCKET=mastcz AWS_ACCESS_KEY_ID=ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=SECRET_ACCESS_KEY #S3_REGION=fr-par S3_PROTOCOL=https S3_HOSTNAME=s3.fr-par.scw.cloud S3_ENDPOINT=https://mastcz.s3.fr-par.scw.cloud -
I followed the joinmastodon instructions, on the Scaleway repository. Now I'm getting this error in the logs in the application.
Aws::S3::Errors::AuthorizationHeaderMalformed (The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'):I chose Paris as the region for the bucket and in env.production. So I really don't know what I'm doing wrong :-(.
# Store data using S3 object S3_ENABLED=true S3_BUCKET=mastcz AWS_ACCESS_KEY_ID=ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=SECRET_ACCESS_KEY #S3_REGION=fr-par S3_PROTOCOL=https S3_HOSTNAME=s3.fr-par.scw.cloud S3_ENDPOINT=https://mastcz.s3.fr-par.scw.cloud@archos looks like your
S3_ENDPOINTis wrong. For me at least it's justhttps://s3.fr-par.scw.cloudwithout the bucket name at the beginning. -
Hi, I managed to set up the S3 repository according to this guide. Now the photos are uploading to the storage. I just have a problem with avatars and pictures that were previously uploaded. Even custom emoji are not visible now. Does anyone else have any ideas on how to fix this please?
I shouldn't have added the sync command. I am very grateful for your advice and information. Thank you very much. -
@nebulon Thank you for your reply. In the instructions it says that first you have to run the command
cd /home/mastodon/live aws s3 sync --acl public-read public/system/ s3://instance-media --endpoint=https://s3.fr-par.scw.cloudTo synchronize local data, cache Mastodon media to the S3 bucket.
I guess that would have to be a different command here on Cloudron.
Now my images load, but the local avatars, server image have disappeared from the instance and the remote images and avatars are not loading. Is it possible that the remote images will load gradually, I don't know.
I've been really struggling with this for a few days now and the result is still not good.
-
Hello
Did you finally succeed to synchronize your local data ?
I've the same issue, no more avatar for example -
@doodlemania2 Could you please elaborate a bit more on how you managed to get awscli working inside the Cloudron container? I'm having trouble installing it (getting "file system is read-only" errors all the way long).
