Backing up media cache

[archos]

Thank you very much, it doesn't look very easy but I will try and let you know. Thanks again for your help.

[archos]

@jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.

Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):

I tried to re-create the keys, but to no avail.

[jdaviescoates]

@archos said in Backing up media cache:

@jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.

Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):

I tried to re-create the keys, but to no avail.

The logs where? Which provider?

[archos]

@jdaviescoates This is the log from Mastodon when I try to post a photo. It's on every provider I've tried. Yesterday I tried following this exact procedure and created a bucket on AWS and the result was the same.

[archos]

I followed the joinmastodon instructions, on the Scaleway repository. Now I'm getting this error in the logs in the application.

Aws::S3::Errors::AuthorizationHeaderMalformed (The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'):

I chose Paris as the region for the bucket and in env.production. So I really don't know what I'm doing wrong :-(.

# Store data using S3 object
S3_ENABLED=true
S3_BUCKET=mastcz
AWS_ACCESS_KEY_ID=ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=SECRET_ACCESS_KEY
#S3_REGION=fr-par
S3_PROTOCOL=https
S3_HOSTNAME=s3.fr-par.scw.cloud
S3_ENDPOINT=https://mastcz.s3.fr-par.scw.cloud

[jdaviescoates]

@archos looks like your S3_ENDPOINT is wrong. For me at least it's just https://s3.fr-par.scw.cloud without the bucket name at the beginning.

[archos]

Yes I tried that first and the same result

[archos]

Hi, I managed to set up the S3 repository according to this guide. Now the photos are uploading to the storage. I just have a problem with avatars and pictures that were previously uploaded. Even custom emoji are not visible now. Does anyone else have any ideas on how to fix this please?
I shouldn't have added the sync command. I am very grateful for your advice and information. Thank you very much.

[nebulon]

Presumably the app does not automatically re-upload the assets. Maybe upstream has a storage migration script for this? But I haven't seen that so far at least.

[archos]

@nebulon Thank you for your reply. In the instructions it says that first you have to run the command

cd /home/mastodon/live
aws s3 sync --acl public-read public/system/ s3://instance-media --endpoint=https://s3.fr-par.scw.cloud

To synchronize local data, cache Mastodon media to the S3 bucket.
I guess that would have to be a different command here on Cloudron.
Now my images load, but the local avatars, server image have disappeared from the instance and the remote images and avatars are not loading. Is it possible that the remote images will load gradually, I don't know.
I've been really struggling with this for a few days now and the result is still not good.

[sebastienserre]

Hello
Did you finally succeed to synchronize your local data ?
I've the same issue, no more avatar for example

[nichu42]

This post is deleted!

[nichu42]

@doodlemania2 Could you please elaborate a bit more on how you managed to get awscli working inside the Cloudron container? I'm having trouble installing it (getting "file system is read-only" errors all the way long).

[nichu42]

Bonus question @nebulon: Would you be able to include awscli in the Mastodon package?

[nebulon]

Not sure if including this in the app image is a good way forward. Trying to understand what the issue here is in the first place. So you want to move the media cache from mastodon to a S3 bucket? Is there any reason to sync this via S3 tooling instead of setting up the bucket in mastodon and then just let mastodon deal with cache misses?

[scooke]

Let me say that moving ANYTHING from a media cache to another S3 bucket is just an exercise in frustration. Listen, I know modern software promises that everything will work, but it just doesn't sometimes. I also have done this dance with Mastodon, and in the end the ONLY thing that works is me saving my original media which I use on my own instance, locally. When I move the instance, I just refer to a screenshot of what my Mastodon looked like, and use the same image files for logo and icon and etc. Trying to get x number of weeks, or even months, of accessed media to transfer over???? Forget about it. Really. It's just not guaranteed. If you are going to move your instance, download your followers, download your personal media, download your hashtags, moderation list, etc., EVERYTHING, and when you are finished setting up your new instances, reupload everything. Inconvenient? Sure. "But mastodon said it should work!" - but it doesn't. I think the reason I accept it and just work around it is because I am the one still in charge of my media. It is my privilege to download my info, save it, and be able to use it again.

But transferring over a cache?? Why? I tried, too, but as I observed the nature of Mastodon, why would I expect that to even be possible? The toots I like, I save. The links that caught my eye, I followed and saved with my ArchiveBox. The toots go flying by, and in a week there's simply been too many.

[nichu42]

Well, all the guides linked in this thread say you should do this. I want to get rid of my locally stored assets altogether.

[nichu42]

I think the thread title might have caused some confusion. I originally started this thread because I was having issues with backups. I managed to resolve those for the most part by aggressively clearing the cache, so I postponed the S3 topic.
However, my (public) instance has continued to grow since then, and some users are posting plenty of pictures and videos. As a result, the local storage usage keeps climbing relentlessly, so now I really need to make the move to S3.

The guides linked here in the thread all mention migrating the cache folder. That's why I had intended to do that too. But honestly, I'm not really care about the cache. As @nebulon rightly pointed out, it'll just get repopulated anyway. Sure, that might lead to some performance hits, but that's a secondary concern.

My main concern is really the assets that are permanently stored on my instance. I can't just leave those behind. Since there's no fallback mechanism, all existing media files will inevitably result in 404 errors. Once S3 is activated, Mastodon will only look for files in the S3 storage. So, uploading everything to the S3 bucket is essential. I intended to use awscli for this, but installing it in the container following Amazon's instructions failed. It's possible I did something wrong, though. I'd be really grateful if someone could point me in the right direction here.

Thanks a lot!

[robi]

@nichu42 said in Backing up media cache:

installing it in the container following Amazon's instructions failed.

please post the error or open a new thread for clarity.

[nichu42]

@robi Good idea, thanks. Continuing in https://forum.cloudron.io/post/105874