Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Mastodon
  3. Backing up media cache

Backing up media cache

Scheduled Pinned Locked Moved Mastodon
mastodonbackupbackup-storage
44 Posts 10 Posters 5.1k Views 9 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • archosA archos

    Hi, I don't want to start a new thread. I need to connect an S3 bucket to the IDrive storage. I am trying to add the configuration from the bucket in the repository to the /app/data/env.production file. I have tried just about everything. Bucket private, public, set Cname record with subdomain pointing to the repository, but after restarting the app, uploading images ends with a 500 error, or the image uploads but can't be opened, it's like hidden. Has anyone solved a similar problem please. I must have tried everything and I don't know how to set the configuration file anymore. Thank you all very much for any advice.

    jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by
    #19

    @archos it could be a CORS issue?

    Have you already done what is described here: https://www.idrive.com/object-storage-e2/faq-dashboard#cross_origin ? 🙂

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    1
    • archosA Offline
      archosA Offline
      archos
      wrote on last edited by
      #20

      Thank you for your reply. I tried it now, but the result is the same.
      According to me, I think I'm making a mistake in the hostname. Please Bucket should be public or private? I've been playing with this all day, I really don't know anymore.
      Do I need a Cname DNS record?

      jdaviescoatesJ 1 Reply Last reply
      0
      • archosA archos

        Thank you for your reply. I tried it now, but the result is the same.
        According to me, I think I'm making a mistake in the hostname. Please Bucket should be public or private? I've been playing with this all day, I really don't know anymore.
        Do I need a Cname DNS record?

        jdaviescoatesJ Offline
        jdaviescoatesJ Offline
        jdaviescoates
        wrote on last edited by
        #21

        @archos said in Backing up media cache:

        Please Bucket should be public or private?

        From my limited experience I think either should work but you probably want it to be private (I think public means the bucket itself can be publicly browsed)

        @archos said in Backing up media cache:

        Do I need a Cname DNS record?

        I don't think so, unless you're trying to set-up an S3 hostname using one of your domains, but I've no idea if that's actually possible if you're not self-hosting the S3 buckets yourself. I tried a while ago but then gave up (although seems this might be possible with iDrive https://www.idrive.com/object-storage-e2/faq-dashboard#use-domain but that's just adding additional complexity, so I'd focus on just getting it working for now!)

        @archos said in Backing up media cache:

        According to me, I think I'm making a mistake in the hostname.

        Quite possible.

        The annoying thing is there doesn't seem to be any standard agreed upon way it all works and it seems different providers want different things.

        On Scaleway S3 this is what my working settings looks like for Mastodon:

        # Store media on Scaleway S3 object
        S3_ENABLED=true
        S3_BUCKET=example_bucknet_name
        AWS_ACCESS_KEY_ID=example_key_id
        AWS_SECRET_ACCESS_KEY=example_secret_access_key
        S3_REGION=fr-par
        S3_PROTOCOL=https
        S3_HOSTNAME=s3.fr-par.scw.cloud
        S3_ENDPOINT=https://s3.fr-par.scw.cloud
        

        I use Cloudron with Gandi & Hetzner

        archosA 1 Reply Last reply
        0
        • jdaviescoatesJ jdaviescoates

          @archos said in Backing up media cache:

          Please Bucket should be public or private?

          From my limited experience I think either should work but you probably want it to be private (I think public means the bucket itself can be publicly browsed)

          @archos said in Backing up media cache:

          Do I need a Cname DNS record?

          I don't think so, unless you're trying to set-up an S3 hostname using one of your domains, but I've no idea if that's actually possible if you're not self-hosting the S3 buckets yourself. I tried a while ago but then gave up (although seems this might be possible with iDrive https://www.idrive.com/object-storage-e2/faq-dashboard#use-domain but that's just adding additional complexity, so I'd focus on just getting it working for now!)

          @archos said in Backing up media cache:

          According to me, I think I'm making a mistake in the hostname.

          Quite possible.

          The annoying thing is there doesn't seem to be any standard agreed upon way it all works and it seems different providers want different things.

          On Scaleway S3 this is what my working settings looks like for Mastodon:

          # Store media on Scaleway S3 object
          S3_ENABLED=true
          S3_BUCKET=example_bucknet_name
          AWS_ACCESS_KEY_ID=example_key_id
          AWS_SECRET_ACCESS_KEY=example_secret_access_key
          S3_REGION=fr-par
          S3_PROTOCOL=https
          S3_HOSTNAME=s3.fr-par.scw.cloud
          S3_ENDPOINT=https://s3.fr-par.scw.cloud
          
          archosA Offline
          archosA Offline
          archos
          wrote on last edited by
          #22

          @jdaviescoates Thank you very much, I will try to create a bucket in Scaleway, the price is almost the same, so hopefully it will work.

          jdaviescoatesJ 2 Replies Last reply
          1
          • archosA archos

            @jdaviescoates Thank you very much, I will try to create a bucket in Scaleway, the price is almost the same, so hopefully it will work.

            jdaviescoatesJ Offline
            jdaviescoatesJ Offline
            jdaviescoates
            wrote on last edited by
            #23

            @archos good luck!

            I use Cloudron with Gandi & Hetzner

            1 Reply Last reply
            0
            • archosA archos

              @jdaviescoates Thank you very much, I will try to create a bucket in Scaleway, the price is almost the same, so hopefully it will work.

              jdaviescoatesJ Offline
              jdaviescoatesJ Offline
              jdaviescoates
              wrote on last edited by
              #24

              @archos Note, you also have to do the CORS stuff on Scaleway too, so it won't work without doing that as well,

              see:

              https://www.scaleway.com/en/docs/storage/object/api-cli/setting-cors-rules/

              But before you can do that you need to set-up the AWS CLI stuff, see:

              https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/

              And before you can do that you need API Keys:

              https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/

              Good luck!

              I use Cloudron with Gandi & Hetzner

              archosA 1 Reply Last reply
              1
              • archosA Offline
                archosA Offline
                archos
                wrote on last edited by
                #25

                Thank you very much, it doesn't look very easy but I will try and let you know. Thanks again for your help.

                1 Reply Last reply
                1
                • jdaviescoatesJ jdaviescoates

                  @archos Note, you also have to do the CORS stuff on Scaleway too, so it won't work without doing that as well,

                  see:

                  https://www.scaleway.com/en/docs/storage/object/api-cli/setting-cors-rules/

                  But before you can do that you need to set-up the AWS CLI stuff, see:

                  https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/

                  And before you can do that you need API Keys:

                  https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/

                  Good luck!

                  archosA Offline
                  archosA Offline
                  archos
                  wrote on last edited by
                  #26

                  @jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.

                  Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):
                  
                  

                  I tried to re-create the keys, but to no avail.

                  jdaviescoatesJ 1 Reply Last reply
                  0
                  • archosA archos

                    @jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.

                    Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):
                    
                    

                    I tried to re-create the keys, but to no avail.

                    jdaviescoatesJ Offline
                    jdaviescoatesJ Offline
                    jdaviescoates
                    wrote on last edited by
                    #27

                    @archos said in Backing up media cache:

                    @jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.

                    Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):
                    
                    

                    I tried to re-create the keys, but to no avail.

                    The logs where? Which provider?

                    I use Cloudron with Gandi & Hetzner

                    archosA 1 Reply Last reply
                    0
                    • jdaviescoatesJ jdaviescoates

                      @archos said in Backing up media cache:

                      @jdaviescoates I didn't really understand the instructions, so I tried Wasabi and Idrive again. The logs show this error.

                      Aws::S3::Errors::InvalidAccessKeyId (The AWS Access Key Id you provided does not exist in our records.):
                      
                      

                      I tried to re-create the keys, but to no avail.

                      The logs where? Which provider?

                      archosA Offline
                      archosA Offline
                      archos
                      wrote on last edited by
                      #28

                      @jdaviescoates This is the log from Mastodon when I try to post a photo. It's on every provider I've tried. Yesterday I tried following this exact procedure and created a bucket on AWS and the result was the same.

                      1 Reply Last reply
                      0
                      • archosA Offline
                        archosA Offline
                        archos
                        wrote on last edited by
                        #29

                        I followed the joinmastodon instructions, on the Scaleway repository. Now I'm getting this error in the logs in the application.

                        Aws::S3::Errors::AuthorizationHeaderMalformed (The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'):
                        

                        I chose Paris as the region for the bucket and in env.production. So I really don't know what I'm doing wrong :-(.

                        # Store data using S3 object
                        S3_ENABLED=true
                        S3_BUCKET=mastcz
                        AWS_ACCESS_KEY_ID=ACCESS_KEY_ID
                        AWS_SECRET_ACCESS_KEY=SECRET_ACCESS_KEY
                        #S3_REGION=fr-par
                        S3_PROTOCOL=https
                        S3_HOSTNAME=s3.fr-par.scw.cloud
                        S3_ENDPOINT=https://mastcz.s3.fr-par.scw.cloud
                        
                        jdaviescoatesJ 1 Reply Last reply
                        0
                        • archosA archos

                          I followed the joinmastodon instructions, on the Scaleway repository. Now I'm getting this error in the logs in the application.

                          Aws::S3::Errors::AuthorizationHeaderMalformed (The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'):
                          

                          I chose Paris as the region for the bucket and in env.production. So I really don't know what I'm doing wrong :-(.

                          # Store data using S3 object
                          S3_ENABLED=true
                          S3_BUCKET=mastcz
                          AWS_ACCESS_KEY_ID=ACCESS_KEY_ID
                          AWS_SECRET_ACCESS_KEY=SECRET_ACCESS_KEY
                          #S3_REGION=fr-par
                          S3_PROTOCOL=https
                          S3_HOSTNAME=s3.fr-par.scw.cloud
                          S3_ENDPOINT=https://mastcz.s3.fr-par.scw.cloud
                          
                          jdaviescoatesJ Offline
                          jdaviescoatesJ Offline
                          jdaviescoates
                          wrote on last edited by jdaviescoates
                          #30

                          @archos looks like your S3_ENDPOINT is wrong. For me at least it's just https://s3.fr-par.scw.cloud without the bucket name at the beginning.

                          I use Cloudron with Gandi & Hetzner

                          1 Reply Last reply
                          0
                          • archosA Offline
                            archosA Offline
                            archos
                            wrote on last edited by
                            #31

                            Yes I tried that first and the same result

                            1 Reply Last reply
                            0
                            • archosA Offline
                              archosA Offline
                              archos
                              wrote on last edited by
                              #32

                              Hi, I managed to set up the S3 repository according to this guide. Now the photos are uploading to the storage. I just have a problem with avatars and pictures that were previously uploaded. Even custom emoji are not visible now. Does anyone else have any ideas on how to fix this please?
                              I shouldn't have added the sync command. I am very grateful for your advice and information. Thank you very much.

                              1 Reply Last reply
                              1
                              • nebulonN Away
                                nebulonN Away
                                nebulon
                                Staff
                                wrote on last edited by
                                #33

                                Presumably the app does not automatically re-upload the assets. Maybe upstream has a storage migration script for this? But I haven't seen that so far at least.

                                1 Reply Last reply
                                1
                                • archosA Offline
                                  archosA Offline
                                  archos
                                  wrote on last edited by archos
                                  #34

                                  @nebulon Thank you for your reply. In the instructions it says that first you have to run the command

                                  cd /home/mastodon/live
                                  aws s3 sync --acl public-read public/system/ s3://instance-media --endpoint=https://s3.fr-par.scw.cloud
                                  
                                  

                                  To synchronize local data, cache Mastodon media to the S3 bucket.
                                  I guess that would have to be a different command here on Cloudron.
                                  Now my images load, but the local avatars, server image have disappeared from the instance and the remote images and avatars are not loading. Is it possible that the remote images will load gradually, I don't know.
                                  I've been really struggling with this for a few days now and the result is still not good. 😞

                                  1 Reply Last reply
                                  0
                                  • S Offline
                                    S Offline
                                    sebastienserre
                                    wrote on last edited by
                                    #35

                                    Hello
                                    Did you finally succeed to synchronize your local data ?
                                    I've the same issue, no more avatar for example

                                    1 Reply Last reply
                                    0
                                    • nichu42N Offline
                                      nichu42N Offline
                                      nichu42
                                      wrote on last edited by
                                      #36
                                      This post is deleted!
                                      1 Reply Last reply
                                      0
                                      • doodlemania2D doodlemania2

                                        @jdaviescoates I moved mine based on this article: https://stanislas.blog/2018/05/moving-mastodon-media-files-to-wasabi-object-storage/#setting-up-a-nginx-reverse-proxy-with-cache-for-the-bucket

                                        nichu42N Offline
                                        nichu42N Offline
                                        nichu42
                                        wrote on last edited by
                                        #37

                                        @doodlemania2 Could you please elaborate a bit more on how you managed to get awscli working inside the Cloudron container? I'm having trouble installing it (getting "file system is read-only" errors all the way long).

                                        1 Reply Last reply
                                        0
                                        • nichu42N Offline
                                          nichu42N Offline
                                          nichu42
                                          wrote on last edited by
                                          #38

                                          Bonus question @nebulon: Would you be able to include awscli in the Mastodon package?

                                          1 Reply Last reply
                                          0
                                          Reply
                                          • Reply as topic
                                          Log in to reply
                                          • Oldest to Newest
                                          • Newest to Oldest
                                          • Most Votes


                                          • Login

                                          • Don't have an account? Register

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Bookmarks
                                          • Search