Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Mastodon
  3. Run Additional Sidekiqs

Run Additional Sidekiqs

Scheduled Pinned Locked Moved Solved Mastodon
6 Posts 3 Posters 1.3k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • doodlemania2D Offline
    doodlemania2D Offline
    doodlemania2
    App Dev
    wrote on last edited by girish
    #1

    I think I may have gotten to a threshold of sorts. The sidekiq running in my Mastodon instance isn't able to keep up. Top shows it running in the container no problem, but apparently I need more than one instance to get some parallelism rocking. Top also showed the container wasn't pegged at 100% CPU, so it had room to grow. I also bumped up the Resources from default 50% to 75%.

    So, I ended up with about a 6 hr backlog (around 40,000 messages) in my queues. In order to get them to clear, I opened up two terminal instances and manually ran bundle exec sidekiq in each and that chewed through them pretty quickly. Once the queues were drained, I closed them out but it started building back up again - the default sidekiq is definitely running, just a lot slower than needed.

    So, wondering if we could have a config option to ramp additional sidekiqs up?

    jdaviescoatesJ 1 Reply Last reply
    2
    • doodlemania2D doodlemania2

      I think I may have gotten to a threshold of sorts. The sidekiq running in my Mastodon instance isn't able to keep up. Top shows it running in the container no problem, but apparently I need more than one instance to get some parallelism rocking. Top also showed the container wasn't pegged at 100% CPU, so it had room to grow. I also bumped up the Resources from default 50% to 75%.

      So, I ended up with about a 6 hr backlog (around 40,000 messages) in my queues. In order to get them to clear, I opened up two terminal instances and manually ran bundle exec sidekiq in each and that chewed through them pretty quickly. Once the queues were drained, I closed them out but it started building back up again - the default sidekiq is definitely running, just a lot slower than needed.

      So, wondering if we could have a config option to ramp additional sidekiqs up?

      jdaviescoatesJ Offline
      jdaviescoatesJ Offline
      jdaviescoates
      wrote on last edited by
      #2

      @doodlemania2 have you seen:

      https://docs.cloudron.io/apps/mastodon/#scaling
      https://docs.joinmastodon.org/admin/scaling/

      ? ๐Ÿ™‚

      I've not hit the need yet, but I think all you need to do is adjust the figures in /app/data/config.sh

      #!/bin/bash
      
      # Setup scaling related environment variables here - https://docs.joinmastodon.org/admin/scaling/
      
      # Puma
      export WEB_CONCURRENCY=2    # number of worker processes
      export MAX_THREADS=5        # the number of threads per process
      
      # Streaming API
      export STREAMING_CLUSTER_NUM=1  # number of worker processes
      
      # Sidekiq
      export SIDEKIQ_THREADS=2
      export DB_POOL=25               # must be at least the same as the number of threads
      

      I use Cloudron with Gandi & Hetzner

      doodlemania2D 1 Reply Last reply
      2
      • jdaviescoatesJ jdaviescoates

        @doodlemania2 have you seen:

        https://docs.cloudron.io/apps/mastodon/#scaling
        https://docs.joinmastodon.org/admin/scaling/

        ? ๐Ÿ™‚

        I've not hit the need yet, but I think all you need to do is adjust the figures in /app/data/config.sh

        #!/bin/bash
        
        # Setup scaling related environment variables here - https://docs.joinmastodon.org/admin/scaling/
        
        # Puma
        export WEB_CONCURRENCY=2    # number of worker processes
        export MAX_THREADS=5        # the number of threads per process
        
        # Streaming API
        export STREAMING_CLUSTER_NUM=1  # number of worker processes
        
        # Sidekiq
        export SIDEKIQ_THREADS=2
        export DB_POOL=25               # must be at least the same as the number of threads
        
        doodlemania2D Offline
        doodlemania2D Offline
        doodlemania2
        App Dev
        wrote on last edited by
        #3

        @jdaviescoates ohhhh! doinks - thanks sir

        doodlemania2D 1 Reply Last reply
        1
        • doodlemania2D doodlemania2

          @jdaviescoates ohhhh! doinks - thanks sir

          doodlemania2D Offline
          doodlemania2D Offline
          doodlemania2
          App Dev
          wrote on last edited by
          #4

          that did the trick nicely!

          A 1 Reply Last reply
          2
          • doodlemania2D doodlemania2

            that did the trick nicely!

            A Offline
            A Offline
            alwynispat
            wrote on last edited by
            #5

            @doodlemania2 whatโ€™s your setup now?
            here's mine:

            #!/bin/bash
            
            # Setup scaling related environment variables here - https://docs.joinmastodon.org/admin/scaling/
            
            # Puma
            export WEB_CONCURRENCY=3    # number of worker processes
            export MAX_THREADS=10        # the number of threads per process
            
            # Streaming API
            export STREAMING_CLUSTER_NUM=3  # number of worker processes
            export DB_POOL=15
            
            # Sidekiq
            export SIDEKIQ_THREADS=75
            export DB_POOL=75               # must be at least the same as the number of threads
            export MALLOC_ARENA_MAX=2
            
            
            

            Find me at Mastodon

            doodlemania2D 1 Reply Last reply
            1
            • A alwynispat

              @doodlemania2 whatโ€™s your setup now?
              here's mine:

              #!/bin/bash
              
              # Setup scaling related environment variables here - https://docs.joinmastodon.org/admin/scaling/
              
              # Puma
              export WEB_CONCURRENCY=3    # number of worker processes
              export MAX_THREADS=10        # the number of threads per process
              
              # Streaming API
              export STREAMING_CLUSTER_NUM=3  # number of worker processes
              export DB_POOL=15
              
              # Sidekiq
              export SIDEKIQ_THREADS=75
              export DB_POOL=75               # must be at least the same as the number of threads
              export MALLOC_ARENA_MAX=2
              
              
              
              doodlemania2D Offline
              doodlemania2D Offline
              doodlemania2
              App Dev
              wrote on last edited by
              #6

              @alwynispat you've got DB_POOL specified twice ๐Ÿ™‚
              I'm doing:
              be945764-0726-41d0-8886-d387d17c123a-image.png

              1 Reply Last reply
              2
              • nebulonN nebulon marked this topic as a question on
              • nebulonN nebulon has marked this topic as solved on
              Reply
              • Reply as topic
              Log in to reply
              • Oldest to Newest
              • Newest to Oldest
              • Most Votes


              • Login

              • Don't have an account? Register

              • Login or register to search.
              • First post
                Last post
              0
              • Categories
              • Recent
              • Tags
              • Popular
              • Bookmarks
              • Search