Solved Run Additional Sidekiqs
I think I may have gotten to a threshold of sorts. The sidekiq running in my Mastodon instance isn't able to keep up. Top shows it running in the container no problem, but apparently I need more than one instance to get some parallelism rocking. Top also showed the container wasn't pegged at 100% CPU, so it had room to grow. I also bumped up the Resources from default 50% to 75%.
So, I ended up with about a 6 hr backlog (around 40,000 messages) in my queues. In order to get them to clear, I opened up two terminal instances and manually ran
bundle exec sidekiqin each and that chewed through them pretty quickly. Once the queues were drained, I closed them out but it started building back up again - the default sidekiq is definitely running, just a lot slower than needed.
So, wondering if we could have a config option to ramp additional sidekiqs up?
jdaviescoates last edited by
@doodlemania2 have you seen:
I've not hit the need yet, but I think all you need to do is adjust the figures in
#!/bin/bash # Setup scaling related environment variables here - https://docs.joinmastodon.org/admin/scaling/ # Puma export WEB_CONCURRENCY=2 # number of worker processes export MAX_THREADS=5 # the number of threads per process # Streaming API export STREAMING_CLUSTER_NUM=1 # number of worker processes # Sidekiq export SIDEKIQ_THREADS=2 export DB_POOL=25 # must be at least the same as the number of threads
@jdaviescoates ohhhh! doinks - thanks sir
that did the trick nicely!
@doodlemania2 what’s your setup now?
#!/bin/bash # Setup scaling related environment variables here - https://docs.joinmastodon.org/admin/scaling/ # Puma export WEB_CONCURRENCY=3 # number of worker processes export MAX_THREADS=10 # the number of threads per process # Streaming API export STREAMING_CLUSTER_NUM=3 # number of worker processes export DB_POOL=15 # Sidekiq export SIDEKIQ_THREADS=75 export DB_POOL=75 # must be at least the same as the number of threads export MALLOC_ARENA_MAX=2
@alwynispat you've got DB_POOL specified twice