Mike Perham mperham

Sidekiq can use Dragonfly, an open source datastore compatible with Redis.

By default, Sidekiq connects localhost:6379. Since Dragonfly (hereafter, DF) is 100% compatible with Redis clients, your configuration will look identical to Redis configuration.

Using an ENV variable

You can configure DF's location using an environment variable. The easiest option is to set REDIS_URL; Sidekiq will pick it up and use it. A Redis URL looks like redis://[hostname]:[port]/[dbnumber], e.g. redis://redis.example.com:7777/11.

This benchmark creates 100,000 jobs in Redis, boots Sidekiq with 10 threads and waits for all jobs to be processed. This is not a microbenchmark -- it is a "macrobenchmark" designed to test real system throughput.

Code is at https://github.com/mperham/sidekiq/blob/main/bin/sidekiqload

❯ bundle exec bin/sidekiqload
truffleruby 21.3.0, like ruby 2.7.4, GraalVM CE Native [x86_64-darwin]
2022-01-07T16:56:17.644Z pid=40281 tid=29xt INFO: Booting Sidekiq 6.3.2 with redis options {:db=>13, :port=>6380}
2022-01-07T16:56:36.557Z pid=40281 tid=29xt ERROR: Created 100000 jobs

	$ bin/sidekiqload
	ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux]
	THREADS: nil, LATENCY: nil, AJ: nil, PROFILE: nil
	2025-01-26T00:12:21.884Z pid=95261 tid=20wl WARN: Setup RSS: 40624
	2025-01-26T00:12:23.099Z pid=95261 tid=20wl WARN: Created 500000 jobs in 1.212474893 sec
	2025-01-26T00:12:23.099Z pid=95261 tid=20wl WARN: Starting load
	2025-01-26T00:12:23.099Z pid=95261 tid=20wl WARN: Simulating 0ms of latency between Sidekiq and redis
	2025-01-26T00:12:46.891Z pid=95261 tid=212t class=LoadWorker jid=c4e151f78a1609db1da5dd8b WARN: 2025-01-25 16:12:46 -0800 Done 100000
	2025-01-26T00:13:10.754Z pid=95261 tid=211p class=LoadWorker jid=30952445f24862cb39df0884 WARN: 2025-01-25 16:13:10 -0800 Done 200000
	2025-01-26T00:13:34.573Z pid=95261 tid=211h class=LoadWorker jid=4e1e0e6ca79f224aca4980fc WARN: 2025-01-25 16:13:34 -0800 Done 300000

	## Setup

	MacBookPro, M1 Pro w/ 32GB RAM, Sequoia 15.0.1
	Ruby 3.3.5
	Sidekiq 7.3.5
	Solid Queue 1.0.2
	Rails 8.0.0

	## Results

	#!/usr/bin/env ruby

	require 'sidekiq'
	require 'sidekiq/cli'

	# Default to running one process per core
	def process_count
	return ENV['SIDEKIQ_COUNT'].to_i unless ENV['SIDEKIQ_COUNT'].to_i == 0

	require 'etc'

	require "benchmark/ips"
	require "concurrent"

	class MutexFixnum
	def initialize
	@value = 0
	@lock = Mutex.new
	end

	def increment(amount = 1)

	irb(main):004> OpenSSL
	(irb):4:in `<main>': uninitialized constant OpenSSL (NameError)
	Did you mean? OpenStruct
	from /Users/mperham/.gem/ruby/3.2.0/gems/irb-1.10.1/exe/irb:9:in `<top (required)>'
	from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `load'
	from /Users/mperham/.gem/ruby/3.2.0/bin/irb:25:in `<main>'
	irb(main):005> ENV["FAKTORY_URL"] = "tcp+tls://test.contribsys.com:7419"
	=> "tcp+tls://test.contribsys.com:7419"
	irb(main):006> f = Faktory::Client.new
	/Users/mperham/src/fwr/lib/faktory/client.rb:318:in `rescue in open_socket': Server not using TLS? Use FAKTORY_URL=tcp://... to disable encryption (RuntimeError)

	require 'benchmark'
	require 'sidekiq-ent/limiter'

	COUNT = 10_000

	Benchmark.bmbm(30) do \|x\|
	x.report("leaky") do
	COUNT.times do \|count\|
	lmt = Sidekiq::Limiter.leaky("leaky_#{count%100}", 10, 10, wait_timeout: 0, policy: :skip)
	lmt.within_limit do

	❯ make runruby
	./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb
	<internal:ractor>:38: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
	#<Thread:0x00007fadd701fa90 run> terminated with exception (report_on_exception is true):
	<internal:ractor>:124:in `take': thrown by remote Ractor. (Ractor::RemoteError)
	from ./test.rb:11:in `<main>'
	./test.rb:5:in `logger': can not access instance variables of classes/modules from non-main Ractors (RuntimeError)
	from ./test.rb:10:in `block in <main>'
	make: *** [runruby] Error 1

	2020-05-08T15:52:22.241Z pid=10710 tid=2me WARN: RuntimeError: boom
	2020-05-08T15:52:22.241Z pid=10710 tid=2me WARN: /Users/mikeperham/src/sidekiq/myapp/app/jobs/some_job.rb:5:in `perform'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/activejob-6.0.3/lib/active_job/execution.rb:40:in `block in perform_now'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/i18n-1.8.2/lib/i18n.rb:313:in `with_locale'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/activejob-6.0.3/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:121:in `instance_exec'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
	/Users/mikeperham/.gem/ruby/2.7.1/gems/activesupport-6.0.3/lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
	/Users/mikeperham/.gem/ruby/2