Sidekiq can use Dragonfly, an open source datastore compatible with Redis.

By default, Sidekiq connects `localhost:6379`. Since Dragonfly (hereafter, DF) is 100% compatible with Redis clients, your configuration will look identical to Redis configuration.

## Using an ENV variable

You can configure DF's location using an environment variable.
The easiest option is to set `REDIS_URL`; Sidekiq will pick it up and use it.
A Redis URL looks like `redis://[hostname]:[port]/[dbnumber]`, e.g. `redis://redis.example.com:7777/11`.

## Using an initializer

Prefer to use the ENV variables above.
If you have to use logic in Ruby, you can configure Redis in the initializer.
Note that to configure the location of Redis, you must define *both* the `Sidekiq.configure_server` and `Sidekiq.configure_client` blocks.
To do this put the following code into `config/initializers/sidekiq.rb`.  

```ruby
Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://redis.example.com:7372/0' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://redis.example.com:7372/0' }
end
```
Unknown parameters are passed to the underlying Redis client so any parameters supported by the driver can go in the Hash.
Keys can be strings or symbols and will be unified before being passed on to the Redis client.

## Life in the Cloud

One problem with cloud-based systems like EC2 and Heroku can be multi-tenancy and unpredictable network performance.
You can tune your network and pool timeouts to be a little more lenient if you are seeing occasional timeout errors, it defaults to 1 second.

```ruby
config.redis = { url: 'redis://...', network_timeout: 5, pool_timeout: 5 }
```

**REMEMBER: THIS IS A BANDAID** You are not solving the actual cause of the slow performance.

If you are seeing Redis timeout errors, you should check your Redis latency by using the `redis-cli` `--latency` and `--latency-history` flags:

```
$ redis-cli --latency-history localhost
min: 0, max: 1, avg: 0.20 (1359 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.18 (1356 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.18 (1355 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.17 (1359 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.19 (1358 samples) -- 15.00 seconds range
min: 0, max: 1, avg: 0.18 (1353 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.19 (1357 samples) -- 15.01 seconds range
```

This says my average latency to localhost is 0.2ms or 200 microseconds: excellent.  With users seeing odd Redis behavior, I regularly see setups with latency over 5 seconds: terrible.  You can move to a different Redis provider or run your own Redis server on a dedicated machine but there's nothing Sidekiq can do if the network performance is terrible.  Contact your Redis provider and ask about your available options.

## Lockdown

If your IT organization insists on locking down Sidekiq's Redis user, you can use this configuration as a starting point:

> acl setuser sidekiquser +@all -@admin -@dangerous +info allkeys

I believe those restrictions work with all Sidekiq functionality today.

## Client naming

If your Redis server supports it, you may instruct Sidekiq to use `CLIENT SETNAME` when establishing connections. This will make it easier to identify Sidekiq clients when debugging with `CLIENT LIST`.

```ruby
config.redis = { id: "Sidekiq-server-PID-#{::Process.pid}" }
```

This is disabled by default because many SaaS Redis hosts do not support it.

## Architecture

Redis offers many different topologies:

* Single node -- offers no fault tolerence
* Redis Sentinel -- offers fault tolerence, fails over to a replica in case of primary failure
* Redis Cluster -- multi-master keyspace spread across many instances

Cluster is designed for large-scale datasets, like caches, that can spread evenly across machines.  **Cluster is NOT appropriate for Sidekiq** as Sidekiq has a few very hot keys which are constantly changing (aka queues) and Cluster cannot guarantee high-performance transactions, necessary to keep your job system fast and consistent. I recommend using Sentinel or use a Redis SaaS which has built-in support for failover.

## Tuning

You can see Redis's config variables with the command `redis-cli info`.

## Scale

Customers have reported processing 20,000+ jobs/sec with a single Redis instance.
With multiple shards, I have gotten reports of 250,000 jobs/sec.

## Memory

Redis runs best when all data fits in memory.
You should set `maxmemory-policy noeviction` in redis.conf so Redis doesn't drop Sidekiq's data silently.

## Multiple Redis instances

Many people use [Redis as a cache](http://redis.io/topics/lru-cache) (it works great as a [Rails cache store](http://guides.rubyonrails.org/caching_with_rails.html#activesupport-cache-rediscachestore)) but it's important that Sidekiq be run against a Redis instance that is **not** configured as a cache but as a persistent store.
I recommend using two separate Redis instances, each configured appropriately, if you wish to use Redis for caching and Sidekiq.
Redis namespaces do not allow for this configuration and come with [many other problems](http://www.mikeperham.com/2015/09/24/storing-data-with-redis/), so using discrete Redis instances is always preferred.

## Timeouts

The most common reasons for Redis networking timeouts are:

0. CPU overload - you've configured your concurrency way too high and threads are starving for CPU time. Reduce your concurrency, create more processes.
1. swapping - you are running out of RAM and disk swapping is causing massive latency spikes.  Using Redis to hold cache data can take a lot of RAM, configure Sidekiq to use a separate Redis instance if necessary.
2. command latency - you are running a Redis command which is taking a large amount of time.  Read the Monitoring blog post below.

## Notes

* `redis-cli --help` shows several useful options, including `--latency-history` and `--bigkeys`.
* [redis-rdb-tools](https://github.com/sripathikrishnan/redis-rdb-tools) has a few options to introspect a dumped RDB file.
* [Monitoring Redis](http://www.mikeperham.com/2017/04/20/monitoring-redis/) covers the most important things you should monitor when using Redis.
* [Datadog has a nice whitepaper on Redis internals and monitoring](https://www.datadoghq.com/pdf/Understanding-the-Top-5-Redis-Performance-Metrics.pdf).
* BRPOPLPUSH and other B* commands are **blocking**; large latency spikes with them are **normal** and not a cause for concern.

Previous: [[Best Practices]] Next: [[Error Handling]]