Redis by default sets `appendonly no`, `maxmemory-policy noeviction`, and `save 3600 1 300 100 60 10000`. Many clients I work for need to learn about this. The default Redis configuration can grow out of memory or stop writing.
In docker, for Sidekiq, this becomes especially painful during heavy load as it crashes and requires health checks and a restart policy. If you process money transactions, you might lose people's money.
The great thing about Redis is that it doesn't have a configuration file; it runs preconfigured with sensible defaults.
This means we only need to provide the configuration we want to override, like a docker-compose override.
Redis has two persistence modes that can be individually configured: AOF and RDB.
RDB - Redis Data Persistence
Let's talk about RDB first. The default:
save 3600 1 300 100 60 10000
The numbers are pairs. The first number (3600) is the frequency in seconds (every 1 hour) we save if we have one key change. Then, we save every 300 seconds (five minutes) if we have 60 key changes. Lastly, we save every minute if we have 10,000 key changes.
RDB is a snapshot in time containing all the data at that point. This is a potentially lossy approach. It should be considered the same as a PostgreSQL backup.
AOF - Append Only
AOF can be thought of as event sourcing or log stream. It writes all commands to the log, which can be read back from the beginning. Writing every command immediately might be a little extreme (performance-wise), but Redis already has our back with sensible defaults.
First, we need to turn on append-only mode, and that's done with `appendonly yes`. Next, it is essential to know about `appendfsync everysec`. It can be configured in different ways, but for processing Sidekiq queues, this should be a great start.
Data directory
This is important; we want a super-fast volume for the Redis database. Preferably, both here and for the PostgreSQL database, we want NVMe disks. SSDs work, too, but they are a little slower.
The data directory is configured with `dir /data` or `dir /redis-data`.
Final redis.conf (1st attempt)
lang-conf # /usr/local/etc/redis/redis.conf appendonly yes appendfsync everysec save 1500 1 300 100 60 1000 dir /data
This is combining what we talked about before.
Deployment
We have several ways to use this configuration override. Let's inspect what redis-server says.
➜ redis-server --help 01/05/24 - 12:18 PM
Usage: ./redis-server [/path/to/redis.conf] [options] [-]
./redis-server - (read config from stdin)
./redis-server -v or --version
./redis-server -h or --help
./redis-server --test-memory <megabytes>
./redis-server --check-system
Examples:
./redis-server (run the server with default conf)
echo 'maxmemory 128mb' | ./redis-server -
./redis-server /etc/redis/6379.conf
./redis-server --port 7777
./redis-server --port 7777 --replicaof 127.0.0.1 8888
./redis-server /etc/myredis.conf --loglevel verbose -
./redis-server /etc/myredis.conf --loglevel verbose
Sentinel mode:
./redis-server /etc/sentinel.conf --sentinelThe easiest, therefore would be to just att these new settings as command line arguments:
lang-yaml
version: '3'
services:
redis:
image: redis
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec", "--save", "900 1 300 10 60 1000", "--dir", "/data"]
volumes:
- redis-data:/data
volumes:
redis-data:There are other alternatives, like building our own custom Dockerfile:
FROM redis:7-alpine COPY docker/redis/redis.conf /usr/local/etc/redis/redis.conf
References