Redis Persistence: 3 Technical Approaches

What Is Redis Persistence?

Redis persistence refers to the mechanisms Redis uses to store data on disk, allowing recovery after a restart or failure. While Redis is an in-memory data store, persistence ensures that data is not lost when the server stops or crashes. It enables Redis to reload datasets into memory when it restarts, making it suitable for scenarios where durability is important.

Redis offers two main persistence options—RDB (Redis Database) snapshots and AOF (Append-Only File) logging—as well as a hybrid approach that combines both. Each method has its own trade-offs in terms of performance, durability, and recovery time.

Method #1: RDB (Redis Database) Snapshots

RDB persistence creates point-in-time snapshots of the dataset at specified intervals. These snapshots are saved to a binary file (dump.rdb), which can be loaded when Redis restarts.

Redis performs a fork() of the current process, creating a child process. This child process writes the snapshot of the dataset to disk, allowing the parent process to continue serving requests. Snapshot creation is configured using SAVE or BGSAVE commands or via settings in the Redis config file (e.g., save 900 1 means save every 900 seconds if at least 1 key changed).

Pros and Cons

Pros

• Performance: Lower impact (compared to AOF) on the main process during snapshotting due to the use of fork().
• Compact Format: The RDB file is in binary format and thus smaller and quicker to load compared to AOF, as we will explore below.
• Good for Backups: Simple to transfer or archive.

Cons

• Data Loss Risk: Only captures data as of the last snapshot. Any changes after that point are lost if Redis crashes.
• Fork Overhead: Can be resource-intensive for large datasets and write-heavy Redis instances, as fork() may need to duplicate a lot of memory during snapshotting. In worst-case scenarios, it can even cause the main Redis process to be killed.

Quick Tutorial

1. To configure RDB snapshots, edit your Redis configuration file (redis.conf) and set save-point rules using the save directive. For example:

save 900 1 300 10 60 10000

This setup tells Redis to create a snapshot (dump.rdb) if at least:

• 1 data change in 900 seconds
• 10 data changes in 300 seconds
• 10,000 data changes in 60 seconds

2. You can trigger snapshot creation manually using the following Redis commands:

• SAVE: Creates a snapshot synchronously. This blocks the server during the save process.
• BGSAVE: Creates a snapshot asynchronously using a child process, allowing the server to continue handling requests during the save.

3. Once the snapshot completes, Redis writes the dataset to a file, which is called dump.rdb by default in the configured data directory.

4. For backups, you can safely copy the dump.rdb file while Redis is running, since Redis writes snapshots to a temporary file and renames it only after the save is complete. This ensures you never copy a partially written snapshot.

5. A good backup strategy is to create cron jobs that archive RDB files at regular intervals (e.g., hourly and daily) and move them off-server for disaster recovery. For example, the following describes a viable cron strategy:

• Hourly snapshots saved for 48 hours
• Daily snapshots saved for 30 days
• Regular offsite transfer using tools like scp or cloud storage like Amazon S3

This ensures you always have restorable backups, even in case of hardware failures or data center issues. Although it requires additional development and operational efforts.

Method #2: AOF (Append-Only File)

AOF persistence logs every write operation received by the server to an append-only log file. On restart, Redis replays this log to reconstruct the dataset.

Each command that modifies the dataset is logged to the AOF file. Redis can be configured to fsync this log:

• always: ensures the highest data safety by forcing a full fsync after every batch of commands (from multiple clients or a pipeline) is written to the AOF file. While this is the most durable option, it significantly impacts performance.
• everysec: commands are logged to AOF every second, although you may lose 1 second of data if there is a disaster.
• no: delegates flushing to the OS, making it the fastest but least safe method. Linux typically flushes data every 30 seconds, but this relies entirely on the kernel's tuning.

Redis can rewrite the AOF file in the background to shrink its size by keeping only the latest write to a key, for instance. This can be done by utilizing the auto-aof-rewrite-* directives in redis.conf or by manually calling the BGREWRITEAOF command.

Pros and Cons

Pros

• More Durable: Minimal data loss, especially with appendfsync=always or appendfsync=everysec.
• Easier to Understand & Debug: The AOF file is human-readable.
• Recoverable from Recent Changes: Better suited for applications that can't tolerate data loss of large gaps.

Cons

• Slower than RDB: Can have more disk I/O overhead.
• File Size Growth: Tends to grow faster than RDB unless rewrites are configured.
• Longer Startup: Replaying commands can be slower than loading an RDB snapshot.

Quick Tutorial

1. To enable AOF persistence, open your Redis configuration file (redis.conf) and set the following option:

appendonly yes

2. This setting turns on the append-only file feature. Once enabled, Redis will start logging every write operation to the AOF file.

You can also control how often Redis syncs AOF data to disk using the appendfsync setting:

• appendfsync always: Redis will sync the AOF file to disk after every write command. This is the safest but slowest option.
• appendfsync everysec (default): Redis will sync the AOF file once per second, offering a balance between performance and durability.
• appendfsync no: Redis relies on the operating system to flush data, which is fast but least durable.

3. If your AOF file grows too large, Redis can rewrite it automatically in the background. This process creates a minimal, compact version of the file without interrupting client service. By using the two configurations below, a rewrite is triggered if the AOF file exceeds a specified size and its growth since the last rewrite surpasses a defined percentage (100% in the example). The minimum size configuration prevents rewrites when the file is still small. Thus, when both conditions are met, an automatic AOF rewrite will be performed.

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Note that you can also trigger an AOF rewrite manually using the BGREWRITEAOF command.

4. To monitor the status of AOF operations, use the INFO persistence command, which shows useful fields like aof_rewrite_in_progress and aof_last_bgrewrite_status, which help track ongoing or recently completed AOF rewrites.

5. When backing up AOF files, especially in Redis 7.0 and above (where AOF files are split into base and incremental parts), disable automatic rewrites during the backup process to avoid inconsistencies:

$> CONFIG SET auto-aof-rewrite-percentage 0

6. After copying the AOF files, remember to re-enable auto-rewrites:

$> CONFIG SET auto-aof-rewrite-percentage "<previous-value>"

This ensures your AOF backups remain consistent and recoverable.

Method #3: Hybrid Approach (RDB + AOF)

Starting with v5.0, Redis supports enabling both RDB and AOF persistence simultaneously to combine their benefits. Redis first creates an RDB snapshot as a base file and then appends all commands received after that snapshot into a separate incremental AOF file. Upon restart, Redis restores state by loading the base file and then applying the incremental AOF commands to restore state up to the last write. In order to use hybrid mode, ensure the following configurations are set properly in your redis.conf:

# Enable AOF persistence.
appendonly yes

# Use RDB mode for the AOF base file.
aof-use-rdb-preamble yes

Pros and Cons

Pros

• High Durability with Fast Recovery: Incremental AOF provides frequent durability, and the RDB base file gives a fallback.
• Backup and Resilience: RDB can act as a fail-safe if the incremental AOF is corrupted.
• Balanced Approach: Mixes performance and durability.

Cons

• Increased Resource Usage: Requires more disk space and I/O.
• More Complex Setup: Needs careful configuration and monitoring to balance both systems effectively.

Which Redis Persistent Approach is Right for You?

Here is a comparison table summarizing the Redis persistence methods—RDB, AOF, and the Hybrid Approach—across key dimensions:

Dragonfly: The Next-Generation In-Memory Data Store

Dragonfly is a modern, source-available, multi-threaded, Redis-compatible in-memory data store that stands out by delivering unmatched performance and efficiency. Designed from the ground up to disrupt existing legacy technologies, Dragonfly redefines what an in-memory data store can achieve. With Dragonfly, you get the familiar API of Redis without the performance bottlenecks, making it an essential tool for modern cloud architectures aiming for peak performance and cost savings. Migrating from Redis to Dragonfly requires zero or minimal code changes.

At the time of writing, Dragonfly doesn't support AOF persistence yet as we prioritize extreme performance. However, Dragonfly utilizes a faster and more stable snapshotting mechanism, which you can read more about in our blog post.

Key Advancements of Dragonfly

• Multi-Threaded Architecture: Efficiently leverages modern multi-core processors to maximize throughput and minimize latency.
• Unmatched Performance: Achieves 25x better performance than Redis, ensuring your applications run with extremely high throughput and consistent latency.
• Cost Efficiency: Reduces hardware and operational costs without sacrificing performance, making it an ideal choice for budget-conscious enterprises.
• Redis API Compatibility: Offers seamless integration with existing applications and frameworks running on Redis while overcoming its limitations.
• Innovative Design: Built to scale vertically and horizontally, providing a robust solution for rapidly growing data needs.

Dragonfly Cloud is a fully managed service from the creators of Dragonfly, handling all operations and delivering effortless scaling so you can focus on what matters without worrying about in-memory data infrastructure anymore.