Announcing Dragonfly 2.0

Yesterday at Dragonfly Ascent, I announced Dragonfly 2.0. Today I want to go over what I covered at our first user conference, and the improvements I talked about on stage. You can watch the recording here.

Dragonfly 2.0 marks a significant milestone in maturity, performance, and production readiness since we released 1.0. There are no major new features since the last minor v1.40.0 release, so the versioning represents that Dragonfly has come a long way from v1.0.0 in 2023, through many small improvements and by constantly thinking through the details. If you haven’t tried Dragonfly recently, please check it out again.

Today I’ll cover what has changed since v1.0.

v1.0 was reaching API parity. v2.0 was hardening for scale.

At first, we had to make Dragonfly compatible with Redis to make it useful. We started with the most common use cases. We supported the core APIs with the Resp2 protocol. At the time, we inherited many choices from Redis that we would later improve.

That hardening is about solving the last 10% that requires 90% of the effort. The way for us to do this was by running our own managed service and a growing production fleet. We learned a lot along the way with real customer use cases in production, and put those improvements back into the project. Dragonfly 2.0 now has support across more than 30 Redis frameworks in all the common languages. We have cut incidents in our production by a factor of 10X. We’ve grown the size of our largest data stores by three orders of magnitude with the help of Dragonfly Swarm. Along the way, we’ve worked through 5,000 community issues to help our users and create features that the community wants.

Multi-threading benefits more from the advances in infrastructure

Hardware innovation has been accelerating the whole time. The compute and memory available in a single server has grown roughly 50 times since 2009, when Redis started, and it has roughly tripled since 2022, when we launched Dragonfly. Single-threaded software cannot use that extra capacity. Other prominent projects reach similar conclusions and attempt to break out the limitations of the original single-thread design. See, for example, the python GIL removal effort, or Bun using worker threads for many of its features. 

Dragonfly was designed from the start to extract every drop of performance out of a modern server, so every hardware generation makes its lead larger. One of our customers now runs Dragonfly on nodes with 224 vCPUs and 3 TB of memory each. They can do that because Dragonfly serves, replicates, and manages data at that scale without a cluster which they could not use. Yes, vertical scale has obvious hardware limits (however large they are today), so we built Dragonfly Swarm to carry workloads into the multi-terabyte range. Dragonfly Swarm on Dragonfly Cloud manages workloads with dozens of shards each holding 200-400GB of data.
It employs intelligent shard rebalancing, autoscaling, read-replicas support, zone-affinity reads, cloud native snapshotting and many other features helping with predictable and stable production environments.

Performance improvements

It wasn’t one big change that led to the performance improvements for Dragonfly, but instead careful attention to many small ones. While this list isn’t all of them, it highlights many of the more important changes.

Reworked I/O and pipelining give 54% more throughput and 35% lower avg. latency

The I/O networking path was one of the more complicated areas. Optimizing it produced much higher throughput for pipelined requests. On a c7gn.8xlarge, comparing the two versions, throughput increased 54% and average latency dropped 35%, from 0.96 ms to 0.61 ms. The p99 latency went from 1.46 ms to 1.12 ms.

That is not the whole story, though. As often happens while shipping bug fixes and features, we had gradually degraded some of the raw performance we started with in 1.0. So we fixed the regressions but also redesigned the networking path to push past the previous 1.0 performance and avoid such regressions in the future.

Reduced latency in pipelines

In single-threaded Redis, a thread reads a request, executes it, and replies. There is no intrinsic latency for this beyond CPU time. In Dragonfly's multi-threaded, shared-nothing architecture, it is not that simple. Our threads communicate with messages, similar to RPCs, so each command can carry 10 to 20 microseconds of latency within the Dragonfly process. With a naive pipelining implementation that grows linearly, so a pipeline of 100 commands could run 100 times slower. In 1.0 we had a partial remedy but in 2.0 we solved it with an intelligent batching and squashing component that amortizes that latency across multiple commands in the pipeline.

Enhanced controls to optimize response rendering path

The response rendering framework was another part of the problem, and it caused some of the regressions above. We rebuilt it to satisfy two requirements that pull in opposite directions: buffer many small (pipelined) responses into a single call, but use zero-copy vectorized I/O for large ones. Think of an HGETALL, or a GET against a large string, where you want to avoid buffering and avoid redundant copies and allocations. Knowing what to do was not the hard part. The hard part was an internal API for the reply builder that allows choosing the optimal strategy for responses.

New data structures and removing the expiry table cut memory 30-40%

We redesigned the core data structures, including sets, sorted sets, and hash maps. More recently we removed the dedicated expiry table that Dragonfly inherited from Redis, which cut the overhead for items with a TTL. We did it with key encodings that use pointer tagging to embed TTL metadata directly in the key. Across common workloads, that now requires up to 30 to 40% less memory than 1.0. Memory is usually the largest line item in these workloads, so a smaller footprint means a smaller bill.

Streaming serialization removes the snapshot memory spike

Taking a snapshot used to spike memory by the size of your largest collection. An 8 GB sorted set meant an 8 GB spike and an out-of-memory risk unless you over-provisioned memory for it. Dragonfly originally shipped with its own snapshotting algorithm, which helped, but was not enough for very large collections. In 2.0 we added streaming support to the serialization format. Replicating Bloom filters, sets, hash maps, and lists now uses predictable memory, with no spike at all, even on massive collections.

Dragonfly APIs grew from 206 commands to 308

The API is where most of the compatibility work went. Since 1.0, we added vector and hybrid search, JSON, ACL, Geo, HyperLogLog, and probabilistic structures including Bloom, CMS, and TopK. We support out of the box all prominent AI frameworks like LangGraph, LangChain and others.

SSD tiering now holds up to 8X more memory

SSD tiering may increase the effective capacity of a Dragonfly data store by up to 8X. Keys and metadata stay in memory while big values move to local NVMe, so Dragonfly can manage datasets larger than your RAM capacity. Dragonfly Community is the only Redis-compatible technology that ships SSD data tiering in its community edition, and SSD data tiering recently became available in Dragonfly Cloud as well.

Dragonfly adoption has grown thanks to you

These improvements only matter because you use them. Community downloads per version have grown from 7,000 at 1.0 to roughly 900,000, which is 128 times more. We have crossed 25 million total downloads, merged about 5,000 commits, and worked with 175 contributing authors. 

C++ is not an easy language and this codebase is demanding, so we see fewer contributors than a typical Node.js or Python project. That makes the people who put in the time matter more. In 2026, we also saw the shape of that contribution changed. Before this year, most community help was filing issues and helping us debug. Now AI lets developers who do not write C++ run root cause analysis in their own environments and often send the fix themselves. 

The organizations deploying Dragonfly are growing too, and some of them are among the most technically demanding in the world. Instacart runs a feature store in the tens of terabytes on Dragonfly Swarm, serving ads in real time, with 70% fewer nodes and 50% lower latency. Hugging Face moved over after their Redis nodes were routinely overheating. xAI runs their in-memory workloads on Dragonfly Swarm because it handles their scale.

AI has accelerated how fast companies are scaling

We built Dragonfly for the scale that went beyond what Redis could deliver. Now, teams are hitting these limits sooner and more often than ever before. AI workloads create more traffic for reads and writes and require larger datasets. Delivering that scale is what the last three years of work were about. The throughput and efficiency gains, Dragonfly Swarm, and SSD data tiering all exist to deliver on that promise. Back in 2022 we didn’t know AI would be the reason, but we did know that the need for data would only continue to grow. 

If you tried Dragonfly a year or two ago and have not looked since, try it again now! We have a new free trial. Or check out the 2.0 release notes. We’d love to hear from you, and discuss the things you’d like to see us build next.