How Instacart Achieved 50% Better Performance with 70% Fewer Nodes

At a Glance

A Feature Store Under Pressure

Delivering a highly personalized experience to every customer is a critical part of Instacart’s business. At the center of that effort is a machine learning architecture and team that has evolved over many years. As the team grew, and the models and serving requirements became more sophisticated, the team decided to implement a feature store in order to achieve better consistency between training and serving as well as to better operationalize and scale their feature pipelines. They implemented feature stores to power use cases such as ad serving, personalization, recommendations and more. 

One of their largest and most critical feature stores was their ad serving feature store. That feature store had grown into a multi-hundred-node deployment spread across multiple clusters. The team had developed a sophisticated architecture around it, including a custom proxy layer, a querying SDK, and a compact "densified vector" storage format that minimized memory footprint. But the underlying infrastructure was hitting its limits in three areas.

The first was instability during cluster mutations. At large cluster sizes, any operation that modified the cluster (scaling up, scaling down, upgrading versions, or migrating hardware) caused prolonged periods of degraded performance. The team traced part of the issue to their existing solution’s reliance on DNS for cluster discovery, but the core problem was simply that the bigger the cluster grew, the more fragile it became during changes.

The second was tail latency due to fan-out. To work around the single-threaded engine lock, the team used smaller instance types, which meant data was spread across more nodes. A single slow node could drag the entire request. 

The third was cost. The team split the workload into multiple clusters to manage stability, but this undermined the cost efficiency they gained from multi-tenancy. Running dedicated read replicas for high-read workloads like ads was expensive, and long-term cost projections were heading in the wrong direction.

Discovering Dragonfly - A New Scaling Paradigm

As the team started researching alternatives, they discovered Dragonfly, and were excited about the potential of its multi-threaded, shared nothing architecture. As they started to dig into the source code on GitHub, they began to envision a solution in which they could scale vertically on each node, allowing them to take advantage of the much larger instances offered. This had the potential to dramatically reduce their cluster size while delivering far better performance from those smaller clusters. The reduction in fan-out could also result in a much more stable, simpler to operate system with lower overall costs.

Evaluation and Migration

Instacart's existing proxy and load-balancing infrastructure, originally built to support multiple clusters, made the evaluation simple to set up. The team stood up a Dragonfly cluster side by side with their existing deployment, backfilled it with identical data using their standard pipeline tooling, and gradually shifted traffic from 0% to 100%.

The evaluation wasn't without friction. It took a combination of client-side tuning and engine-level improvements from the Dragonfly team, including changes to the compactor, to reach the performance the Instacart team expected. Once those changes landed, Dragonfly outperformed the previous solution across all latency percentiles.

Physical deployment and data hydration took a matter of days (rather than the multiple months effort required for a typical database migration). End-to-end validation, including the tuning iterations, took roughly four to six weeks.

Results - Less Infrastructure, Better Performance

"By migrating to Dragonfly Cloud we've been able to cut our cluster size by 80% while also reducing our average and P99 latencies by 50% while further optimizing network cost in ways previously not possible."

Tristan Fletcher, Instacart

Average and P99 latencies dropped by 50%. The migration contributed approximately two milliseconds of improvement to the ad serving pipeline, a gain that mattered in the team’s goal to always satisfy their customer SLAs.  

On the infrastructure side, the team replaced hundreds of nodes with approximately 100 Dragonfly nodes, a roughly 70% reduction in cluster size.

Operationally, the change was just as significant. Previously, the team had no ability to control the pace of upgrades, run mixed-version states, or redistribute hash slots. Dragonfly opened up all of these. The team can now run single-shard tests before rolling out changes, upgrade at controlled rates, and roll back cleanly when issues come up. This has made it much easier to adopt new versions and manage cluster operations with confidence.

Advice for Machine Learning Infrastructure Leaders

Reflecting on the project, Tristan Fletcher offered three pieces of guidance for teams building or evolving online feature stores. Know your data access patterns and decide what you're optimizing for, whether that's cost, query speed, or developer productivity, because the trade-offs are real. Abstract your storage layer from your end users: Instacart moved tens of terabytes of data and swapped its underlying storage technology without any ML engineers noticing, because the right abstractions were in place. And be deliberate about storage formats and key design. Shortcuts in schema design, like relying on hashes for node selection, create problems that compound at scale.