Building a Feature Store with Feast, DuckDB, and Dragonfly: A Hands-On Guide

Feature Store Concepts Recap

Before getting started, let’s quickly recap what a feature store is and why it’s a critical piece of infrastructure for machine learning systems. If you’re looking for a deeper dive into the architecture and theory, we covered that in a previous blog post on feature store architecture and storage.

At its core, a feature store is a system designed to manage and serve machine learning features, which are the measurable properties used by machine learning models. Its primary purpose is to solve two key challenges: providing consistent data for model training and enabling low-latency data access for real-time inference. This is achieved through two main components:

• The Offline Store: Acts as the source of truth for historical feature data, used for training and batch scoring. It’s built on scalable data warehouses or lakes, optimized for large-scale scans.
• The Online Store: A low-latency, highly available database that serves the latest feature values for real-time inference. It’s optimized for fast random reads, often requiring millisecond-level response times.

A key design principle for Feast, and other feature stores, is a strict separation of concerns. The feature store is solely responsible for the storage, retrieval, and (sometimes) transformation of features. It is explicitly not responsible for housing or executing the ML models themselves.

Choosing Proper Offline/Online Stores for Your Workloads

For our build, we’ll continue using Feast, one of the most popular and mature open-source feature stores. Its abstraction allows us to plug in different storage backends for our offline and online needs.

Feast supports a wide array of powerful offline stores like Google BigQuery, Snowflake, and AWS Redshift. If your organization has already invested in one of these cloud data platforms and your datasets are in the petabyte scale, they are excellent, scalable choices. You should absolutely continue using them.

However, for a vast number of organizations, the scale of "big data" for model training is often in the hundreds of gigabytes to a few terabytes. For these workloads, DuckDB can be a great offline store choice because:

• Simplicity: DuckDB is an embedded database, as it is essentially a library that lives within your code. It requires no separate server process, which makes setup and development incredibly straightforward.
• Performance: Don’t let its simplicity fool you. DuckDB is a powerhouse for analytical processing (OLAP), with its vectorized query execution engine running complex analytical queries on large datasets faster than many solutions.
• Suitability: It handles the TB-scale data range with ease, making it a perfect, cost-effective fit for the majority of feature store use cases with minimal overhead.

For the online store, the requirements are entirely different. During model inference (i.e., real-time recommendation, fraud detection, personalization, etc.), every millisecond of latency directly impacts user experience and conversion. The online store must serve thousands to millions of feature lookups per second with consistent, sub-millisecond performance.

Dragonfly, as the most performant open-source Redis-compatible in-memory data store, is engineered precisely for this kind of demanding workload:

• Performance & Scalability with Modern Architecture: The multi-threaded, shared-nothing architecture allows Dragonfly to fully utilize modern hardware, delivering massive throughput while maintaining ultra-low latency. While a single Dragonfly instance delivers millions of RPS and TB-level in-memory data, for an even greater scale, Dragonfly Swarm facilitates horizontal scaling to exceed 100M+ RPS and 100TB+ capacity.
• Cost Efficiency: When the code is more efficient, it requires fewer servers. This is universally true. Besides, this efficiency allows you to potentially consolidate workloads like feature serving and caching onto a single, powerful Dragonfly data store, significantly reducing your cloud infrastructure costs and operational complexity.

Alright, enough theory! In this hands-on guide, we will build a complete feature store with both an offline and an online store, showcasing their distinct capabilities with practical code examples.

Complete Feature Store Setup: A Hands-on Guide

Let’s build a feature store from the ground up. We’ll define our features, populate both our offline (DuckDB) and online (Dragonfly) stores, and then see how to retrieve data from them for training and inference.

Prerequisites

To follow along, you can download our dragonfly-examples repository and install a few tools. We recommend using uv, a fast and modern Python package manager, to create a clean environment and handle dependencies:

# Install uv with curl:
$> curl -LsSf https://astral.sh/uv/install.sh | sh

# Install uv with wget:
$> wget -qO- https://astral.sh/uv/install.sh | sh

Next, make sure you have a Dragonfly server running, either by using Docker locally as shown below or giving Dragonfly Cloud a try.

 $> docker run -p 6379:6379 --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly

From this point on, ensure your working directory is dragonfly-examples/feast-recommendation-duckdb-dragonfly, as all subsequent commands rely on the files contained within it.

$> cd PATH/TO/dragonfly-examples/feast-recommendation-duckdb-dragonfly

Now, let’s install all the dependencies needed.

# Make sure all commands are executed within 'dragonfly-examples/feast-recommendation-duckdb-dragonfly'.
$> uv sync

Running uv sync installs dependencies and automatically creates a Python virtual environment. We then prefix all commands with uv to ensure they use the correct Python version and dependencies as specified in pyproject.toml:

# pyproject.toml
requires-python = "==3.11.9"
dependencies = [
    "feast[redis,duckdb]==0.51.0",
    "numpy>=2.3.2",
    "pandas>=2.3.1",
    "pyarrow<=17.0.0",
    "python-dateutil>=2.9.0.post0",
]

Configuring the Feature Repository

With our dependencies installed, the next step is to configure Feast. This is done via the feature_store.yaml file, which tells Feast everything it needs to know about our setup: the project name, where to store metadata, and crucially, which offline and online stores to use. For online_store, we specify redis and point it to localhost:6379, where our Dragonfly instance is running. Dragonfly’s full Redis compatibility means we can use the standard Feast Redis connector (this is true for your other Redis client libraries too) without any modifications.

# feature_store.yaml
project: recommendation_system
registry: data/registry.db
provider: local
offline_store:
  type: duckdb
online_store:
  type: redis
  connection_string: "localhost:6379"

Generating the Dataset

For our tutorial, we’ll generate a small but illustrative dataset that simulates the core entities of a recommendation system. The code below creates three data frames:

• users: Contains static user profile information like age, gender, and their historical average rating.
• items: Contains static attributes for items, such as category, price, and a popularity score.
• interactions: Captures the dynamic events where users engage with items, including view counts and ratings.

A crucial field in each data frame is event_timestamp. Feast uses this field to track the validity of feature values over time, which is a critical field used for point-in-time correctness when retrieving features from the feature store.

# recommendation_01_data.py
import pandas as pd

users = pd.DataFrame({
    "user_id": [1, 2, 3, 4, 5],
    "age": [25, 31, 45, 22, 38],
    "gender": ["M", "F", "F", "M", "M"],
    "avg_rating": [4.2, 3.8, 4.5, 3.2, 4.1],
    "preferred_category": ["electronics", "books", "clothing", "electronics", "home"],
    "event_timestamp": [pd.Timestamp("2025-08-28 00:00:00", tz="UTC") for _ in range(5)]
})

items = pd.DataFrame({
    # ...
})

interactions = pd.DataFrame({
    # ...
})

# Save the data frames to Parquet files.
users.to_parquet("data/users.parquet", engine="pyarrow", index=False)
items.to_parquet("data/items.parquet", engine="pyarrow", index=False)
interactions.to_parquet("data/interactions.parquet", engine="pyarrow", index=False)

# Run the data generation Python script.
$> uv run recommendation_01_data.py

In a real-world production system, this historical dataset would be vastly larger, spanning months or years of user activity and containing millions of records. This sample gives us a manageable foundation to define our feature repository and understand the data flow. Note that we also save the data frames as Parquet files, which can be easily queried and served by DuckDB.

Defining Feature Objects

Feast uses a declarative Python API to define the core components of your feature store: the data sources, entities, and feature views. This code acts as the blueprint that tells Feast how to map your raw data to meaningful, retrievable features.

# recommendation_02_repo.py
from datetime import timedelta

from feast import Entity, FeatureView, Field, ValueType, FileSource
from feast.data_format import ParquetFormat
from feast.types import Float32, Int64, String

# User data source.
users_file_source = FileSource(
    file_format=ParquetFormat(),
    path="data/users.parquet",
)

# User entity.
user = Entity(name="user", value_type=ValueType.INT64, join_keys=["user_id"])

# User feature view.
user_features = FeatureView(
    name="user_features",
    source=users_file_source,
    entities=[user],
    schema=[
        Field(name="age", dtype=Int64),
        Field(name="gender", dtype=String),
        Field(name="avg_rating", dtype=Float32),
        Field(name="preferred_category", dtype=String),
    ],
    ttl=timedelta(days=365), # An optional TTL, which limits how far back Feast will look when generating historical datasets.
)

# We also define data sources, entities, and feature views for items and interactions.

# Run 'feast apply' to register feature object definitions in Feast.
$> uv run feast apply

The feast apply command scans the current directory’s Python files (like recommendation_02_repo.py) for Feast object definitions, such as feature views, entities, and data sources. After validating them, it registers these definitions by syncing the metadata to the Feast registry. This registry, typically a protobuf binary file stored locally or in the cloud, has been configured in our feature_store.yaml to reside at data/registry.db.

Retrieving Historical Features for Training

One of the key advantages of using DuckDB as an offline store is its flexibility. There’s no need for a complex "loading" process—DuckDB can query our Parquet files directly and efficiently, treating them like native tables.

Following our configuration, Feast can orchestrate the historical feature retrieval. The following code demonstrates how to retrieve a set of features for specific user_id and item_id pairs at a given event_timestamp. The resulting data frame is exactly what you would use to train a model.

# recommendation_03_historical_features.py
import pandas as pd
from feast import FeatureStore

def fetch_historical_features():
    store = FeatureStore(repo_path=".")

    # Build an entity dataframe as input.
    entity_df = pd.DataFrame({
        "user_id": [1, 2],
        "item_id": [101, 102],
        "event_timestamp": [pd.Timestamp("2025-08-28 00:00:00", tz="UTC") for _ in range(2)]
    })

    # Specify which features to retrieve.
    features = [
        "user_features:age",
        "user_features:gender",
        "item_features:price",
        "interaction_features:view_count"
    ]

    historical_df = store.get_historical_features(
        entity_df=entity_df,
        features=features
    ).to_df()

    print(historical_df)

if __name__ == "__main__":
    fetch_historical_features()

# Run the Python script above as an example of historical feature retrieval.
$> uv run recommendation_03_historical_features.py

Materializing Recent Features for Real-Time Serving

Using DuckDB to access our complete historical dataset for training is generally fast. However, for real-time inference, we need the latest feature values available with even lower latency. This is where the online store comes in.

The process of copying the latest feature values from the offline store into the online store is called materialization. It establishes the crucial bridge between our batch data (in DuckDB/Parquet) and our low-latency serving layer (Dragonfly). We use the Feast CLI to materialize data for a specific time range up to the present. This command instructs Feast to find the latest feature values for all entities and push them to Dragonfly.

$> uv run feast materialize '2024-08-01T00:00:00' '2025-08-31T23:59:59'

After materialization, our Dragonfly instance is populated with the latest features. We can now verify this by fetching features for specific entities with a direct online lookup. This operation queries Dragonfly directly, simulating what would happen in a real-time inference request.

# recommendation_04_online_features.py
from feast import FeatureStore

def fetch_online_features():
    store = FeatureStore(repo_path=".")

    online_features = store.get_online_features(
        features=[
            "user_features:age",
            "user_features:gender",
            "item_features:price",
            "interaction_features:view_count"
        ],
        entity_rows=[
            {"user_id": 1, "item_id": 101},
            {"user_id": 2, "item_id": 102}
        ]
    ).to_dict()

    print(online_features)

if __name__ == "__main__":
    fetch_online_features()

# Run the Python script above as an example of real-time serving for recent features.
$> uv run recommendation_04_online_features.py

Running the Feast Server

Our recommendation_03_historical_features.py and recommendation_04_online_features.py scripts above show examples of calling the feature store directly using the Python API. We can also run the Feast server simply by using the following command:

$> uv run feast serve

$> curl --request POST \
  --url http://localhost:6566/get-online-features \
  --header 'Content-Type: application/json' \
  --data '{
	"features": [ ... ],
	"entities": { ... },
	"full_feature_names": true
}'

Scaling Feast with Dragonfly

A key strength of Feast is its horizontal scalability. You can deploy multiple stateless Feast server instances to handle high volumes of inference requests. However, all these instances share a single, critical dependency: the online store.

The performance of this online store ultimately determines the scalability of your entire feature-serving infrastructure. This is why Dragonfly is an ideal choice. Its multi-threaded architecture delivers massive throughput on a standalone instance and can be scaled horizontally with Dragonfly Swarm. By eliminating the online store bottleneck, Dragonfly ensures your entire Feast deployment can scale seamlessly. For concrete performance numbers, refer to our previous blog posts:

• Dragonfly vs. Valkey on Google Cloud (2025)
• Dragonfly Achieves 6.43 Million Ops/Sec on an AWS Graviton3E Instance (2024)
• Redis vs. Dragonfly Scalability and Performance (2023)

Final Thoughts

In this hands-on guide, we’ve built a fully functional feature store for a recommendation system using Feast. We used DuckDB as a powerful and simple embedded offline store for historical data and Dragonfly as a high-performance, Redis-compatible online store for low-latency feature serving.

This combination provides a robust and scalable architecture. DuckDB’s simplicity made setup a breeze for our offline workloads, while Dragonfly’s immense throughput and ultra-low latency ensure our online feature retrieval is never a bottleneck during real-time inference.

We encourage you to try this setup yourself. For the easiest way to get started with a production-ready Dragonfly instance, sign up for Dragonfly Cloud today. Join our community (Discord, forum) to share your experiences and learn more!