Syncing Data from PostgreSQL to Dragonfly Using Airbyte

Modern applications often rely on moving data between different systems to optimize the experience for their users. Whether it’s for caching frequently accessed records, offloading analytical workloads, or syncing data across services, transferring data from a transactional database like PostgreSQL to a high-performance in-memory store like Dragonfly is a common requirement.

Let’s take a practical example: a dashboard that displays some daily metrics reports of our application, such as the number of active users and sales amount. Once computed at the end of each day, this summary doesn’t change anymore, but it is accessed frequently by multiple teams the next day. Instead of querying PostgreSQL over and over, especially if those queries are resource intensive, we can extract the summary data once a day and store it in Dragonfly for fast lookups.

ELT tools can make this process seamless. In this post, we’ll show you how to load data from PostgreSQL into Dragonfly using Airbyte, an open-source platform that helps you move data from any source to any destination.

What is Airbyte?

Airbyte is a leading open-source ELT platform that provides an easy-to-use interface, the largest catalog of data connectors (for both data sources and destinations), and support for custom transformations and configurations. It’s ideal for syncing data between different databases and data stores within your infrastructure on a massive scale.

ELT stands for extract, load, and transform. ELT pipelines typically leverage modern cloud-based data warehouses, which support massive parallel processing. As a result, ELT solutions can handle large datasets far more efficiently than traditional ETL systems. Airbyte has a great blog post available if you want to learn more about the key differences between ELT and ETL.

Airbyte | Syncing PostgreSQL & Dragonfly

Loading Data From PostgreSQL to Dragonfly

Let’s walk through the full setup of loading data from Postgres to Dragonfly using Airbyte. For simplicity, we’ll be running everything locally in containers, but the same steps should be valid if you’re using the cloud offerings of Dragonfly or Airbyte. Let’s continue with the example we discussed earlier: loading a daily_metrics table from Postgres into Dragonfly for fast read access.

Prerequisites

Before we begin, ensure you have the following tools installed:

Redis CLI: A command-line client for interacting with Redis-compatible databases (like Dragonfly). You’ll use this to verify data in Dragonfly. Here’s how you can install Redis CLI on your system.
Docker: We’ll run Postgres and Dragonfly in Docker containers. For Airbyte, which runs on Kubernetes, we’ll use its dedicated CLI tool, abctl, for deployment and management. While you won’t interact with Docker directly for Airbyte, you’ll need to have the Docker engine running in the background to support the local Kubernetes cluster. Ensure you have installed Docker before proceeding.

Running Postgres

Now we are ready with the tools above installed. Let’s start a Postgres container first:

docker run --name airbyte-postgres -e POSTGRES_PASSWORD=password -p '5432:5432' -d postgres

Next, we’ll seed the database with sample daily metrics data. Keep in mind that the daily_metrics table serves as a summary, aggregating transaction details from other tables. This reflects a realistic scenario, which we’ll explore further in the upcoming analysis.

CREATE TABLE daily_metrics (
  metric_date DATE PRIMARY KEY,
  total_users_count INT NOT NULL,
  active_users_count INT NOT NULL,
  new_signups_count INT NOT NULL,
  total_sales_amount DECIMAL(12, 2) NOT NULL
);

INSERT INTO
  daily_metrics (
    metric_date,
    total_users_count,
    active_users_count,
    new_signups_count,
    total_sales_amount
  )
VALUES
  ('2025-06-01', 100, 90, 10, 1000.00),
  ('2025-06-02', 110, 95, 15, 1200.00),
  ('2025-06-03', 125, 98, 25, 1500.00);

You can verify that the data was added correctly by running:

SELECT * FROM daily_metrics;

This will output something like:

 metric_date | total_users_count | active_users_count | new_signups_count | total_sales_amount 
-------------+-------------------+--------------------+-------------------+--------------------
 2025-06-03  |               125 |                 98 |                25 |            1500.00
 2025-06-02  |               110 |                 95 |                15 |            1200.00
 2025-06-01  |               100 |                 90 |                10 |            1000.00
(3 rows)

Running Dragonfly

In this example we’ll be running Dragonfly locally in a container. If you’re running this in production, the best option would be using Dragonfly Cloud to avoid the hassle of hosting and managing your own Dragonfly instance.

Spin up a Dragonfly container by running:

docker run --name airbyte-dragonfly -p '6379:6379' --ulimit memlock=-1 -d docker.dragonflydb.io/dragonflydb/dragonfly

You can read more about the different ways to run Dragonfly here.

Running Airbyte

Using the Airbyte CLI is the easiest way to get started. It will run all the different containers Airbyte needs for us without us having to manually set things up. Install the Airbyte CLI by running the following commands. Note that I am running macOS for demonstration purposes. If you are running other operating systems, please follow the Airbyte installation guide.

brew tap airbytehq/tap
brew install abctl

And then run Airbyte locally:

abctl local install

This will take you to the Airbyte login screen. If it doesn’t, you can simply go to http://localhost:8000 to access it.

You can get the credentials required to log in by running:

abctl local credentials

Postgres as a Source in Airbyte

Now we need to set up our locally running Postgres database as a source in Airbyte. To do this, head over to the Sources section from the menu on the sidebar and search for Postgres.

Add the following configuration and leave everything else at its default values:

Host: host.docker.internal (since we’re running Postgres using Docker)
Port: 5432 (port we specified in our docker run command)
Database Name: postgres (default DB name in Postgres)
Username: postgres (the default user in Postgres)
Password: password (the password we specified in our docker run command)
Update Method: Detect Changes with Xmin System Column. This incrementally reads new inserts and updates via Postgres Xmin system column. Best option for cases like these where databases have low transaction pressure.)

Dragonfly as a Destination in Airbyte

Now that we have our source set up, it’s time to configure our destination in Airbyte. Airbyte’s Redis connector works perfectly with Dragonfly, as the latter is fully compatible with the Redis wire protocol. You can search for the Redis connector in the Marketplace tab for Airbyte Destinations.

Just like the Postgres source, configure the Dragonfly server as a destination by adding the following values:

Host: host.docker.internal (since we’re running Dragonfly using Docker)
Port: 6379 (port we specified in our docker run command)
Username: default (the default username associated with our local Dragonfly instance)

Postgres to Dragonfly Connection in Airbyte

Now that we have our source and destination set up, it’s time to configure the connection in Airbyte. Head over to the Connections tab from the side menu. Create a new connection here and choose the Postgres source and the Dragonfly destination we just created.

After this, Airbyte will take you to the Select streams step, where you can see and select which items you want to be loaded from Postgres to Dragonfly. Let’s select only the daily_metrics table and all of its fields for our demo right now:

Head over to the next step where you can configure more details related to your connection. By selecting Custom format, change the Destination Namespace to something more meaningful like pg_daily_metrics since we are syncing from Postgres and all fields of the daily_metrics table only. This will ensure our keys in Dragonfly start with a reasonable prefix.

We’re going with the default values for the rest of the settings, but please feel free to change them based on your particular use case. After you’re done, click Finish & Sync to start the loading of data from Postgres to Dragonfly!

Verifying the Data Load

Once the sync completes, it’s time for us to verify the keys in our Dragonfly data store. Since the Dragonfly container’s port is mapped to the host machine as well, one easy way to access it is using Redis CLI.

redis-cli -p 6379

By running the command below, you should see keys synced from Postgres to Dragonfly:

dragonfly$> KEYS pg_daily_metrics:daily_metrics*
#=> 1) "pg_daily_metrics:daily_metrics:3"
#=> 2) "pg_daily_metrics:daily_metrics:2"
#=> 3) "pg_daily_metrics:daily_metrics:1"
#=> 4) "pg_daily_metrics:daily_metrics"

To further inspect the data, run the commands below:

# This key holds the highest sequence value of synced records for this prefix.
dragonfly$> GET "pg_daily_metrics:daily_metrics"
#=> "3"

# The '_airbyte_data' field holds the full JSON blob of a record.
dragonfly$> HGET "pg_daily_metrics:daily_metrics:3" "_airbyte_data"
#=> "{\\"metric_date\\":\\"2025-06-03\\",\\"total_users_count\\":125,\\"active_users_count\\":98,\\"new_signups_count\\":25,\\"total_sales_amount\\":1500}"

And there you have it! This confirms that we’ve successfully loaded data from our Postgres database into Dragonfly. In this example, we performed an initial sync, but Airbyte offers even greater flexibility—allowing you to control sync frequency (scheduled or on-demand) and choose between full refreshes or incremental updates.

Like most modern data platforms, Airbyte adapts to your application’s needs, ensuring data is synced efficiently. As a leading solution in this space, it provides the reliability and configurability needed for seamless integration.

Use Case Considerations

Airbyte excels at scheduled and incremental data synchronization, making it a great choice for many data integration workflows. It is ideal for use cases like the daily_metrics example above, where data is aggregated once per day and doesn’t require real-time updates once the metric values are computed. Airbyte can efficiently sync these results to downstream systems like Dragonfly without the overhead of constant polling or additional event-driven pipelines. And obviously, we are just scratching the surface here, as Airbyte can support plenty of other use cases.

However, for real-time application caching—where frequently accessed keys need to be populated immediately in cache after being written to the database to ensure data freshness and optimal performance—Airbyte’s periodic or incremental syncs may introduce some delay.

For use cases that require immediate cache updates or depend heavily on real-time data freshness, we recommend complementing Airbyte with direct writes to Dragonfly from your application layer. This hybrid approach ensures your cache stays aligned with your application’s latest state while leveraging Airbyte for scheduled data workflows.

Dragonfly Integrates Seamlessly with Popular ELT Tools

Dragonfly is a drop-in Redis replacement, built for performance and scale. It handles large amounts of concurrent traffic with low latency and is ideal for use cases like:

Frequently accessed summary data (e.g., reports, dashboards)
Read-heavy workloads that don’t require real-time consistency
Precomputed analytics and statistical models

With Airbyte, you can keep your analytical cache up-to-date with minimal effort and without burdening your production PostgreSQL. Here we just showed one example, but Airbyte is extremely versatile and has hundreds of data sources and destinations supported, facilitating different use cases.

Whether you’re syncing product analytics, user preferences, or daily reports, Dragonfly integrates smoothly with ELT platforms like Airbyte. Thanks to its compatibility with Redis connectors, you can load structured, transformed data from virtually any source. As for the next steps, I’d recommend trying out what we did in this tutorial on your own to get hands-on experience with Dragonfly and Airbyte!