Question: How can we prevent duplicate jobs in BullMQ?


In BullMQ, there's no built-in functionality to automatically prevent the addition of duplicate jobs. However, you can implement this yourself by assigning unique job IDs.

The job ID must be unique for every job in a given queue. If you try to add a job with an ID that already exists in the queue, BullMQ will throw an error. Therefore, you can use this feature to prevent the addition of duplicate jobs.

Here's how you can do it:

const Queue = require('bullmq').Queue; const queue = new Queue('my-queue'); async function addJob(data) { try { await queue.add(`job:${data}`, { jobId: `job:${data}` }); console.log(`Job ${data} added successfully`); } catch (err) { if (err.message.includes('Job is already waiting')) { console.log(`Job ${data} already exists`); } else { console.error(err); } } } // Add unique jobs addJob('unique_task1'); addJob('unique_task2');

In this example, we're using the data as the job ID, which means that a job with the same data cannot be added twice. This can effectively prevent duplicate jobs based on your definition of duplication.

Remember, choosing a job ID depends on how you define a "duplicate" job. If two jobs with different data but doing the same kind of work are considered duplicates, it might make sense to generate job IDs based on the type of work instead of the data. Just ensure that the chosen job ID accurately represents a unique piece of work in your application context.

Was this content helpful?

White Paper

Free System Design on AWS E-Book

Download this early release of O'Reilly's latest cloud infrastructure e-book: System Design on AWS.

Free System Design on AWS E-Book
Start building today

Dragonfly is fully compatible with the Redis ecosystem and requires no code changes to implement.