November 7th: Exploring an 80% lower cost hosted Redis alternative - register

Question: How can you set global concurrency in BullMQ?

Answer

In BullMQ, the concept of global concurrency allows you to control the maximum number of jobs that are processed concurrently across all workers.

However, BullMQ does not support global concurrency out of the box. In BullMQ, concurrency is controlled on a per-worker basis via the concurrency parameter when creating a worker. Here's an example:

const worker = new Worker("my-queue", async job => { // process your job here. }, { concurrency: 5 // Process up to 5 jobs concurrently. });

With this setup, if you have multiple workers, each one would process up to 5 jobs concurrently, but there would be no overall limit across all workers.

To achieve global concurrency with BullMQ, you might need to design some custom logic inside or outside of your job handlers. One possible approach could be using a global semaphore like Redis to limit the total number of concurrent jobs.

Remember that handling concurrency at a global level can be complex and might not always be necessary, depending upon your use case. It's often more efficient and simpler to allow each worker to handle its own concurrency.

Was this content helpful?

White Paper

Free System Design on AWS E-Book

Download this early release of O'Reilly's latest cloud infrastructure e-book: System Design on AWS.

Free System Design on AWS E-Book

Switch & save up to 80% 

Dragonfly is fully compatible with the Redis ecosystem and requires no code changes to implement. Instantly experience up to a 25X boost in performance and 80% reduction in cost