Error: bullmq memory leak

What's Causing This Error

The error 'bullmq memory leak' typically arises when excessive data is stored in the BullMQ queues without getting processed or removed. A few common causes for this are:

  1. Insufficient Worker Processes: If there aren't enough workers to process jobs, they may pile up in the queue, consuming increasing amounts of memory.
  2. Lack of Job Removal: Completed and failed job records might not be removed from the queue, causing unnecessary accumulation.
  3. Delayed Jobs: These are jobs that are not ready to be processed immediately, and a notably large number of them may cause memory usage to increase over time.

Solution - Here's How To Resolve It

To resolve the memory leak issue in BullMQ, consider the following strategies:

  1. Scale the Workers: Make sure you have sufficient worker processes to handle the incoming flow of jobs. You can use clustering or child processes to scale horizontally.
  2. Remove Jobs After Completion: Implement auto-removal of completed and failed jobs from the queue by setting the 'removeOnComplete' and 'removeOnFail' options.
  3. Monitor Delayed Jobs: Keep an eye on delayed jobs and limit their quantity if possible. Alternatively, consider using a priority queue for critical tasks to ensure they're handled even when many jobs are delayed.
  4. Regular Monitoring and Auditing: Regularly monitor and audit your queues to identify unexpected behaviour like sudden memory spikes. Tools like Prometheus and Grafana can help with monitoring and visualizing queue performance.

Was this content helpful?

Start building today

Dragonfly is fully compatible with the Redis ecosystem and requires no code changes to implement.