Handling Multiple requests with Redis and Bullmq

In our previous post, we explored how to efficiently download a large dataset from MongoDB. We utilised the built-in fs module to create temporary files on the server, enabling us to handle substantial amounts of data. However, upon deploying our solution for testing, an unexpected issue arose during load testing.

Our tester discovered that when multiple requests were made to the download API within a short span of time, both requests received the same data, despite requesting different datasets. …

bullmq create data dataset download enabling files however issue mongodb multiple node redis server solution testing webdev