javascript - Insane amount of RAM usage with BullMQ + fetch - Stack Overflow

admin管理员组

文章数量:1434955

I have a small app that runs several requests(node-fetch) with in a queue with bullMQ.

And i have 3 worker threads (if i dont use this main event loop gets blocked and my whole app freezes.) (This is an another issue why main loop gets blocked with a few fetch requests ?)

queue A , queue B and queue C

all queues have same configuration

new Queue('a-queue', {
connection: {
    url: process.env.REDIS_URL,
    enableReadyCheck: true,
},
defaultJobOptions: {
    attempts: 5,
    removeOnComplete: true,
    removeOnFail: false,
},

worker for the queue (a,b,c all same):

const processor = pathToFileURL(__dirname + '/a_processor.js')
new Worker('a-queue', processor, {
    connection: {
        url: process.env.REDIS_URL,
        enableReadyCheck: true,
    },
    concurrency: 1,
    useWorkerThreads: true,
})

And a_processor.js is sending some requests with fetch and returns the values.

Also i have a setting for the global concurrency limit in index.js (start of the express app):

await queue.setGlobalConcurrency(5)

Upto this points it seems like it works as i expected (bullmq dashboard blue ones are active yellow ones are waiting):

But after like a minute redis instance is crashing due high memory consumption:

Memory usage is like skyrocket when i run this app.And what is doing is just sending fetch requests (10-15 in paralel) and these requests are not like responsing mbs of data its only few kb of json each time. There is nothing much but this consumes so much memory idk how.Or someting else causing that idk.

Here's the request method:

const headerGenerator = new HeaderGenerator()
const headers = headerGenerator.getHeaders()

const response = await fetch(`${domain}`, {
    headers: headers,
    agent: agentDcIp,
    signal: AbortSignal.timeout(15000) as any,
})

const body = await response.text()

return {
    body: body,
    ok: response.ok,
    status: response.status,
}

Any idea why this is happening and how to prevent it.

本文标签： javascriptInsane amount of RAM usage with BullMQfetchStack Overflow