node.js – RabbitMQ not receiving all messages

I have a NodeJS script that reads all files in a directory and sends a message to a RabbitMQ queue to process them.

There are 129k files. What I do is basically read all filenames with fs.readdir, iterate over the array of filenames and send a message to RabbitMQ. And for some reason, the queue does not store 129k messages, but only 3k.

This is my code:

files.forEach((fileName) => {
    dispatchMessage(channel, fileName)
    dispatchedMessagesCount++
})

const dispatchMessage = (channel, targetPath) => {
    try {
        channel.sendToQueue(queue, Buffer.from(targetPath), {
            persistent: true,
            expiration: 10800000,
        })

        logEvent(`Published %s`, targetPath)
    } catch (error) {
        logEvent(`Error publishing  %s`, targetPath)
    }
}

I tried to set a very big TTL -3hours- via the expiration attribute with no luck.
I don’t see any errors on the console output.

Looked like RabbitMQ had some kind of rate limiting policy for incoming messages, but I couldn’t find anything about it.

Any ideas?

Overall files vs jobs pushed

Read more here: Source link