node.js – RabbitMQ not receiving all messages
I have a NodeJS script that reads all files in a directory and sends a message to a RabbitMQ queue to process them.
There are 129k files. What I do is basically read all filenames with fs.readdir, iterate over the array of filenames and send a message to RabbitMQ. And for some reason, the queue does not store 129k messages, but only 3k.
This is my code:
files.forEach((fileName) => {
dispatchMessage(channel, fileName)
dispatchedMessagesCount++
})
const dispatchMessage = (channel, targetPath) => {
try {
channel.sendToQueue(queue, Buffer.from(targetPath), {
persistent: true,
expiration: 10800000,
})
logEvent(`Published %s`, targetPath)
} catch (error) {
logEvent(`Error publishing %s`, targetPath)
}
}
I tried to set a very big TTL -3hours- via the expiration attribute with no luck.
I don’t see any errors on the console output.
Looked like RabbitMQ had some kind of rate limiting policy for incoming messages, but I couldn’t find anything about it.
Any ideas?
Read more here: Source link

