How to Implement NSQ Message Queue in NodeJS

Why NSQ?

Built by bit.ly
Written in Go
Plain and simple (the one I really liked, not complex as RabbitMQ)
Easy to set up, a very low learning curve
Distributed and horizontally scalable
Supports timeouts, back off and retries
Inbuilt admin panel

NSQ vs RabbitMQ?

You may be thinking “Why NSQ instead of RabbitMQ?”.

The client libraries of RabbitMQ seems outdated and needs too much of learning and setup.

But to be frank, NSQ is just a subset or RabbitMQ. RabbitMQ is a more mature one and has a lot of features. However, we found that we just need a Simple Message Queue system, easy to set up, low learning curve and have good client libraries.

Who Uses NSQ?

NSQ is built by bit.ly worlds best URL shortener. It is also using by some big companies like:

Bitly
Stripe
Docker
Segment – (Scaling NSQ to 750 Billion Messages)
Fastly
Moz

Implement NSQ in NodeJS

We’ll setup NSQ, configure it in NodeJS, and write a small code that will basically create a reader and writer that will process the job.

Let’s dive in

Install and boot NSQ

Install NSQ is a little tricky. I lost too much time installing and configuring it. The easiest solution is using Docker Compose

Step 1

Install Docker (Ubuntu, Mac) and Docker Compose

Step 2

Create a new file in your NodeJS project named docker-compose.yml and paste the following code. It will install the necessary components and configure ports.

Step 3

Start NSQ using docker-compose up -d

Add NSQ credentials to NodeJS

Add the following to your .env file.

If you’re new to .env read my blog post – How I Setup Environment Variables in NodeJS

Create a Reader and Writer in NodeJS

Run the script using node nsq.js

Open the NSQ admin panel on http://127.0.0.1:4171/

What we did is created a topic new_user and a channel send_email. Channels come under the topic. Or in another way, there will be multiple topic subscribed to a channel.

For example, when there is a new_user (topic), multiple channels like send_email, send_to_crm, send_slack_notification etc will be subscribed to it. All of these channels will receive the same user.

Writer publish to a topic, read subscribes to a channel inside a topic

Note: Make sure msg.finish() is called. That’s how NSQ understands the message is processed. Otherwise, it will be timed out and will re-queue

Conclusion

What we’d setup is a very basic message queue. NSQ has a lot more features as I mentioned above. Go through the docs to know more in detail – nsqjs

How to Implement NSQ Message Queue in NodeJS

Why NSQ?