Building Distributed Systems and Microservices in Go with NATS Streaming

A while ago, I have written a blog post Introducing NATS to Go Developers for using Apcera NATS as the messaging system for building distributed systems and Microservices in Go. In this post, I will take a look into NATS Streaming server, which built on the top of basic NATS server that provides a persistent log for your messages which you publish on the NATS.

NATS is an open source, lightweight, high-performance cloud native messaging system. I love to use NATS for building distributed systems in Go because for its performance and simplicity. Because building distributed system always bring lot of complexities, using a simple and high performant messaging system is an important and vital decision. NATS is available in two interoperable modules: the core NATS platform -the NATS server (executable name is gnatsd) referred to simply as NATS, and NATS Streaming(executable name is nats-streaming-server).

Introducing NATS Streaming

The basic NATS server is designed for high performance and simplicity, which doesn’t provide a persistent store for the messages that you publish over the NATS. Having the lack of a persistent store for messages will be a problem for many of the distributed systems. For instance, let’s say that one of your subscriber systems, is down when you publish a message, then the subscriber system doesn’t receive the message, hence you have to provide an architecture approach for dealing with such scenarios. For another instance, let’s say that you would like to add a new system into an existing distributed systems environment in which you would like to receive all messages from the beginning to get a history of data, but you don’t get messages from the basic NATS server because of the lack of a persistent store. On the other hand, NATS Streaming comes with a persistent store for having a log for the messages that publish over the NATS server. If you need persistent messaging and delivery guarantees, you can use NATS Streaming instead of the core NATS platform.

NATS Streaming is an extremely performant, lightweight reliable streaming platform built on the top of core NATS platform that provides persistent logs. NATS Streaming is written in Go. It can be employed to add event streaming, delivery guarantees, and historical data replay to NATS. Keep in mind that NATS Streaming is not a separate server, but it uses the NATS server (gnatsd). In short, NATS Streaming embeds a NATS server as the messaging server, and provides an extra capability of having a persist logs to be used for event streaming systems.

NATS Streaming provides the following high-level feature set:

• Log based persistence
• At-Least-Once Delivery model, giving reliable message delivery
• Rate matched on a per subscription basis
• Replay/Restart
• Last Value Semantics

The high-level features of NATS Streaming is similar like the capabilities of Apache Kafka, but the former wins when you consider simplicity over complexity. Because NATS Streaming is relatively a new technology, it needs to improve in some areas especially on providing a better solution for load balancing scenarios, as compared to Apache Kafka.

Channels

Channels are the most important concept in NATS Streaming Server. Channels are subject clients that send data to and consume from. Unlike the basic NATS Server, NATS Streaming server does not support wildcard for channels. You can control the number of channels by using configuration. Messages published to a channel are stored in a message log inside the channel as shown in the below picture.

The picture above depicts a file store for persistent logs in which a directory named order-notification is used to store messages for a channel with same name.

Message Log

Messages that published to a channel are appended to the end of the log in the persistent store. The limit for the message can be configured. If a limit is configured for globally for all channels, or specifically for a particular channel, when the limit is reached, older messages will be pruned to limit the size in the persistent log in order to append newer messages. By default, NATS Streaming uses an in-memory store for the messages, but this behaviour can be changed using configuration.

To read the full article and learning the NATS Streaming with a practical demo, check out the original post written on Medium here.