Why we built InfiniStream

Summary

InfiniStream is a NATS/JetStream® compatible data streaming platform built directly on Object Storage like S3. It is a single stateless Go language binary that has no disks to manage, no routers or leafs to configure and no clusters to operate. InfiniStream can be upto 5x cheaper than NATS/JetStream to operate in the cloud because data streams are sent directly to your object storage instead of inter-zone networking and local storage that can be upto 80% of your NATS/JetStream costs if deployed at scale.

Simply try our demo in under 30s.

curl https://infinistream.io/download | sh
infini-agent demo

Now that we have you interested, read on!

Our Experience

We are big fans of NATS from its early days. Later when JetStream® came along we have used it in many of our projects and products we tried to commercialize. Once we tried to deploy it in production, things got complicated. Between being able to configure clustering, super clusters with Gateways, making sure the nodes are communicating, having enough memory to load sequences, managing space, balancing the cluster and making sure it stays up.

If applications were still using a mix of classic NATS, push subscription, JetStream with pull and push being able to tune it got to be challenging. Eventually the developers of NATS did start recommending that users go on to use JetStream and Pull for their projects.

Our Inspiration

We are upfront about this. This is not our idea. We are heavily inspired by what WarpStream® did to Kafka® and have seen positive response to be acquired by Confluent® who are in the process of being acquired by IBM!

What have we done!

We have created a stateless agent for NATS/JetStream® and we support only JetStream protocol and pull consumers. That’s it.

In the process we have separated the control plane from the data plane, offloaded data storage to S3, making the server a stateless agent and the control plane handling the state machine, like stream state and consumer states.

Let’s look at our architecture:

Instead of NATS/JetStream servers we have InfiniStream agents. Agents are stateless and there are groups, hence no leaders.

This brings many things to the table:

• it requires no configuration or provisioning of storage as Object Storage is infinite
• there’s no need to configure clusters, gateways, super clusters, leaf nodes or any thing. since S3 is accessible across a region data can be shared between VPC’s and the need to copy or route goes away
• you can use smaller server configurations (like m5) for your agents than larger ones (i3)
• scaling up or scaling down is as simple of starting or stopping an agent (or auto scale it)

Simplicity with Security

InfiniStream agent with data plane residing in your VPC and communicating with an InfiniStream hosted control plane is a prototypical implementation of BYOC(Bring Your Own Cloud). All your data stays in your VPC and S3 while meta data is communicated with the hosted control plane. By using S3 as your infinite storage that gives you eleven nines (99.999999999%) reliability your data is guaranteed to be safe and under your control. The added bonus is applications that previously couldn’t share data across VPC’s can now do so and all you need is the agents belong to the same group. That’s it. Your streams are accessible across VPC’s.

Of course, there are some trade offs. The significant one with InfiniStream is latency. Our P99 is ~ 200ms for Publisher because we will not acknowledge till data is written to S3 and the status committed to control plane. In addition the Publisher to Consumer latency is ~ 400ms.

If your workload can tolerate ~ 500ms publisher to consumer latency, then InfiniStream can reduce your streaming costs by 5x with nearly no operational overhead. There is nothing proprietary either when it comes to usage. NATS libraries work unchanged so applications stay the same.

If you have read upto this, you will see we have addressed two major things when it comes running NATS. Cloud economics and operational costs. Finally, what we have done is not something extraordinary or special. We believe we are the first to do it and it’s possible there may be more implementations of the same idea. We hope the best implementation will win and we want to be that.

We are excited to put this in developers hands and offer them an improved version of the tool they are familiar with.

Simply try our demo in under 30s.

curl https://infinistream.io/download | sh
infini-agent demo