Setup Agent

Overview

This page describes how to setup your InfiniStream Agent in your environment. At the moment we only support direct binary downloads, with docker images and Helm charts coming soon.

Required settings

The InfiniStream Agent is completely stateless and thus can be deployed however you prefer. It can be run directly as a binary via a service or as a docker image or any container technology.

The agent requires three mandatory variable and one optional variable that need to be passed as environement variables. By making them environement variables its easily adaptable to container environements where command line arguments may be cumbersome.

They are:

1. INFINISTREAM_GROUP_ID
2. INFINISTREAM_GROUP_KEY
3. INFINISTREAM_BUCKET_NAME

The values for INFINISTREAM_GROUP_ID and INFINISTREAM_GROUP_KEY can be obtained from the InfiniStream Admin Console ↗.

Object Storage

INFINISTREAM_BUCKET_NAME is the object storage bucket InfiniStream should write data to. Details on how to construct the bucket name and permissions can be found here.

Optional Settings

1. INFINISTREAM_BATCH_WRITE
2. INFINISTREAM_EXPIRY_CHECK_DEFAULT

Batch Write

InfiniStream agent by default waits for 250ms to coalesce writes from many streams (or hits 4Mb record size) into a single object. This can be slow. You can set INFINISTREAM_BATCH_WRITE to change it. Any value less than 50ms is invalid and same is any over 500ms. We personally recommend 100ms as a good compromise.

Expiry Check

InfiniStream agent has a designated agent that runs tasks. These include aging sequences and stream actions. Typically retention is set by MaxAge parameter when a stream is created or updated. When MaxAge is hit the task engine deletes sequences older than that and decides when the scheduler should be run next for that stream. This duration depends on the age of the next sequence for that stream. If the next sequences age is milliseconds after the last expired sequence, the scheduler is set to run in the next second as the default granularity is 1 second. However, this can be over ridden by setting the INFINISTREAM_EXPIRY_CHECK_DEFAULT variable. Continuining with the example, setting INFINISTREAM_EXPIRY_CHECK_DEFAULT=1m, even if the next sequence ages in milliseconds it will not be deleted for the next minute because that’s when the scheduler will run for that stream. However, if its greater than the default it will only run at that time.

Caution

Use this setting carefully. This is a global setting and applies to all streams.

Note

Every delete call is an update call to the control plane. By batching more deletes into single control plane update call, it will count as one update request only.

Region

The region that the InfiniStream control plane is running in. This corresponds to the value that was selected when the BYOC cluster was created and can be obtained from the InfiniStream Admin UI.

Cloud Provider	Region	End Point
AWS	US-East-1	https://api-use1.infinistream.io

Permissions and Ports

InfiniStream needs various permissions to perform operations against the object store. Please refer to the section on the pemissions needed.

In addition to object storage access InfiniStream agents need permissions to communicate to the control plane. Please refer to the the table above and match your group’s location with the address. Raw data flowing through your InfiniStream agent will never leave your cloud account, only metadata required to generate sequence numbers, object ID generation and performing virtual group operations are sent.

Finally, the InfiniStream agent requires 2 ports to be opened. For simplicity we recommend the InfiniStream agent listen on port 4222 for incoming NATS/JetStream connections and 9222 for intra InfiniStream agent communication for retrieving cached data in an availability zone. You can open 8222 as a HTTP port for standard agent level metrics that NATS/JetStream servers provide.

NATS Port

Default: 4222 Override: -p $PORT, --port $PORT

This port exposes the NATS/JetStream TCP protocol to NATS clients.

GRPC Port

Default: 9222 [$HTTP_PORT+1000] (which is $PORT+4000)

Used for inter agent communication in an availability zone to retrieve cached objects.

HTTP Port

Default: 8222 Override: -m $PORT, --http_port $PORT

This port exposes HTTP port for monitoring. Optional.

Instance Selection

The InfiniStream agent does not store data on disks locally, but uses the network extensively. We recommend network optimized instances with with atleast 4GiB of RAM per vCPU.

Our suggestion is to use m5n (m5n.large) and similar m5 family instances on AWS.

On our support of GCP we will update this section.