Setup Agent
Instructions to install the agent and configure it
Overview
Section titled “Overview”This page describes how to setup your InfiniStream Agent in your environment. At the moment we only support direct binary downloads, with docker images and Helm charts coming soon.
Required settings
Section titled “Required settings”The InfiniStream Agent is completely stateless and thus can be deployed however you prefer. It can be run directly as a binary via a service or as a docker image or any container technology.
The agent requires three mandatory variable and one optional variable that need to be passed as environement variables. By making them environement variables its easily adaptable to container environements where command line arguments may be cumbersome.
They are:
INFINISTREAM_GROUP_IDINFINISTREAM_GROUP_KEYINFINISTREAM_BUCKET_NAMEINFINISTREAM_BATCH_WRITE
The values for INFINISTREAM_GROUP_ID and INFINISTREAM_GROUP_KEY can be obtained
from the InfiniStream Admin Console ↗.
Object Storage
Section titled “Object Storage”INFINISTREAM_BUCKET_NAME is the object storage bucket InfiniStream should write data to.
Details on how to construct the bucket name and permissions can be found here.
Batch Write
Section titled “Batch Write”InfiniStream agent by default waits for 250ms to coalesce writes from many streams (or hits 4Mb record size) into
a single object. This can be slow. You can set INFINISTREAM_BATCH_WRITE to change it. Any value less than 50ms
is invalid and same is any over 500ms. We personally recommend 100ms as a good compromise.
Region
Section titled “Region”The region that the InfiniStream control plane is running in. This corresponds to the value that was selected when the BYOC cluster was created and can be obtained from the InfiniStream Admin UI.
| Cloud Provider | Region | End Point |
|---|---|---|
AWS | US-East-1 | https://api-use1.infinistream.io |
Permissions and Ports
Section titled “Permissions and Ports”InfiniStream needs various permissions to perform operations against the object store. Please refer to the section on the pemissions needed.
In addition to object storage access InfiniStream agents need permissions to communicate to the control plane. Please refer to the the table above and match your group’s location with the address. Raw data flowing through your InfiniStream agent will never leave your cloud account, only metadata required to generate sequence numbers, object ID generation and performing virtual group operations are sent.
Finally, the InfiniStream agent requires 2 ports to be opened. For simplicity we recommend the InfiniStream
agent listen on port 4222 for incoming NATS/JetStream connections and 9222 for intra InfiniStream agent
communication for retrieving cached data in an availability zone. You can open 8222 as a HTTP port for
standard agent level metrics that NATS/JetStream servers provide.
Default: 4222
Override: -p $PORT, --port $PORT
This port exposes the NATS/JetStream TCP protocol to NATS clients.
Default: 9222 [$HTTP_PORT+1000] (which is $PORT+4000)
Used for inter agent communication in an availability zone to retrieve cached objects.
Default: 8222
Override: -m $PORT, --http_port $PORT
This port exposes HTTP port for monitoring. Optional.
Instance Selection
Section titled “Instance Selection”The InfiniStream agent does not store data on disks locally, but uses the network extensively. We recommend network optimized instances with with atleast 4GiB of RAM per vCPU.
Our suggestion is to use m5n (m5n.large) and similar m5 family instances on AWS.
On our support of GCP we will update this section.