Reading Data
How reads works when a client makes Fetch request
Introduction
Section titled “Introduction”InfiniStream Agent maintains cost efficiency for reads by ensuring that the Agents only load data from object storage once per availability zone and minimize GET requests to object storage by loading data in large chunks. To accomplish this goal, InfiniStream Agents maintain a per-Availability Zone cache, which completely decouples the number of streams and consumers from the number of object storage GET requests that the Agents have to perform to serve consumer clients’ Fetch() requests.
Reading Data
Section titled “Reading Data”Every availability zone has an agent with a cache role, if there’s an agent running in it. This agent serves the requested stream sequence if available in the cache, else downloads it once from the object store.
A cache is maintained per Availability Zone, which ensures that Fetch() requests do not need to cross zonal boundaries in order to be fulfilled. While this results in slightly more GET requests to object storage, the avoidance of cross-AZ data transfer fees more than makes up for this increase. In addition, the Agents serve subsequent Fetch() requests from the cache, which further reduces the impact of the per-AZ GET requests.
This approach results in a workload cost profile that is both more cost-efficient than what is possible with NATS servers with local disks, and cost-efficient with respect to usage of cloud object storage.
Ordering
Section titled “Ordering”Any InfiniStream Agent can serve any Fetch() request from any client. InfiniStream still maintains JetStreams’s ordering guarantees. This can be seen as a pull request group where the load can be balanced across many consumers across multiple agents across availability zones.
Consumers can fetch data from any Agent, and reads occur within the same AZ due to the caching approach discussed above. In the case where there is zero alignment between the consumer and the group, users can trivially deploy Agents in the correct Availability Zone.
Notes: 1
Footnotes
Section titled “Footnotes”-
We have followed the same order as WarpStream documentation for easy reading and collation ↩