Lightweight & elastic kubernetes-native event streaming system
PolarStreams is a lightweight, elastic, Kubernetes-native event streaming system. It acts as a persistent buffer between services providing at-least-once delivery guarantees.
PolarStreams is optimized to be developer friendly, resource efficient, have minimal operational overhead, and be a good K8s neighbor.
We honestly value your contribution to make this project ready for general availability. If you want to contribute, check out the Contributing Guide.
Producing and consuming messages from PolarStreams is as simple as sending a HTTP request. Use your favorite HTTP client in your technology stack to send and receive events. Additionally, we also provide a Go Client Library.
Get started with PolarStreams on Kubernetes using this guide.
PolarStreams is distributed by default with a minimal size of 3 brokers for production use. You can run a single-broker using Docker / Podman with developer mode. Read more about how to get started with PolarStreams on Docker.
PolarStreams was created to provide a developer friendly and resource efficient event streaming solution for Kubernetes that is easy to use and operate.
We believe that deploying and managing a persistent event streaming for a microservice architecture should be as easy as deploying a stateless service. Users should be able to start with small pods (512MiB of memory!) and elastically scale to support larger workloads with no operational overhead.
Ease of use and resource efficiency is what the Cloud is all about. Pay for the resources that you need and avoid overprovisioning in advance.
go build .
go test -v ./...
The Contributing Guide has more information about environment setup and other topics that can help you to build PolarStreams from source.
Persistent event streaming should act as a buffer between services, supporting peaks by seamlessly scaling, allowing events to be consumed at a later time from the peak. Once a certain amount of events have been produced and not consumed, the following events can be stored in object storage to be pulled once the consumers catch up, reducing costs for storage.
Message streaming should be highly available by default and only consume the minimum required resources to run effectively.
Regular maintenance operations like upgrading the cluster or decommissioning Kubernetes nodes are common and should be handled smoothly. For example, when a consumer is being restarted due to an upgrade, there's no need for re-balance the data assignment among consumers as it's very likely that the consumer will be ready in a few seconds. Querying the Kubernetes API can provide valuable insights to understand what is occurring and what to expect.
The system should elastically scale intelligently, to support the future web scale and without affecting running services.
We are always happy to have contributions to the project whether it is source code, documentation, bug reports, feature requests or feedback. To get started with contributing:
Copyright (C) 2022 Jorge Bay
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.