LGouellec / Kafka Streams Dotnet
Labels
Projects that are alternatives of or similar to Kafka Streams Dotnet
·
.NET Stream Processing Library for Apache Kafka TM ·Quality Statistics
Project Statistics
Streamiz Kafka .NET is .NET stream processing library for Apache Kafka.
It's allowed to develop .NET applications that transform input Kafka topics into output Kafka topics. It's supported .NET Standard 2.1.
So this library can be used by >= .NET Core 3.0 and/or .NET 5.0 applications.
It's a rewriting inspired by Kafka Streams. Finally it will provide the same functionality as Kafka Streams.
This project is being written. Thanks for you contribution !
Timeline
- Begin 2021 - 1.2.0 - Persistent state store (eg: RocksDB Store), Repartition and Changelog topics
- April 2021 - 1.3.0 - Processor API, Metrics, Interactive Queries, Standby Replica
Documentation
Read the full documentation on https://lgouellec.github.io/kafka-streams-dotnet/
Installation
Nuget packages are listed to nuget.org
Install the last version with :
dotnet add package Streamiz.Kafka.Net
Usage
There, a sample streamiz application :
static async System.Threading.Tasks.Task Main(string[] args)
{
var config = new StreamConfig<StringSerDes, StringSerDes>();
config.ApplicationId = "test-app";
config.BootstrapServers = "192.168.56.1:9092";
config.SaslMechanism = SaslMechanism.Plain;
config.SaslUsername = "admin";
config.SaslPassword = "admin";
config.SecurityProtocol = SecurityProtocol.SaslPlaintext;
config.AutoOffsetReset = AutoOffsetReset.Earliest;
config.NumStreamThreads = 2;
StreamBuilder builder = new StreamBuilder();
var kstream = builder.Stream<string, string>("stream");
var ktable = builder.Table("table", InMemory<string, string>.As("table-store"));
kstream.Join<string, string, StringSerDes, StringSerDes>(ktable, (v, v1) => $"{v}-{v1}")
.To("join-topic");
Topology t = builder.Build();
KafkaStream stream = new KafkaStream(t, config);
Console.CancelKeyPress += (o, e) => {
stream.Dispose();
};
await stream.StartAsync();
}
TODO implementation
- [X] Global state store
- [X] Refactor Subtopology & StreamTask assignment
- [X] Refactor Topology description for corresponding to java implementation
- [X] Refactor Partition grouper + record queue
- [X] Statefull processors impl
- [ ] Transform and Through Processor
- [ ] Supress Processor (.suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded())))
- [ ] Rocks DB state implementation
- [ ] Changelog Topic
- [ ] Task restoring
- [ ] Repartition topic
- [ ] Repartition Processor KAFKA-8611 | PR #7170
- [ ] Processor API
- [ ] Sample projects (Micro-services, console sample, topology implementation, etc ..) which use Streamiz package
- [ ] Optimizing Kafka Streams Topologies
- [ ] Standby Replica
- [ ] Interactive Queries
- [ ] Metrics
Contributing
Owners:
Maintainers:
Streamiz Kafka .Net is a community project. We invite your participation through issues and pull requests! You can peruse the contributing guidelines.
When adding or changing a service please add tests and documentations.
Support
You can found support here