All Categories → Data Processing → kafka

Top 766 kafka open source projects

Components Contrib

Community driven, reusable components for distributed apps

✭ 131

go kafka sns sqs

Iot Traffic Monitor

✭ 131

java spring-boot kafka spark cassandra

Metronome

Metronome is a distributed and fault-tolerant event scheduler

✭ 131

go http kafka cron event fault-tolerance

Kafka Junit

This library wraps Kafka's embedded test cluster, allowing you to more easily create and run integration tests using JUnit against a "real" kafka server running within the context of your tests. No need to stand up an external kafka cluster!

✭ 131

java kafka zookeeper junit

Abris

Avro SerDe for Apache Spark structured APIs.

✭ 130

scala kafka spark avro

Scrapy demo

all kinds of scrapy demo

✭ 128

python mongodb demo kafka spider example pipeline scrapy sqlalchemy oss

Faust

Python Stream Processing. A Faust fork

✭ 124

python redis kafka distributed-systems asyncio

Docker Bro

Bro IDS Dockerfile

✭ 126

docker elasticsearch kafka network-analysis network-monitoring ids

Neo4j Streams

Neo4j Kafka Integrations, Docs =>

✭ 126

kotlin kafka neo4j stream-processing graph-database kafka-producer

Distributed framework

python通用分布式函数调度框架 pip install function_scheduling_distributed_framework

✭ 123

python redis kafka rabbitmq disk rocketmq

Azure Event Hubs For Kafka

Azure Event Hubs for Apache Kafka Ecosystems

✭ 124

java kafka azure microsoft messaging apache apache-kafka

Apiproject

[https://www.sofineday.com], golang项目开发脚手架,集成最佳实践(gin+gorm+go-redis+mongo+cors+jwt+json日志库zap(支持日志收集到kafka或mongo)+消息队列kafka+微信支付宝支付gopay+api加密+api反向代理+go modules依赖管理+headless爬虫chromedp+makefile+二进制压缩+livereload热加载)

✭ 124

go golang redis makefile kafka jwt spider headless alipay mongo api-server cors gorm livereload compress wxpay

Tunnel

PG数据同步工具（Java实现）

✭ 122

java postgresql elasticsearch kafka postgres stream sync replication

Php Rdkafka

Production-ready, stable Kafka client for PHP

✭ 1,703

c PHP shell powershell kafka php-extension kafka-client

Slimmessagebus

Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

✭ 120

redis kafka azure messaging avro dot-net apache-kafka net-core message-bus bus pub-sub

Kafka Zk Restapi

Kafka Zookeeper RESTful API to perform topic/consumer group administration/metric(offset\lag\message) collection and monitor

✭ 121

java rest kafka restful-api zookeeper consumer topic

Beast

Load data from Kafka to any data warehouse

✭ 119

java kafka bigquery

Example Spark Kafka

Apache Spark and Apache Kafka integration example

✭ 120

scala kafka spark spark-streaming

a command-line Protobuf parser with Kafka support and JSON output

✭ 120

rust parser kafka protobuf

Kafka Avro Course

Learn the Confluent Schema Registry & REST Proxy

✭ 120

java kafka learning udemy

Kafka Stack Docker Compose

docker compose files to create a fully working kafka stack

✭ 1,836

shell docker kafka docker-compose zookeeper kafka-rest-proxy kafka-schema-registry kafka-topics-ui

Sephsplace

My own version of r/place, done in a weekend

✭ 119

javascript kafka cqrs event-sourcing hackathon

Fast Data Dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors

✭ 1,707

shell HTML Dockerfile javascript docker kafka schema-registry dataops kafka-rest-proxy

Kafka Flow

KafkaFlow is a .NET framework to consume and produce Kafka messages with multi-threading support. It's very simple to use and very extendable. You just need to install, configure, start/stop the bus with your app and create a middleware/handler to process the messages.

✭ 118

csharp dotnet kafka dotnet-core nuget workers consumer

Awesome Kafka

A collection of kafka-resources

✭ 116

awesome awesome-list kafka awesome-lists kafka-streams

Cmak

CMAK is a tool for managing Apache Kafka clusters

✭ 10,544

scala HTML shell kafka big-data cluster-management

Spec

The AsyncAPI specification allows you to create machine-readable definitions of your asynchronous APIs.

✭ 1,860

javascript hacktoberfest kafka websockets mqtt reactive amqp specification stomp asyncapi

Kafka Connect Tools

Kafka Connect Tooling

✭ 115

scala cli kafka kafka-connect

Lenses Docker

❤for real-time DataOps - where the application and data fabric blends - Lenses

✭ 115

shell docker security kubernetes kafka openshift enterprise

Mmo Server

Distributed Java game server, including login, gateway, game demo

✭ 114

java game spring-boot server kafka grpc netty zookeeper game-server rpg mmo

Ultimate Go

This repo contains my notes on working with Go and computer systems.

✭ 1,530

go algorithms kafka docker-compose data-structures grpc example gin gorm

Kkbinlog

支持mysql、MongoDB数据变更订阅分发

✭ 112

java mysql mongodb spring-boot spring kafka rabbitmq etcd

Myth

Reliable messages resolve distributed transactions

✭ 1,470

java Vue spring kafka translation springboot rabbitmq protobuf dubbo springcloud aop rocketmq activemq soa motan kryo

Schema Registry

Confluent Schema Registry for Kafka

✭ 1,647

java json rest-api kafka protobuf schema json-schema avro confluent schemas schema-registry avro-schema protobuf-schema

Bireme

Bireme is an incremental synchronization tool for the Greenplum / HashData data warehouse

✭ 110

java mysql postgresql kafka synchronization incremental

Go Kafka Example

Golang Kafka consumer and producer example

✭ 108

go golang kafka examples stream-processing

Springboot Labs

一个涵盖六个专栏：Spring Boot 2.X、Spring Cloud、Spring Cloud Alibaba、Dubbo、分布式消息队列、分布式事务的仓库。希望胖友小手一抖，右上角来个 Star，感恩 1024

✭ 12,804

java HTML spring-boot kafka rabbitmq spring-cloud dubbo rocketmq activemq spring-cloud-alibaba

Flink Learning

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

✭ 11,378