GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → skynyrd → Kafka Connect Elastic Sink

skynyrd / Kafka Connect Elastic Sink

Kafka connect Elastic sink connector, with just in time index/delete behaviour.

Programming Languages

java
68154 projects - #9 most used programming language

Labels

elasticsearchkafkakafka-connect

Projects that are alternatives of or similar to Kafka Connect Elastic Sink

Dataengineeringproject
Example end to end data engineering project.
Stars: ✭ 82 (+256.52%)
Mutual labels:  kafka, kafka-connect, elasticsearch
Kafka Connect Elasticsearch Source
Kafka Connect Elasticsearch Source
Stars: ✭ 22 (-4.35%)
Mutual labels:  kafka, kafka-connect, elasticsearch
Stream Reactor
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
Stars: ✭ 753 (+3173.91%)
Mutual labels:  kafka, kafka-connect, elasticsearch
Demo Scene
👾Scripts and samples to support Confluent Demos and Talks. ⚠️Might be rough around the edges ;-) 👉For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/
Stars: ✭ 806 (+3404.35%)
Mutual labels:  kafka, kafka-connect, elasticsearch
Kafka Connect Ui
Web tool for Kafka Connect |
Stars: ✭ 388 (+1586.96%)
Mutual labels:  kafka, kafka-connect, elasticsearch
Kafka Connect Elasticsearch
Kafka Connect Elasticsearch connector
Stars: ✭ 550 (+2291.3%)
Mutual labels:  kafka, kafka-connect, elasticsearch
Kafkacenter
KafkaCenter is a unified platform for Kafka cluster management and maintenance, producer / consumer monitoring, and use of ecological components.
Stars: ✭ 896 (+3795.65%)
Mutual labels:  kafka, kafka-connect
Ksql
The database purpose-built for stream processing applications.
Stars: ✭ 4,668 (+20195.65%)
Mutual labels:  kafka, kafka-connect
All Things Cqrs
Comprehensive guide to a couple of possible ways of synchronizing two states with Spring tools. Synchronization is shown by separating command and queries in a simple CQRS application.
Stars: ✭ 474 (+1960.87%)
Mutual labels:  kafka, kafka-connect
Debezium
Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
Stars: ✭ 5,937 (+25713.04%)
Mutual labels:  kafka, kafka-connect
Gohangout
golang版本的hangout, 希望能省些内存. 使用了自己写的Kafka lib .. 虚. 不过我们在生产环境已经使用近1年, kafka 版本从0.9.0.1到2.0都在使用, 目前情况稳定. 吞吐量在每天2000亿条以上.
Stars: ✭ 507 (+2104.35%)
Mutual labels:  kafka, elasticsearch
Javakeeper
✍️ Java 工程师必备架构体系知识总结：涵盖分布式、微服务、RPC等互联网公司常用架构，以及数据存储、缓存、搜索等必备技能
Stars: ✭ 502 (+2082.61%)
Mutual labels:  kafka, elasticsearch
Cookbook
🎉🎉🎉JAVA高级架构师技术栈==任何技能通过 “刻意练习” 都可以达到融会贯通的境界，就像烹饪一样，这里有一份JAVA开发技术手册，只需要增加自己练习的次数。🏃🏃🏃
Stars: ✭ 428 (+1760.87%)
Mutual labels:  kafka, elasticsearch
Gpmall
【咕泡学院实战项目】-基于SpringBoot+Dubbo构建的电商平台-微服务架构、商城、电商、微服务、高并发、kafka、Elasticsearch
Stars: ✭ 4,241 (+18339.13%)
Mutual labels:  kafka, elasticsearch
Hangout
用java实现一下Logstash的几个常用input/filter/output, 希望能有效率上面的大提升. 现在我们迁移到golang了 https://github.com/childe/gohangout
Stars: ✭ 469 (+1939.13%)
Mutual labels:  kafka, elasticsearch
Gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Stars: ✭ 398 (+1630.43%)
Mutual labels:  kafka, elasticsearch
Books Recommendation
程序员进阶书籍（视频），持续更新（Programmer Books）
Stars: ✭ 558 (+2326.09%)
Mutual labels:  kafka, elasticsearch
Goodskill
🐂基于springcloud +dubbo构建的模拟秒杀项目，模块化设计，集成了分库分表、elasticsearch🔍、gateway、mybatis-plus、spring-session等常用开源组件
Stars: ✭ 786 (+3317.39%)
Mutual labels:  kafka, elasticsearch
Kafka Connect Github Source
Get a stream of issues and pull requests for your chosen GitHub repository
Stars: ✭ 327 (+1321.74%)
Mutual labels:  kafka, kafka-connect
Springboot Learning
基于Gradle构建，使用SpringBoot在各个场景的应用，包括集成消息中间件、前后端分离、数据库、缓存、分布式锁、分布式事务等
Stars: ✭ 340 (+1378.26%)
Mutual labels:  kafka, elasticsearch
View All Similar Projects ➔

Kafka Connect Elastic Sink Connector

Default Elastic sink connector and open source alternatives read data from Kafka topic, and index/delete them with respect to startup configuration.

This custom connector created for reading the configuration from the data itself.

That is,

  • If data has "status" field set to "insert", then connector sends index request.
  • If data has "status" field set to "delete", then connector delete request.

"status" flag name is configurable.

About record

Your record must be a JSON string and these fields should be included:

  • flag : Used to get behaviour, should be insert or delete, field name flag is configurable by flag.field property. (e.g. status in example configuration below)
  • You can also change insert or delete values Constants.java file.
  • payload: To send Elastic, should contain Json data, field name dataList is configurable by data.array property. (e.g. dataList in example configuration below). Payload can be array or object.
  • Your data in dataList must include id field. You can change id field name from Constants.java

Example Configuration

elastic.url=elasticsearch
name=ElasticSinkConnector
topics=first_topic
tasks.max=1
type.name=targettype
connector.class=com.skynyrd.kafka.ElasticSinkConnector
elastic.port=9200
index.name=targetindex
flag.field=status
data.array=dataList
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
key.converter=org.apache.kafka.connect.storage.StringConverter

Run with local Kafka cluster and Elastic instance

You should have Docker and Docker Compose to run and test it in your local.

  • Clone this repo
  • mvn clean and mvn install
  • Change absolute path in volumes under kafka-cluster in docker-compose.yml
  • Run with docker-compose up
  • Wait a bit and open connect console http://localhost:3030/kafka-connect-ui/#/cluster/fast-data-dev/select-connector
  • Select ElasticSinkConnector in right bottom of the page. (Appears if volume is correctly set in compose file.)
  • For logs: http://localhost:3030/logs/connect-distributed.log

Curious about how I implemented this repository? Here is my medium post

TODOs

  • Missing tests.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].