lackhurt / Flume Canal Source
Licence: apache-2.0
Flume NG Canal source
Stars: ✭ 56
Programming Languages
java
68154 projects - #9 most used programming language
Projects that are alternatives of or similar to Flume Canal Source
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (+85.71%)
Mutual labels: kafka, mysql, elasticsearch, mq
Echo
🦄 开源社区系统：基于 SpringBoot + MyBatis + MySQL + Redis + Kafka + Elasticsearch + Spring Security + ... 并提供详细的开发文档和配套教程。包含帖子、评论、私信、系统通知、点赞、关注、搜索、用户设置、数据统计等模块。
Stars: ✭ 129 (+130.36%)
Mutual labels: kafka, mysql, elasticsearch
Flink Learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Stars: ✭ 11,378 (+20217.86%)
Mutual labels: kafka, mysql, elasticsearch
Operators
Collection of Kubernetes Operators built with KUDO.
Stars: ✭ 175 (+212.5%)
Mutual labels: kafka, mysql, elasticsearch
Learningsummary
涵盖大部分Java进阶需要掌握的知识，包括【微服务】【中间件】【缓存】【数据库优化】【搜索引擎】【分布式】等等，欢迎Star~
Stars: ✭ 201 (+258.93%)
Mutual labels: mysql, elasticsearch, mq
Dcos Commons
DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.
Stars: ✭ 162 (+189.29%)
Mutual labels: kafka, elasticsearch, hdfs
Nagios Plugins
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
Stars: ✭ 1,000 (+1685.71%)
Mutual labels: kafka, mysql, elasticsearch
Gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Stars: ✭ 398 (+610.71%)
Mutual labels: kafka, mysql, elasticsearch
Kafka Connect Ui
Web tool for Kafka Connect |
Stars: ✭ 388 (+592.86%)
Mutual labels: kafka, elasticsearch, hdfs
Demo Scene
👾Scripts and samples to support Confluent Demos and Talks. ⚠️Might be rough around the edges ;-) 👉For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/
Stars: ✭ 806 (+1339.29%)
Mutual labels: kafka, mysql, elasticsearch
Javakeeper
✍️ Java 工程师必备架构体系知识总结：涵盖分布式、微服务、RPC等互联网公司常用架构，以及数据存储、缓存、搜索等必备技能
Stars: ✭ 502 (+796.43%)
Mutual labels: kafka, mysql, elasticsearch
Spring Boot Vue Bank
我,请始皇[打钱]是一个前后端分离的工具人系统，项目采用 SpringBoot+Go+Vue 开发，项目加入常见的企业级应用所涉及到的技术点，例如 Redis、RabbitMQ 等(主要是多用用工具多踩踩坑)。
Stars: ✭ 157 (+180.36%)
Mutual labels: kafka, mysql, mq
Storagetapper
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Stars: ✭ 232 (+314.29%)
Mutual labels: kafka, mysql, hdfs
Cookbook
🎉🎉🎉JAVA高级架构师技术栈==任何技能通过 “刻意练习” 都可以达到融会贯通的境界，就像烹饪一样，这里有一份JAVA开发技术手册，只需要增加自己练习的次数。🏃🏃🏃
Stars: ✭ 428 (+664.29%)
Mutual labels: kafka, mysql, elasticsearch
Books Recommendation
程序员进阶书籍（视频），持续更新（Programmer Books）
Stars: ✭ 558 (+896.43%)
Mutual labels: kafka, mysql, elasticsearch
Mysql Es
Sync MySQL to ElasticSearch, Support Relationship
Stars: ✭ 53 (-5.36%)
Mutual labels: mysql, elasticsearch
Goodskill
🐂基于springcloud +dubbo构建的模拟秒杀项目，模块化设计，集成了分库分表、elasticsearch🔍、gateway、mybatis-plus、spring-session等常用开源组件
Stars: ✭ 786 (+1303.57%)
Mutual labels: kafka, elasticsearch
Kafka Connect Elasticsearch Source
Kafka Connect Elasticsearch Source
Stars: ✭ 22 (-60.71%)
Mutual labels: kafka, elasticsearch
flume-canal-source
flume version : 1.7.0
canal version : 1.0.24
flume 是什么
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
canal 是什么
https://github.com/alibaba/canal
阿里巴巴mysql数据库binlog的增量订阅&消费组件
flume-canal-source 做了什么
flume-canal-source 是对 flume 的 source 扩展。从 canal 获取数据到 flume channel。 进而可以实现binlog数据到 kafka / hdfs / hive / elasticsearch 等等。 **canal 和 flume 都有高可用的解决方案，这种方式同步 binlog 可用性非常高。**组合前人的优秀轮子，不重复造轮子。
如何使用
部署 canal、flume 这里忽略。
配置 flume
- 配置 source 类型*
agent.sources = canalSource
agent.sources.canalSource.type = com.weiboyi.etl.flume.source.canal.CanalSource
- 配置连接 canal 的三种方式*
# 1. zookeeper servers
agent.sources.canalSource.zkServers = zookeeper-host:2181
# 2. canal server urls
agent.sources.canalSource.serverUrls = canal-server1:111111,canal-server2:111111
# 3. canal server urls
agent.sources.canalSource.serverUrl = canal-server1:111111
- 配置 canal destination*
agent.sources.canalSource.destination = example
- 配置用户名密码
agent.sources.canalSource.username = user
agent.sources.canalSource.password = passwd
- binlog batch size, default 1024
agent.sources.canalSource.batchSize = 1024
- 是否需要 MySQL 修改前的数据, default true
agent.sources.canalSource.oldDataRequired = true
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].