All Projects → E-SoulDataGroup → Spark_streaming_kafka_offset

E-SoulDataGroup / Spark_streaming_kafka_offset

SparkStreaming中利用MySQL保存Kafka偏移量保证0数据丢失

Programming Languages

scala
5932 projects

Projects that are alternatives of or similar to Spark streaming kafka offset

Workflow
C++ Parallel Computing and Asynchronous Networking Engine
Stars: ✭ 6,680 (+16192.68%)
Mutual labels:  kafka, mysql
Books Recommendation
程序员进阶书籍(视频),持续更新(Programmer Books)
Stars: ✭ 558 (+1260.98%)
Mutual labels:  kafka, mysql
Gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Stars: ✭ 398 (+870.73%)
Mutual labels:  kafka, mysql
Spring Cloud Shop
spring cloud 版分布式电商项目,全力打造顶级多模块,高可用,高扩展电商项目
Stars: ✭ 248 (+504.88%)
Mutual labels:  kafka, mysql
Nagios Plugins
450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...
Stars: ✭ 1,000 (+2339.02%)
Mutual labels:  kafka, mysql
Datafaker
Datafaker is a large-scale test data and flow test data generation tool. Datafaker fakes data and inserts to varied data sources. 测试数据生成工具
Stars: ✭ 327 (+697.56%)
Mutual labels:  kafka, mysql
Javakeeper
✍️ Java 工程师必备架构体系知识总结:涵盖分布式、微服务、RPC等互联网公司常用架构,以及数据存储、缓存、搜索等必备技能
Stars: ✭ 502 (+1124.39%)
Mutual labels:  kafka, mysql
Synch
Sync data from the other DB to ClickHouse(cluster)
Stars: ✭ 200 (+387.8%)
Mutual labels:  kafka, mysql
Szt Bigdata
深圳地铁大数据客流分析系统🚇🚄🌟
Stars: ✭ 826 (+1914.63%)
Mutual labels:  kafka, mysql
Demo Scene
👾Scripts and samples to support Confluent Demos and Talks. ⚠️Might be rough around the edges ;-) 👉For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/
Stars: ✭ 806 (+1865.85%)
Mutual labels:  kafka, mysql
Storagetapper
StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service
Stars: ✭ 232 (+465.85%)
Mutual labels:  kafka, mysql
Javaok
必看!java后端,亮剑诛仙。java发展路线技术要点。
Stars: ✭ 867 (+2014.63%)
Mutual labels:  kafka, mysql
Devops Bash Tools
550+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Kafka, Docker, APIs, Hadoop, SQL, PostgreSQL, MySQL, Hive, Impala, Travis CI, Jenkins, Concourse, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, .tmux.conf, .psqlrc ...
Stars: ✭ 226 (+451.22%)
Mutual labels:  kafka, mysql
Full Stack Notes
全栈工程师手册
Stars: ✭ 366 (+792.68%)
Mutual labels:  kafka, mysql
Bcmall
以教学为目的的电商系统。包含ToB复杂业务、互联网高并发业务、缓存应用;DDD、微服务指导。模型驱动、数据驱动。了解大型服务进化路线,编码技巧、学习Linux,性能调优。Docker/k8s助力、监控、日志收集、中间件学习。前端技术、后端实践等。主要技术:SpringBoot+JPA+Mybatis-plus+Antd+Vue3。
Stars: ✭ 188 (+358.54%)
Mutual labels:  kafka, mysql
Cookbook
🎉🎉🎉JAVA高级架构师技术栈==任何技能通过 “刻意练习” 都可以达到融会贯通的境界,就像烹饪一样,这里有一份JAVA开发技术手册,只需要增加自己练习的次数。🏃🏃🏃
Stars: ✭ 428 (+943.9%)
Mutual labels:  kafka, mysql
Back End Interview
后端面试题汇总(Python、Redis、MySQL、PostgreSQL、Kafka、数据结构、算法、编程、网络)
Stars: ✭ 188 (+358.54%)
Mutual labels:  kafka, mysql
Istio Micro
istio 微服务示例代码 grpc+protobuf+echo+websocket+mysql+redis+kafka+docker-compose
Stars: ✭ 194 (+373.17%)
Mutual labels:  kafka, mysql
Pmacct
pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].
Stars: ✭ 677 (+1551.22%)
Mutual labels:  kafka, mysql
Kudo
Kubernetes Universal Declarative Operator (KUDO)
Stars: ✭ 849 (+1970.73%)
Mutual labels:  kafka, mysql

spark_streaming_kafka_offset

  • 在Spark Streaming中使用Kafka作为数据源
  • 利用MySQL保存Kafka offset偏移量来记录消费的位置
  • Spark程序重启后可以从上次消费位置继续消费,不会丢失数据
  • 使用checkpoint保证状态结果不丢失
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].