knoldus / Real Time Stream Processing Engine
Licence: apache-2.0
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
Stars: ✭ 37
Programming Languages
scala
5932 projects
Projects that are alternatives of or similar to Real Time Stream Processing Engine
Azure Event Hubs Spark
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs
Stars: ✭ 140 (+278.38%)
Mutual labels: kafka, spark, apache-spark, spark-streaming
Gimel
Big Data Processing Framework - Unified Data API or SQL on Any Storage
Stars: ✭ 216 (+483.78%)
Mutual labels: kafka, spark, spark-streaming, elasticsearch
Data Accelerator
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Stars: ✭ 247 (+567.57%)
Mutual labels: kafka, spark, apache-spark, spark-streaming
Logisland
Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.
Stars: ✭ 97 (+162.16%)
Mutual labels: kafka, spark, elasticsearch
Flink Learning
flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》
Stars: ✭ 11,378 (+30651.35%)
Mutual labels: kafka, spark, elasticsearch
Bigdata Playground
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Stars: ✭ 177 (+378.38%)
Mutual labels: kafka, apache-spark, spark-streaming
Spark Streaming With Kafka
Self-contained examples of Apache Spark streaming integrated with Apache Kafka.
Stars: ✭ 180 (+386.49%)
Mutual labels: kafka, spark, spark-streaming
Coolplayspark
酷玩 Spark: Spark 源代码解析、Spark 类库等
Stars: ✭ 3,318 (+8867.57%)
Mutual labels: spark, apache-spark, spark-streaming
Wirbelsturm
Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
Stars: ✭ 332 (+797.3%)
Mutual labels: kafka, spark, apache-spark
Agile data code 2
Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
Stars: ✭ 413 (+1016.22%)
Mutual labels: kafka, spark, apache-spark
Spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Stars: ✭ 1,721 (+4551.35%)
Mutual labels: spark, apache-spark, spark-streaming
Spark States
Custom state store providers for Apache Spark
Stars: ✭ 83 (+124.32%)
Mutual labels: spark, apache-spark, spark-streaming
Example Spark Kafka
Apache Spark and Apache Kafka integration example
Stars: ✭ 120 (+224.32%)
Mutual labels: kafka, spark, spark-streaming
Kafka Storm Starter
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Stars: ✭ 728 (+1867.57%)
Mutual labels: kafka, spark, apache-spark
Sparta
Real Time Analytics and Data Pipelines based on Spark Streaming
Stars: ✭ 513 (+1286.49%)
Mutual labels: kafka, spark, spark-streaming
Mobius
C# and F# language binding and extensions to Apache Spark
Stars: ✭ 929 (+2410.81%)
Mutual labels: spark, apache-spark, spark-streaming
Goodreads etl pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Stars: ✭ 793 (+2043.24%)
Mutual labels: spark, apache-spark
Goodskill
🐂基于springcloud +dubbo构建的模拟秒杀项目，模块化设计，集成了分库分表、elasticsearch🔍、gateway、mybatis-plus、spring-session等常用开源组件
Stars: ✭ 786 (+2024.32%)
Mutual labels: kafka, elasticsearch
#real-time-stream-processing-engine
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
#Pre-Requisites for this project ####Elasticsearch Setup i) Download the Elasticsearch 5.0.0-alpha5 or latest version and unzip it.
ii) Run the following command.
$ bin/elasticsearch
####Kafka Setup i) Download the Kafka-0.10.0.1 or latest version and unzip it.
ii) Run the following commands for starting Zookeeper & Kafka:
$ bin/zookeeper-server-start.sh config/zookeeper.properties
$ bin/kafka-server-start.sh config/server.properties
###Getting Started:
Clone and run in local mode:
$ [email protected]:knoldus/sentiment-analysis-engine.git
$ cd sentiment-analysis-engine
$ bin/activator run
##Limitations
-
Elasticsearch 5.0.0-alpha5 is not production ready.
-
Kafka-0.10.0.1 is experimental in current spark version(spark-2.0.0).
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].