All Projects → Azure → Azure Event Hubs Spark

Azure / Azure Event Hubs Spark

Licence: apache-2.0
Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Programming Languages

scala
5932 projects

Projects that are alternatives of or similar to Azure Event Hubs Spark

Spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Stars: ✭ 1,721 (+1129.29%)
Mutual labels:  azure, microsoft, spark, bigdata, apache-spark, spark-streaming, streaming
Azure Event Hubs
☁️ Cloud-scale telemetry ingestion from any stream of data with Azure Event Hubs
Stars: ✭ 233 (+66.43%)
Mutual labels:  azure, microsoft, stream, spark, streaming, apache
Data Accelerator
Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Stars: ✭ 247 (+76.43%)
Mutual labels:  azure, kafka, spark, apache-spark, spark-streaming, streaming
Sparta
Real Time Analytics and Data Pipelines based on Spark Streaming
Stars: ✭ 513 (+266.43%)
Mutual labels:  kafka, spark, spark-streaming, streaming, real-time
Mobius
C# and F# language binding and extensions to Apache Spark
Stars: ✭ 929 (+563.57%)
Mutual labels:  spark, bigdata, apache-spark, spark-streaming, streaming
Azure Event Hubs For Kafka
Azure Event Hubs for Apache Kafka Ecosystems
Stars: ✭ 124 (-11.43%)
Mutual labels:  azure, microsoft, kafka, apache
Mmlspark
Simple and Distributed Machine Learning
Stars: ✭ 2,899 (+1970.71%)
Mutual labels:  azure, microsoft, spark, apache-spark
Real Time Stream Processing Engine
This is an example of real time stream processing using Spark Streaming, Kafka & Elasticsearch.
Stars: ✭ 37 (-73.57%)
Mutual labels:  kafka, spark, apache-spark, spark-streaming
Bigdata Notebook
Stars: ✭ 100 (-28.57%)
Mutual labels:  kafka, spark, bigdata, streaming
Streamline
StreamLine - Streaming Analytics
Stars: ✭ 151 (+7.86%)
Mutual labels:  kafka, spark-streaming, streaming, real-time
Spark States
Custom state store providers for Apache Spark
Stars: ✭ 83 (-40.71%)
Mutual labels:  spark, apache-spark, spark-streaming, apache
Bigdataguide
大数据学习,从零开始学习大数据,包含大数据学习各阶段学习视频、面试资料
Stars: ✭ 817 (+483.57%)
Mutual labels:  kafka, spark, bigdata
Bigdata Interview
🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结
Stars: ✭ 857 (+512.14%)
Mutual labels:  kafka, spark, bigdata
Spark Streaming Monitoring With Lightning
Plot live-stats as graph from ApacheSpark application using Lightning-viz
Stars: ✭ 15 (-89.29%)
Mutual labels:  bigdata, apache-spark, spark-streaming
Stream Reactor
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
Stars: ✭ 753 (+437.86%)
Mutual labels:  kafka, connector, streaming
Kafka Storm Starter
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Stars: ✭ 728 (+420%)
Mutual labels:  kafka, spark, apache-spark
Pyspark Examples
Code examples on Apache Spark using python
Stars: ✭ 58 (-58.57%)
Mutual labels:  spark, spark-streaming, apache
Hydra
A real-time data replication platform that "unbundles" the receiving, transforming, and transport of data streams.
Stars: ✭ 68 (-51.43%)
Mutual labels:  kafka, streaming, real-time
Example Spark Kafka
Apache Spark and Apache Kafka integration example
Stars: ✭ 120 (-14.29%)
Mutual labels:  kafka, spark, spark-streaming
Streamx
kafka-connect-s3 : Ingest data from Kafka to Object Stores(s3)
Stars: ✭ 96 (-31.43%)
Mutual labels:  kafka, connector, streaming

Azure Event Hubs + Apache Spark Connector

Azure Event Hubs Connector for Apache Spark

chat on gitter build status star our repo

This is the source code of the Azure Event Hubs Connector for Apache Spark.

Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them into multiple applications. Spark Streaming and Structured Streaming are scalable and fault-tolerant stream processing engines that allow users to process huge amounts of data using complex algorithms expressed with high-level functions like map, reduce, join, and window. This data can then be pushed to filesystems, databases, or even back to Event Hubs.

By making Event Hubs and Spark easier to use together, we hope this connector makes building scalable, fault-tolerant applications easier for our users.

Latest Releases

Spark

Spark Version Package Name Package Version
Spark 3.0 azure-eventhubs-spark_2.12 Maven Central
Spark 2.4 azure-eventhubs-spark_2.11 Maven Central
Spark 2.4 azure-eventhubs-spark_2.12 Maven Central
Spark 2.3 azure-eventhubs-spark_2.11 Maven Central
Spark 2.2 azure-eventhubs-spark_2.11 Maven Central
Spark 2.1 azure-eventhubs-spark_2.11 Maven Central

Databricks

Databricks Runtime Version Artifact Id Package Version
Databricks Runtime 7.X azure-eventhubs-spark_2.12 Maven Central
Databricks Runtime 6.X azure-eventhubs-spark_2.11 Maven Central
Databricks Runtime 6.X azure-eventhubs-spark_2.12 Maven Central
Databricks Runtime 5.X azure-eventhubs-spark_2.11 Maven Central
Databricks Runtime 5.X azure-eventhubs-spark_2.12 Maven Central
Databricks Runtime 4.X azure-eventhubs-spark_2.11 Maven Central
Databricks Runtime 3.5 azure-eventhubs-spark_2.11 Maven Central

Roadmap

There is an open issue for each planned feature/enhancement.

FAQ

We maintain an FAQ - reach out to us via gitter if you think anything needs to be added or clarified!

Usage

Linking

For Scala/Java applications using SBT/Maven project definitions, link your application with the artifact below. Note: See Latest Releases to find the correct artifact for your version of Apache Spark (or Databricks)!

groupId = com.microsoft.azure
artifactId = azure-eventhubs-spark_2.11
version = 2.3.18

or

groupId = com.microsoft.azure
artifactId = azure-eventhubs-spark_2.12
version = 2.3.18

Documentation

Documentation for our connector can be found here. The integration guides there contain all the information you need to use this library.

If you're new to Apache Spark and/or Event Hubs, then we highly recommend reading their documentation first. You can read Event Hubs documentation here, documentation for Spark Streaming here, and, the last but not least, Structured Streaming here.

Further Assistance

If you need additional assistance, please don't hesitate to ask! General questions and discussion should happen on our gitter chat. Please open an issue for bug reports and feature requests! Feedback, feature requests, bug reports, etc are all welcomed!

Contributing

If you'd like to help contribute (we'd love to have your help!), then go to our Contributor's Guide for more information.

Build Prerequisites

In order to use the connector, you need to have:

More details on building from source and running tests can be found in our Contributor's Guide.

Build Command

// Builds jar and runs all tests
mvn clean package

// Builds jar, runs all tests, and installs jar to your local maven repository
mvn clean install
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].