All Categories → Data Processing → spark

Top 625 spark open source projects

SparkJobServerClient

Java Client of the Spark Job Server implementing the arranged Rest APIs

✭ 50

java spark jar java-client pom

spark-root

Apache Spark Data Source for ROOT File Format

✭ 28

Jupyter Notebook scala big-data spark root histogrammar

oshinko-s2i

This is a place to put s2i images and utilities for spark application builders for openshift

✭ 16

shell Dockerfile scala Makefile python java awk spark openshift pyspark s2i-image oshinko-s2i

datalake-etl-pipeline

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

✭ 39

python big-data spark apache-spark hadoop etl xml xml-parsing pyspark data-pipeline datalake hadoop-mapreduce spark-sql etl-framework hadoop-hdfs etl-pipeline etl-components

gan deeplearning4j

Automatic feature engineering using Generative Adversarial Networks using Deeplearning4j and Apache Spark.

✭ 19

java Jupyter Notebook data-science machine-learning big-data spark apache-spark computer-vision deep-learning bigdata datascience generative-adversarial-network gan machinelearning deeplearning generative-adversarial-networks deeplearning4j

qs-hadoop

大数据生态圈学习

✭ 18

java scala elasticsearch spark hadoop storm bigdata spark-streaming mapreduce

dockerfiles

Multi docker container images for main Big Data Tools. (Hadoop, Spark, Kafka, HBase, Cassandra, Zookeeper, Zeppelin, Drill, Flink, Hive, Hue, Mesos, ... )

✭ 29

shell Dockerfile python Makefile Batchfile XSLT javascript dockerfile kafka spark cassandra hive hadoop docker-image bigdata hbase zookeeper mesos hue flink zeppelin drill

flask-spark-docker

Just a boilerplate for PySpark and Flask

✭ 32

python Dockerfile HTML javascript CSS docker flask spark pyspark redis-queue

zingg

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

the-apache-ignite-book

All code samples, scripts and more in-depth examples for The Apache Ignite Book. Include Apache Ignite 2.6 or above

✭ 65

java streaming memoization sql spark hive hadoop spring-data bigdata hibernate distributed-database ignite nosql-database in-memory-database streaming-data gridgain hibernate-ogm in-memory-computations in-memory-caching

zdh web

大数据采集,抽取平台

✭ 292

java HTML CSS data collection spark etl scheduler bigdata data-collection datapipeline pipline sparketl datax-web

mongo-spark-jupyter

Docker environment that spins up MongoDB replica set, Spark, and Jupyter Lab. Example code uses PySpark and the MongoDB Spark Connector.

✭ 29

python shell powershell Dockerfile spark mongodb jypyter jyputer-notebook

sbt-spark-submit

sbt plugin for spark-submit

✭ 94

scala spark sbt

osm4scala

Scala and Spark library focused on reading OpenStreetMap Pbf files.

✭ 62

scala javascript Jupyter Notebook CSS spark openstreetmap osm gis pbf openstreetmap-pbf-files

spark-notebook-examples

Some notebook examples related to Apache Spark, IPython / Jupyter, Zeppelin

✭ 49

Jupyter Notebook spark jupyter notebook ipython zeppelin

spark-on-k8s-gcp-examples

Example Spark applications that run on Kubernetes and access GCP products, e.g., GCS, BigQuery, and Cloud PubSub

✭ 36

java shell kubernetes bigquery spark gcs cloud-pubsub gcs-connector

opaque-sql

An encrypted data analytics platform

✭ 169

scala C++assembly shell python c security machine-learning privacy spark analytics enclave spark-sql

uberscriptquery

UberScriptQuery, a SQL-like DSL to make writing Spark jobs super easy

✭ 54

java ANTLR scala spark uber

iis

Information Inference Service of the OpenAIRE system

✭ 16

java HTML PigLatin python scala shell HiveQL text-mining data-mining big-data spark hadoop iis openaire information-inference data-processing-system

local-hashicorp-stack

Local Hashicorp Stack for DevOps Development without Hypervisor or Cloud

✭ 23

HCL shell spark packer ubuntu virtualbox consul terraform hdfs hashicorp nomad

epitweetr

ECDC Early warning tool using Twitter data

✭ 50

scala r HTML machine-learning twitter spark lucene signal-detection early-warning-systems epidemic-surveillance

SparkFastDataAnalysis

《Spark 快速大数据分析》学习笔记

✭ 26

java scala python Batchfile spark

featuretoolsOnSpark

A simplified version of featuretools for Spark

✭ 24

python machine-learning spark feature-engineering automl automated-machine-learning automated-feature-engineering featuretools deep-feature-synthesis

interview-refresh-java-bigdata

a one-stop repo to lookup for code snippets of core java concepts, sql, data structures as well as big data. It also consists of interview questions asked in real-life.

✭ 25

java snippets kafka spark interview recursion garbage-collection spark-streaming mapreduce java-collection

seatunnel-example

seatunnel plugin developing examples.

✭ 27

scala java spark spark-streaming flink sql-engine etl-framework waterdrop etl-pipeline

LSTM-TensorSpark

Implementation of a LSTM with TensorFlow and distributed on Apache Spark

✭ 40

python distributed-systems machine-learning spark tensorflow distributed-computing lstm rnn

cdp-spark-datasource

Spark data source for Cognite Data Fusion

✭ 18

scala spark datasource cognite

openblockchain

{START HERE} docker engine to roll your own openblockchain

✭ 16

search-engine crypto spark bitcoin docker-engine cryptocurrency openblockchain

rulegin

基于JavaScript Engine的轻量级规则引擎系统，重构于开源IOT项目thingboard

✭ 100

java kafka spark netty zk sping grpc-java

open-stream-processing-benchmark

This repository contains the code base for the Open Stream Processing Benchmark.

✭ 37

Jupyter Notebook scala shell Dockerfile distributed-systems benchmark real-time kafka spark distributed-computing stream-processing spark-streaming flink kafka-streams benchmark-suite real-time-processing structured-streaming

algobox

Open Source algorithmic trading platform in Java / Python

✭ 90

javascript python java CSS HTML Jupyter Notebook shell spark trading trading-platform trading-strategies algorithmic-trading

almaren-framework

The Almaren Framework provides a simplified consistent minimalistic layer over Apache Spark. While still allowing you to take advantage of native Apache Spark features. You can still combine it with standard Spark code.

✭ 30

scala spark

T-Watch

Real Time Twitter Sentiment Analysis Product

✭ 20