Top 625 spark open source projects

Coursera's big data course with Scala and Spark

Sparkling Water provides H2O functionality inside Spark cluster

深圳地铁大数据客流分析系统🚇🚄🌟

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

A connector for Spark that allows reading and writing to/from Redis cluster

R interface for Apache Spark

A Flexible and Powerful Parameter Server for large-scale machine learning

学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等

An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset

CTR prediction model based on spark(LR, GBDT, DNN)

hadoop各组件使用，持续更新

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Expressive types for Spark.

Scalable genomic data analysis.

Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

电商用户行为分析大数据平台

Example project implementing best practices for PySpark ETL jobs and applications.

A cohesive & pragmatic framework of FP centric Scala libraries

macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

DataFusion has now been donated to the Apache Arrow project

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

The MongoDB Spark Connector

Alluxio, data orchestration for analytics and machine learning in the cloud

Learning Apache spark,including code and data .Most part can run local.

Essential Spark extensions and helper methods ✨😲

A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs.

Training of Locally Optimized Product Quantization (LOPQ) models for approximate nearest neighbor search of high dimensional data in Python and Spark.

Real Time Analytics and Data Pipelines based on Spark Streaming

An open source framework for building data analytic applications.

Geo Spatial Data Analytics on Spark

Data validation and organization of metadata for data frames and database tables

编程电子书，电子书，编程书籍，包括C，C#，Docker，Elasticsearch，Git，Hadoop，HeadFirst，Java，Javascript，jvm，Kafka，Linux，Maven，MongoDB，MyBatis，MySQL，Netty，Nginx，Python，RabbitMQ，Redis，Scala，Solr，Spark，Spring，SpringBoot，SpringCloud，TCPIP，Tomcat，Zookeeper，人工智能，大数据类，并发编程，数据库类，数据挖掘，新面试题，架构设计，算法系列，计算机类，设计模式，软件测试，重构优化，等更多分类

Cross-platform real-time collaboration client optimized for business and organizations.

大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

大数据博客、笔试题、教程、项目、面经的整理

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

Examples for High Performance Spark

Web UI for Trino, Presto, Hive, Elasticsearch, SparkSQL

Tools for handling firmwares of DJI products, with focus on quadcopters.

Moonbox is a DVtaaS (Data Virtualization as a Service) Platform

A Scala feature transformation library for data science and machine learning

Scala examples for learning to use Spark

Server for the ListenBrainz project

Haskell on Apache Spark.

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.

Generic Data Ingestion & Dispersal Library for Hadoop

Tools for reading data from Solr as a Spark RDD and indexing objects from Spark into Solr using SolrJ.

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Java全栈知识架构体系总结

一个大数据架构师应该掌握的技能

Iceberg is a table format for large, slow-moving tabular data

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

Learn and understand Docker technologies, with real DevOps practice!

Building Large-Scale AI Applications for Distributed Big Data

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.