9362 Open source projects that are alternatives of or similar to Spark With Python

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Apache Hive

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

A Clojure dataframe library that runs on Spark

Griffon Data Science Virtual Machine

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

Apache Spark Connector for Azure Cosmos DB

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of data in real-time.

A complete (distributed) BigData stack, running in containers

Big Data Ecosystem Docker

Simple and Distributed Machine Learning

Big Data for Data Engineers Coursera Specialization from Yandex

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

Web scrapping and related analytics using Python tools

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

A simplified, lightweight ETL Framework based on Apache Spark

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Apache Ignite

大数据入门指南 ⭐

Haskell on Apache Spark.

Server for the ListenBrainz project

A pandas-like deferred expression system, with first-class SQL support

DataFusion has now been donated to the Apache Arrow project

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

Kafka Connect HDFS connector

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Real Time Analytics and Data Pipelines based on Spark Streaming

一个大数据架构师应该掌握的技能

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Support content for my blog

HDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS

ArcGIS GeoEvent Server sample Hadoop connector for storing GeoEvents in HDFS.

Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks

An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset

A simple Spark TDD example

A large-scale entity and relation database supporting aggregation of properties

🎯 🌟[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结

RCS Intro to Databases workshop materials

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Apache ORC - the smallest, fastest columnar storage for Hadoop workloads

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

C# and F# language binding and extensions to Apache Spark

Mirror of Apache Phoenix

Mirror of Apache Calcite - Avatica

零基础学习spark，大数据学习

Jupyter magics and kernels for working with remote Spark clusters

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

A Spark cluster setup running on Docker containers

MapReduce, Spark, Java, and Scala for Data Algorithms Book

MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)

Apache Spark on AWS Lambda

Distributed "massively parallel" SQL query engine

Carefully curated resource links for data science in one place

🌐 Interactive Workshop on GeoAnalysis using PySpark