1517 Open source projects that are alternatives of or similar to Pyspark Example Project

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Spark 2.0 Python Machine Learning examples

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Fundamentals of Spark with Python (using PySpark), code examples

Process Common Crawl data with Python and Spark

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

Updated repository

Apache Spark (PySpark) Practice on Real Data

State of the Art Natural Language Processing

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Enterprise-grade, production-hardened, serverless data lake on AWS

Python Helper library for Jupyter Notebooks

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Example of an ETL Pipeline using Airflow

A python package to create a database on the platform using our moj data warehousing framework

ETL with Python - Taught at DWH course 2017 (TAU)

Fancy stream processing made operationally mundane

Distributed scikit-learn meta-estimators in PySpark

Interactive and Reactive Data Science using Scala and Spark.

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

📊 📋 Dashboards using YAML or JSON files

Data science and Big Data with Python

The Accelerator is a tool for fast and reproducible processing of large amounts of data.

PipelineX: Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

A simplified, lightweight ETL Framework based on Apache Spark

The premier open source Data Quality solution

R client for the Elasticsearch HTTP API

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

Koalas: pandas API on Apache Spark

Categorical Query Language IDE

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Metric collection, data testing and monitoring for SQL accessible data

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A lightweight data processing framework for Apache Spark

ETL jobs for Firefox Telemetry

Some class materials for a data processing course using PySpark

My presentation at ODSC India 2018 about Deep Learning with Apache Spark

🎼 Docker compose files for various kafka stacks

A library that provides useful extensions to Apache Spark and PySpark.

Beneath is a serverless real-time data platform ⚡️

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

ELT Code for your Data Warehouse

ETL Library for Machine Learning - data pipelines, data munging and wrangling

A Data Engineering & Machine Learning Knowledge Hub

An orchestration platform for the development, production, and observation of data assets.

An automatic ML model optimization tool.

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

📝 A compilation of everything that I learn; Computer Science, Software Development, Engineering, Math, and Coding in General. Read the rendered results here ->

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Always know what to expect from your data.

A set of useful tools for competitive data science.

[HELP REQUESTED] Generalized Additive Models in Python

An open source framework for building data analytic applications.