1517 Open source projects that are alternatives of or similar to Pyspark Example Project

A tool for building feature stores.

A simple Spark-powered ETL framework that just works 🍺

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

🐍 Quick reference guide to common patterns & functions in PySpark.

Word2Vec models with Twitter data using Spark. Blog:

A Clojure dataframe library that runs on Spark

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Categorical Query Language IDE

A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.

Applying Data Science and Machine Learning to Solve Real World Business Problems

A client for connecting and running DDLs on hive metastore.

A convention over configuration workflow orchestrator. Develop locally (Jupyter or your favorite editor), deploy to Airflow or Kubernetes.

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

ETL jobs for Firefox Telemetry

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A schema-aware Scala library for data transformation

Generate beautiful documentation for your data pipelines in markdown format

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

My presentation at ODSC India 2018 about Deep Learning with Apache Spark

Python library for building Pangeo Forge recipes.

Metric collection, data testing and monitoring for SQL accessible data

R client for the Elasticsearch HTTP API

An automatic ML model optimization tool.

Blog post on ETL pipelines with Airflow

Koalas: pandas API on Apache Spark

Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes

Content for architecting a data science platform for products using Luigi, Spark & Flask.

An Elasticsearch client tailored to data science workflows.

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

This repo contains commands that data engineers use in day to day work.

A lightweight data processing framework for Apache Spark

Some class materials for a data processing course using PySpark

A library that provides useful extensions to Apache Spark and PySpark.

Beneath is a serverless real-time data platform ⚡️

ELT Code for your Data Warehouse

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

Example of an ETL Pipeline using Airflow

A Data Engineering & Machine Learning Knowledge Hub

Distributed scikit-learn meta-estimators in PySpark

ETL Library for Machine Learning - data pipelines, data munging and wrangling

Interactive and Reactive Data Science using Scala and Spark.

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

A python package to create a database on the platform using our moj data warehousing framework

Fancy stream processing made operationally mundane

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

A simplified, lightweight ETL Framework based on Apache Spark

📝 A compilation of everything that I learn; Computer Science, Software Development, Engineering, Math, and Coding in General. Read the rendered results here ->

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift