203 Open source projects that are alternatives of or similar to soda-spark

re_data - fix data issues before your users & CEO would discover them 😊

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Easy way to define, execute and store quality rules for your data.

Always know what to expect from your data.

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

A tool for building feature stores.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Example project implementing best practices for PySpark ETL jobs and applications.

Data validation library for PySpark 3.0.0

This repo contains commands that data engineers use in day to day work.

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

pyspark🍒🥭 is delicious，just eat it!😋😋

MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …

Relation Extraction using Deep learning(CNN)

Big Data Processing Framework - Unified Data API or SQL on Any Storage

PySpark Code for Hands-on Learners

Spark 2.0 Python Machine Learning examples

Apache Spark (PySpark) Practice on Real Data

🌐 Interactive Workshop on GeoAnalysis using PySpark

A curated list of awesome Apache Spark packages and resources.

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile y…

🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Isolation Forest on Spark

Jupyter magics and kernels for working with remote Spark clusters

Training models with Apache Spark, PySpark for Titanic Kaggle competition

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

A simple Spark TDD example

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

🐍 Quick reference guide to common patterns & functions in PySpark.

pyspark-cassandra is a Python port of the awesome @datastax Spark Cassandra connector. Compatible w/ Spark 2.0, 2.1, 2.2, 2.3 and 2.4

Apache (Py)Spark type annotations (stub files).

pyspark methods to enhance developer productivity 📣 👯 🎉

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.

[NOT MAINTAINED] Predicting Bit coin price using Time series analysis and sentiment analysis of tweets on bitcoin

Simple and Distributed Machine Learning

Word2Vec models with Twitter data using Spark. Blog:

Data quality estimations for OpenStreetMap

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

State of the Art Natural Language Processing

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

CSVs sliced, diced & analyzed.

Spark Application for analysis of Apache Access logs and detect anamolies! Along with Medium Article.

Apache Spark Connector for Azure Cosmos DB

Demo of PySpark and Jupyter Notebook with the Jupyter Docker Stacks

hive_compared_bq compares/validates 2 (SQL like) tables, and graphically shows the rows/columns that are different.

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

HandySpark - bringing pandas-like capabilities to Spark dataframes

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Blog post on ETL pipelines with Airflow

Código para workshops Spark com ambiente de desenvolvimento em docker

LearningApacheSpark

This is a repo documenting the best practices in PySpark.

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Fundamentals of Spark with Python (using PySpark), code examples

A boilerplate for writing PySpark Jobs

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

A curated list of awesome dbt resources

Process Common Crawl data with Python and Spark