1283 Open source projects that are alternatives of or similar to Goodreads_etl_pipeline

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Udacity Data Engineering Nano Degree (DEND)

Example end to end data engineering project.

A Data Engineering & Machine Learning Knowledge Hub

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Blog post on ETL pipelines with Airflow

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Open-source distribute workflow schedule tools, also support streaming task.

Identifies and collects data on cc-licensed content across web crawl data and public apis.

简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .

Open Source Feature Flagging and A/B Testing Platform

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

Continuously synchronize directories from remote object store to local filesystem

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Personal Data Engineering Projects

Bitnami Docker Image for Apache Airflow Scheduler

Go Wrapper for using localstack

A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support

Interactive and Reactive Data Science using Scala and Spark.

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

Qubole Sparklens tool for performance tuning Apache Spark

酷玩 Spark: Spark 源代码解析、Spark 类库等

A simplified, lightweight ETL Framework based on Apache Spark

The Internals of Spark Structured Streaming

Metric collection, data testing and monitoring for SQL accessible data

Airflow Deployment on AWS ECS Fargate Using Cloudformation

⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Example project implementing best practices for PySpark ETL jobs and applications.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Haskell on Apache Spark.

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A production-grade data pipeline has been designed to automate the parsing of user search patterns to analyze user engagement. Extract data from S3, apply a series of transformations and load into S3 and Redshift.

Apache Spark in your IDE with gradle

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

A daily digest of the articles or videos I've found interesting, that I want to share with you.

Spark structured streaming examples with using of version 3.0.0

Docker Apache Airflow

A SQL-like language for performing ETL transformations.

Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.

Fair job scheduler on Kubernetes and Mesos for batch workloads and Spark

This is the development repository of SparkMeasure, a tool for performance troubleshooting of Apache Spark workloads. It simplifies the collection and analysis of Spark task metrics data.

Minikube for big data with Scala and Spark

Turn complex requirements to workflows without leaving the comfort of your technology stack.

Data validation and organization of metadata for data frames and database tables

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

The Internals of Spark SQL

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

Example of an ETL Pipeline using Airflow

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

R interface for Apache Spark