293 Open source projects that are alternatives of or similar to Discreetly

Blog post on ETL pipelines with Airflow

Example DAGs using hooks and operators from Airflow Plugins

Udacity Data Engineering Nano Degree (DEND)

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Example of an ETL Pipeline using Airflow

🔮 Transform, query, and download geospatial data on the web.

The backoffice web application of d:swarm (https://github.com/dswarm/dswarm-documentation/wiki)

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Data ETL & Analysis on the dataset 'Baby Names from Social Security Card Applications - National Data'.

AWS Auto Terminate Idle AWS EMR Clusters Framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Elyra extends JupyterLab Notebooks with an AI centric approach.

The premier open source Data Quality solution

Power of appbase.io via CLI, with nifty imports from your favorite data sources

Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3

python ETL framework

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

AIStore: scalable storage for AI applications

A simplified, lightweight ETL Framework based on Apache Spark

Turbine: the bare metals that gets you Airflow

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

Extract your data from the Yunmai weighing scales cloud API so you can use it elsewhere

A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.

A python library to push ethereum blockchain data into an sql database.

A serverless cluster computing system for the Go programming language

Detect threats with log data and improve cloud security posture

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Workflow engine for Kubernetes

Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

🐟 A streaming ETL for fish

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

Dynamically generate Apache Airflow DAGs from YAML configuration files

A facebook for data

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

A distributed workflow runner focusing on performance and simplicity.

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

Automating Your Data Pipeline with Apache Airflow

Continuously synchronize directories from remote object store to local filesystem

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具

Airflow basics tutorial

Docker Apache Airflow

Kiba enhancement for Ruby ETL.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

This repository is a getting started guide to Singer.

An extensible Java framework for building XML and non-XML streaming applications

An orchestration platform for the development, production, and observation of data assets.

a go daemon that syncs MongoDB to Elasticsearch in realtime

A plugin for Apache Airflow that exposes rest end points for the Command Line Interfaces

Fancy stream processing made operationally mundane

Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/

React components to build CSV files on the fly basing on Array/literal object of data