308 Open source projects that are alternatives of or similar to pangeo-forge-recipes

Example of an ETL Pipeline using Airflow

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

An Elasticsearch client tailored to data science workflows.

A tool for building feature stores.

A client for connecting and running DDLs on hive metastore.

Enterprise-grade, production-hardened, serverless data lake on AWS

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

A python package to create a database on the platform using our moj data warehousing framework

A simple Spark-powered ETL framework that just works 🍺

Distributed Xarray with Apache Beam

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

Blog post on ETL pipelines with Airflow

Fancy stream processing made operationally mundane

A schema-aware Scala library for data transformation

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

Example project implementing best practices for PySpark ETL jobs and applications.

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

ELT Code for your Data Warehouse

Beneath is a serverless real-time data platform ⚡️

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Notes and practicals for my "Physics of the Climate System" lecture

Extension to hypothesis for testing numpy general universal functions

consume maxwell generated message from kafka,export it to another mysql.

Personal Data Engineering Projects

Python ETL(Extract-Transform-Load) tool / Data migration tool

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

Multi-core processing of 'Gameday' data from Major League Baseball Advanced Media. Additional tools to parallelize large data sets and write them to a database.

Real-Time Event Streaming & Change Data Capture

Python toolkit for GEOS-Chem.

Publish Xarray Datasets via a REST API.

Ruby scripting for pentaho-kettle

OpenRefine is a free, open source power tool for working with messy data and improving it. This repository contains Dockerbuild files for automated builds.

ETL for the SPDR ETF holdings XLS documents

Python package to call processed EE objects via the REST API to local data

A persistence framework for game developers

Data transformation framework for LinkML data models

an open-source data management platform for knowledge workers (https://github.com/dswarm/dswarm-documentation/wiki)

网站点击流离线日志分析

The OpenRefine Python Client from Paul Makepeace provides a library for communicating with an OpenRefine server. This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, Mac). It is also available via Docker Hub, PyPI and Binder.

ElasticSearch to PostgreSQL loader

ETL for OIC Options Chains

Climate Simulation Operations

Enterprise Grade Single-Step Streaming Data Infrastructure Setup. (Under Development)

In-memory Java DataFrame library

Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.

A Python interface between Earth Engine and xarray for processing time series data

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步，主要定位是为实时交易系统服务，亦可用于大数据的数据同步（ETL领域）。

Python package for automated analysis and management of gridded climate data

Generate beautiful documentation for your data pipelines in markdown format

Code relating to scraping public police data.

Earth System Model Lab (esmlab). ⚠️⚠️ ESMLab functionality has been moved into <https://github.com/NCAR/geocat-comp>. ⚠️⚠️

Fast & furious GroupBy operations for dask.array

This repo contains commands that data engineers use in day to day work.

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.