2021 Open source projects that are alternatives of or similar to astro

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

locopy: Loading/Unloading to Redshift and Snowflake using Python.

tbls is a CI-Friendly tool for document a database, written in Go.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

Web-based SQL editor run in your own private cloud. Supports MySQL, Postgres, SQL Server, Vertica, Crate, ClickHouse, Trino, Presto, SAP HANA, Cassandra, Snowflake, BigQuery, SQLite, and more with ODBC

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Udacity Data Engineering Nano Degree (DEND)

🐳📊🤓Cookiecutter template to launch an awesome dockerized Data Science toolstack (incl. Jupyster, Superset, Postgres, Minio, AirFlow & API Star)

Fluent migrations framework for .NET

A set of UDFs and Procedures to extend BigQuery, Snowflake, Redshift and Postgres with Spatial Analytics capabilities

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

Archival and Restoration for Postgres

Database migrations. CLI and Golang library.

TypeScript clients for databases that prevent SQL Injection

Open Source Feature Flagging and A/B Testing Platform

AlphaSQL provides Integrated Type and Schema Check and Parallelization for SQL file set mainly for BigQuery

Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

🎣 List of `pre-commit` hooks to ensure the quality of your `dbt` projects.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Pandas Google BigQuery

A minimalist (geospatial) ETL

A Python framework for data processing on GCP.

⬡ Shyft is a server-side framework for building powerful GraphQL APIs 🚀

Data ETL & Analysis on the global and Mexican datasets of the COVID-19 pandemic.

A database migration tool. Supports SQL migrations and Go functions.

An SQL wrapper for vim

Go library providing common interface for working across multiple cloud storage backends

UGin is an API boilerplate written in Go (Golang) with Gin Framework.

Listen to your to PostgreSQL database in realtime via websockets. Built with Elixir.

Async database support for Python. 🗄

Use Airflow to move data from multiple MySQL databases to BigQuery

A Common DSL for Migrating Databases

Tellery lets you build metrics using SQL and bring them to your team. As easy as using a document. As powerful as a data modeling tool.

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)

☁️ nodejs kafka connect connector for Google BigQuery

bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.

An example mini data warehouse for python project stats, template for new projects

Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake

Blog post on ETL pipelines with Airflow

Database CRUD Application Built on Electron | MySQL, Postgres, SQLite

This repository contains a docker image configured to use the SSH File Transfer Protocol (SFTP) to transfer all its files to Cloud Blob Storage Services. This image can be deployed on a Kubernetes cluster with Helm.

A collection of Google Cloud Platform (GCP) plugins

Example Spark applications that run on Kubernetes and access GCP products, e.g., GCS, BigQuery, and Cloud PubSub

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A vendor-neutral storage library for Golang: Write once, run on every storage service.

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

Python based Wikidata framework for easy dataframe extraction

Entity-Relationship diagram code generator library

Go io/fs.FS filesystem implementations for various URL schemes

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

A SQL port of python's scikit-learn preprocessing module, provided as cross-database dbt macros.

Rivery CLI

A Singer.io Target for Postgres

Ebean ORM

Public reusable components for Polyaxon

Connection-Pooling Compile-Time ORM for Nim