299 Open source projects that are alternatives of or similar to flock

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

Dinky is an out of the box one-stop real-time computing platform dedicated to the construction and practice of Unified Streaming & Batch and Unified Data Lake & Data Warehouse. Based on Apache Flink, Dinky provides the ability to connect many big data frameworks including OLAP and Data Lake.

📜 A CLI toolkit for extracting and working with your digital history

Chaos Injection library for AWS Lambda

Python based Wikidata framework for easy dataframe extraction

Live import all your Google Sheets to your data warehouse

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics framework that can be used to monitor, log, audit and control data integration / ETL processes.

大数据采集,抽取平台

ETL flow framework based on Yaml configs in Python

Learn how to how to use the lambda, map, filter and reduce functions in Python to transform data structures.

Thain is a distributed flow schedule platform.

AWS S3 Video Thumbnailer with Lambda

Another OLAP database

DataX 是异构数据广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

proc(ess)-that - easy extendable ETL tool for Node.js. Written in TypeScript.

An Open Source Serverless Platform

Yet Another ETL in PHP

A client for connecting and running DDLs on hive metastore.

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

A collection of tools for extracting FHIR resources and analytics services on top of that data.

AWS account provisioning and management service

Monitor the internet attack surface of various public cloud environments. Currently supports AWS, GCP, Azure, DigitalOcean and Oracle Cloud.

Data migration framework for Django that migrates legacy data into your new django app

ETL for the IEX Stocks API

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

Deploy your React app to production on Netlify, Vercel and Heroku

MySQL / MariaDB import for DTD feeds (fares, timetable and routeing)

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

⚙️ Schedule notebooks, run them like APIs, expose securely your assets: Jupyter as a viable ⚡️ Production environment

Example use of AWS batch

Java sitemap generator. This library generates a web sitemap, can ping Google, generate RSS feed, robots.txt and more with friendly, easy to use Java 8 functional style of programming

🗺 GIS plugins for Pentaho Data Integration

Go HTTP router library for AWS API Gateway-invoked Lambda Functions

A dbt adapter for Databricks.

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

A minimalist (geospatial) ETL

BETL. Meta data driven ETL generation using T-SQL

NBi is a testing framework (add-on to NUnit) for Business Intelligence and Data Access. The main goal of this framework is to let users create tests with a declarative approach based on an Xml syntax. By the means of NBi, you don't need to develop C# or Java code to specify your tests! Either, you don't need Visual Studio or Eclipse to compile y…

Synchronization Algorithm Exploration: Techniques to synchronize a SQL database with external destinations.

Blog post on ETL pipelines with Airflow

数据采集平台zdh,etl 处理服务

Data logistics system enabling real-time pathogen surveillance. Built for the Seattle Flu Study.

Treats CSV and JSON files as SQL tables, and exports SQL SELECTs back to CSV or JSON.

TT Tech Research Notes

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

www.vixtract.ru

Unleash the power of cloud

📘 Making Data, the DataMade Way

The metrics layer for your data. Join us at https://metriql.com/slack

Example DAGs using hooks and operators from Airflow Plugins

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

A curated list of awesome system integration software and resources.

A Terraform module for deploying and managing Lambda functions on Amazon Web Services (AWS). https://aws.amazon.com/lambda/

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Jamstack app using Gatsby, Netlify, and FaunaDB.