Metlmito ETL tool
Stars: ✭ 153 (+856.25%)
SetlA simple Spark-powered ETL framework that just works 🍺
Stars: ✭ 79 (+393.75%)
polygon-etlETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub
Stars: ✭ 53 (+231.25%)
DataX-srcDataX 是异构数据广泛使用的离线数据同步工具/平台,实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。
Stars: ✭ 21 (+31.25%)
mydataharbor🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步,主要定位是为实时交易系统服务,亦可用于大数据的数据同步(ETL领域)。
Stars: ✭ 28 (+75%)
AirflowETLBlog post on ETL pipelines with Airflow
Stars: ✭ 20 (+25%)
StetlStetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.
Stars: ✭ 64 (+300%)
Aws Ecs AirflowRun Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
Stars: ✭ 107 (+568.75%)
Linq2dbLinq to database provider.
Stars: ✭ 2,211 (+13718.75%)
Pyetlpython ETL framework
Stars: ✭ 33 (+106.25%)
AddaxAddax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.
Stars: ✭ 615 (+3743.75%)
sparklanesA lightweight data processing framework for Apache Spark
Stars: ✭ 17 (+6.25%)
DiscreetlyETLy is an add-on dashboard service on top of Apache Airflow.
Stars: ✭ 60 (+275%)
Bulk WriterProvides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
Stars: ✭ 210 (+1212.5%)
AirbyteAirbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.
Stars: ✭ 4,919 (+30643.75%)
Csv2dbThe CSV to database command line loader
Stars: ✭ 102 (+537.5%)
etlM-Lab ingestion pipeline
Stars: ✭ 15 (-6.25%)
Mara PipelinesA lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Stars: ✭ 1,841 (+11406.25%)
Go StreamsA lightweight stream processing library for Go
Stars: ✭ 615 (+3743.75%)
DataspherestudioDataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
Stars: ✭ 1,195 (+7368.75%)
DataxDataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server
Stars: ✭ 116 (+625%)
Example Airflow DagsExample DAGs using hooks and operators from Airflow Plugins
Stars: ✭ 243 (+1418.75%)
lineageGenerate beautiful documentation for your data pipelines in markdown format
Stars: ✭ 16 (+0%)
astroAstro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Stars: ✭ 79 (+393.75%)
basinBasin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser
Stars: ✭ 25 (+56.25%)
naas⚙️ Schedule notebooks, run them like APIs, expose securely your assets: Jupyter as a viable ⚡️ Production environment
Stars: ✭ 219 (+1268.75%)
DatavecETL Library for Machine Learning - data pipelines, data munging and wrangling
Stars: ✭ 272 (+1600%)
Ttyplota realtime plotting utility for terminal/console with data input from stdin
Stars: ✭ 532 (+3225%)
Lambdacda library to define a continuous delivery pipeline in code
Stars: ✭ 655 (+3993.75%)
OkElegant error/exception handling in Elixir, with result monads.
Stars: ✭ 517 (+3131.25%)
PipelineA cloud-native Pipeline resource.
Stars: ✭ 6,751 (+42093.75%)
Vagrant ProjectsVagrant projects for Oracle products and other examples
Stars: ✭ 642 (+3912.5%)
Koop🔮 Transform, query, and download geospatial data on the web.
Stars: ✭ 505 (+3056.25%)
Laravel Oci8Oracle DB driver for Laravel 4|5|6|7|8 via OCI8
Stars: ✭ 639 (+3893.75%)
Docker ImagesOfficial source for Docker configurations, images, and examples of Dockerfiles for Oracle products and projects
Stars: ✭ 5,120 (+31900%)
OpenrecordMake ORMs great again!
Stars: ✭ 474 (+2862.5%)
Goodreads etl pipelineAn end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Stars: ✭ 793 (+4856.25%)
ToilA scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
Stars: ✭ 733 (+4481.25%)
Pyspark Example ProjectExample project implementing best practices for PySpark ETL jobs and applications.
Stars: ✭ 633 (+3856.25%)
BigsliceA serverless cluster computing system for the Go programming language
Stars: ✭ 469 (+2831.25%)
GaiaBuild powerful pipelines in any programming language.
Stars: ✭ 4,534 (+28237.5%)
Bk Sops蓝鲸智云标准运维(SOPS)
Stars: ✭ 632 (+3850%)
JooqjOOQ is the best way to write SQL in Java
Stars: ✭ 4,695 (+29243.75%)
SmartcodeSmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!
Stars: ✭ 464 (+2800%)
React CsvReact components to build CSV files on the fly basing on Array/literal object of data
Stars: ✭ 732 (+4475%)
Argo CdDeclarative continuous deployment for Kubernetes.
Stars: ✭ 7,887 (+49193.75%)
EtlalchemyExtract, Transform, Load: Any SQL Database in 4 lines of Code.
Stars: ✭ 460 (+2775%)
SqlinjectionwikiA wiki focusing on aggregating and documenting various SQL injection methods
Stars: ✭ 623 (+3793.75%)
Udacity Data Engineering ProjectsFew projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Stars: ✭ 458 (+2762.5%)
PglogicalLogical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
Stars: ✭ 455 (+2743.75%)
GalaxyData intensive science for everyone.
Stars: ✭ 812 (+4975%)
SmartsqlSmartSql = MyBatis in C# + .NET Core+ Cache(Memory | Redis) + R/W Splitting + PropertyChangedTrack +Dynamic Repository + InvokeSync + Diagnostics
Stars: ✭ 775 (+4743.75%)
Syntax sugar pythonA library adding some anti-Pythonic syntatic sugar to Python
Stars: ✭ 721 (+4406.25%)
DbshieldDatabase firewall written in Go
Stars: ✭ 620 (+3775%)
DashboardA dashboard for Tekton!
Stars: ✭ 448 (+2700%)