204 Open source projects that are alternatives of or similar to Dswarm Backoffice Web

The premier open source Data Quality solution

网站点击流离线日志分析

Move your messages from MQTT to Apache Kafka in real-time 🚀

In-memory Java DataFrame library

A lightweight stream processing library for Go

🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步，主要定位是为实时交易系统服务，亦可用于大数据的数据同步（ETL领域）。

Example of an ETL Pipeline using Airflow

Configurable data bridge for permanent ETL jobs

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark

C++ implementation of CAN ISO 15765-2 also known as CAN ISO transport protocol. CPP CAN isotp.

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

This repository is a getting started guide to Singer.

The Move to Islandora Kit is an extensible PHP command-line tool for converting source content and metadata into packages suitable for importing into Islandora (or other digital repository and preservations systems).

ELT Code for your Data Warehouse

☁️ nodejs kafka connect connector for Google BigQuery

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

A CLI and library to run Singer Taps and Targets

Virtual Machine for Desktop Search with Open Semantic Search

🔄 Simple tool for ETL. From SQL to Redis.

A hackable data integration & analysis tool to enable non technical users to edit data processing jobs and visualise data on demand.

数据质量控制系统

GAMECHANGER aspires to be the Department’s trusted solution for evidence-based, data-driven decision-making across the universe of DoD requirements

Data ETL & Analysis on the global and Mexican datasets of the COVID-19 pandemic.

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Python3 lib and cli for operating a Cardano Passive Node and using the API's. (PRE-ALPHA)

CVparser is software for parsing or extracting data out of CV/resumes.

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

A Django app to download, extract and load campaign finance and lobbying activity data from the California Secretary of State's CAL-ACCESS database

Multi-core processing of 'Gameday' data from Major League Baseball Advanced Media. Additional tools to parallelize large data sets and write them to a database.

Synchronization Algorithm Exploration: Techniques to synchronize a SQL database with external destinations.

An extensible Java framework for building XML and non-XML streaming applications

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

The OpenRefine Python Client from Paul Makepeace provides a library for communicating with an OpenRefine server. This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, Mac). It is also available via Docker Hub, PyPI and Binder.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

A serverless cluster computing system for the Go programming language

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Python ETL(Extract-Transform-Load) tool / Data migration tool

Treats CSV and JSON files as SQL tables, and exports SQL SELECTs back to CSV or JSON.

Fancy stream processing made operationally mundane

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

ElasticSearch to PostgreSQL loader

Live import all your Google Sheets to your data warehouse

React components to build CSV files on the fly basing on Array/literal object of data

BETL. Meta data driven ETL generation using T-SQL

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

A python package to create a database on the platform using our moj data warehousing framework

ETL for the IEX Stocks API

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer） 分布式运行（DataX on YARN） 功能

JDBC driver for Neo4j

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

A Kafka Connect source connector that generates data for tests

🐟 A streaming ETL for fish

Example project implementing best practices for PySpark ETL jobs and applications.

Logical Replication extension for PostgreSQL 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.

Real-Time Event Streaming & Change Data Capture