1544 Open source projects that are alternatives of or similar to Datavec

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

A simple Spark-powered ETL framework that just works 🍺

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步，主要定位是为实时交易系统服务，亦可用于大数据的数据同步（ETL领域）。

⚙️ Schedule notebooks, run them like APIs, expose securely your assets: Jupyter as a viable ⚡️ Production environment

Example project implementing best practices for PySpark ETL jobs and applications.

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

An Awesome [/osom/] Object Data Modeling (Database Agnostic).

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

M-Lab ingestion pipeline

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Udacity Data Engineering Nano Degree (DEND)

A lightweight data processing framework for Apache Spark

A lightweight stream processing library for Go

WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. Currently the source code of Scriptis and Linkis has already been released to the open-source community. WeDataSphere, Big Data Made Easy!

A graphql query language and schema definition language parser and formatter for rust

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

mito ETL tool

A luigi powered analytics / warehouse stack

A simplified, lightweight ETL Framework based on Apache Spark

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

Generate beautiful documentation for your data pipelines in markdown format

O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian

📋 A schema-based form generator component for Vue.js

A linter, formatter for finding and removing unused import statements.

Safe, Reliable, and Fast Production Deployments for Kubernetes

The uncompromising Snakemake code formatter

Immutable and structurally typed data

Code snippets for solving common big data problems in various platforms. Inspired by Rosetta Code

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

Pipeline for training LSA models using Scikit-Learn.

A Python framework for creating reproducible, maintainable and modular data science code.

spark structured streaming via HTTP communication

Django Debug Query (ddquery) beautiful colored SQL statements for logging

Dynamic schema-based form rendering for VueJS

The Hunting ELK

Spark structured streaming examples with using of version 3.0.0

A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.

Godot Engine Automation Pipeline Android – iOS – Linux – MacOS – Windows – HTML5 – Itch.io.

Profile Photo Camera support for Laravel Spark

Flexibile and extendable form builder with constructor

Shell supporting pipelines to and from multiple processes

🛠 hammer is a command-line tool to schema management for Google Cloud Spanner.

Kylo integration with PDND (previously DAF).

dllib is a distributed deep learning library running on Apache Spark

Realtime data processing and plotting pipelines in Python

UC Berkeley team's submission for RecSys Challenge 2018

Toml sorting library

🌊🐷 Utility for generalized composition of React components

Smooth scrolling, rotation, pull to refresh, page transition and any motion for the web - 丝般顺滑的触摸运动方案

A simple spark standalone cluster for your testing environment purposses

Simple currency formatter for Android EditText

A simplified pipeline for ctDNA sequencing data analysis

Blazor component for Microsoft's Monaco Editor which powers Visual Studio Code.

A python package to create a database on the platform using our moj data warehousing framework

尚硅谷大数据Spark-2019版最新 Spark 学习

Developing Spark External Data Sources using the V2 API