935 Open source projects that are alternatives of or similar to bigquery-kafka-connect

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

A simplified, lightweight ETL Framework based on Apache Spark

Esper instance for TV news analysis

Example end to end data engineering project.

Campaign Manager 360 and Display & Video 360 Reports to BigQuery connector

A Kafka Connect source connector that generates data for tests

equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨

Export a whole BigQuery table to Google Datastore with Apache Beam/Google Dataflow

The simplest tool to manage views of BigQuery.

An upgraded and improved version of the Iris automatic GCP-labeling project

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

TypeScript Bigtable Client with 🔋🔋 included.

kafka-connect-s3 : Ingest data from Kafka to Object Stores(s3)

A visual ETL development and debugging tool for big data

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Real-Time Event Streaming & Change Data Capture

An extensible Java framework for building XML and non-XML streaming applications

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

Open-Source Web GUI for Apache Kafka Management

Tokenize Japanese text on BigQuery with Kuromoji in Apache Beam/Google Dataflow at scale

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

An example mini data warehouse for python project stats, template for new projects

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

A Scala API for Apache Beam and Google Cloud Dataflow.

A collection of Magnolia add-on modules

consume maxwell generated message from kafka,export it to another mysql.

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.

Big Data Toolkit for the JVM

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

A simple Spark-powered ETL framework that just works 🍺

This service is meant to simplify running Google Cloud operations, especially BigQuery tasks. This means you do not have to worry about installation, configuration or ongoing maintenance related to an SDK environment. This can be helpful to those who would prefer to not to be responsible for those activities.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)

FaaS (Function as a service) framework for writing portable PHP functions

Kafka Connect Connector for Jenkins Open Source Continuous Integration Tool

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer

Ruby on Rails with Active Model and Google Cloud Datastore. Extracted from Agrimatics Aero.

InsightEdge Core

Nim wrapper for libssh2

Data ETL & Analysis on the global and Mexican datasets of the COVID-19 pandemic.

Google Cloud BigTable and PubSub emulator tools to make development a breeze

Crate for authenticating Server to Server Apps for Google Cloud Engine.

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

PowerShell cmdlets for the Google Cloud Platform

Apache Arrow DataFusion SQL Query Engine

Big Data Visualization

Google Vision Api for PHP (https://cloud.google.com/vision/)

An open-source, real-time Security Information & Event Management tool based on big data technologies, providing a scalable, advanced security analytics framework.

🔄 Simple tool for ETL. From SQL to Redis.

rastercube is a python library for big data analysis of georeferenced time series data (e.g. MODIS NDVI)

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Comparing the free tier offers of the major cloud providers like AWS, Azure, GCP, Oracle etc.

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

Mirror of Apache Airavata PHP Gateway

A collection of Google Cloud Platform (GCP) plugins