1099 Open source projects that are alternatives of or similar to krawler

Detect threats with log data and improve cloud security posture

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

Renderers and renderer extensions for JupyterLab

PostgreSQL Logical Replication tool into Kinesis, S3 and RabbitMQ

Python library for building Pangeo Forge recipes.

Boolean operations on polygons (union, intersection, difference, xor)

Configurable Extract, Transform, and Load

Maps SDK for Android Utility Library

Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.

Arquivos Geojson com perímetros dos municípios brasileiros por estado ( Brasil / Brazil )

Terraform module that creates an S3 bucket with an optional IAM user for external CI/CD systems

A set of spiders and scrapers to extract location information from places that post their location on the internet.

OpenRefine is a free, open source power tool for working with messy data and improving it. This repository contains Dockerbuild files for automated builds.

A turf.js-like geospatial analysis library working with GeoJSON, written in pure Dart.

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

M-Lab ingestion pipeline

ETL flow framework based on Yaml configs in Python

Calculates extent/bbox for a given valid geojson object.

⚡️ An efficient cache for the execution of dask graphs.

ETL for the SPDR ETF holdings XLS documents

No description or website provided.

A MongoDB to Elasticsearch connector

The backoffice web application of d:swarm (https://github.com/dswarm/dswarm-documentation/wiki)

ETL for the IEX Stocks API

CountriesNow is an Open source API for retrieving geo-information for countries, including their states, cities, population, etc. 🌎

The Taxonomy for ETL Automation Metadata (TEAM) is a metadata management tool for data warehouse automation. It is part of the ecosystem for data warehouse automation, alongside the Virtual Data Warehouse pattern manager and the generic schema for Data Warehouse Automation.

Query any data source using SQL, works with the local filesystem, s3, and more. It should be a very tiny and lightweight alternative to AWS Athena, Presto ... etc.

The Tombolo Digital Connector enables users to combine different sources of data in a transparent and reproducible way.

Data transformation framework for LinkML data models

Command line tool to convert GeoJSON to SVG.

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

Electron App for Loading GeoJSON files with Mapbox

♨️ Using dddplus-archetype build a WMS in 5 minutes. 5分钟搭建一个仓储中台WMS！

A curated list of awesome system integration software and resources.

🐟 A streaming ETL for fish

JDBC driver for Neo4j

Create GeoJSON from Entity Framework Spatial Data or WKT

Generate beautiful documentation for your data pipelines in markdown format

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

A lightweight data processing framework for Apache Spark

Show GeoJSON Layer like as Interactive Menu List

Concat multiple files in s3

Rivery CLI

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Read more at www.jumpmind.com/products/metl/overview

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

MySQL / MariaDB import for DTD feeds (fares, timetable and routeing)

📝 This repository stores the swagger specifications of the openrouteservice API. Browse to swagger for a detailed overview.

The CSV to database command line loader

GpsPrune is a map-based application for viewing, editing and converting coordinate data from GPS systems.

Real-Time Event Streaming & Change Data Capture

Open Geospatial Datasets for GIS Education: This is a repository of open geospatial datasets to be used in an educational context. I created these files over years of teaching Geographic Data Science and GIS. All original datasets are freely available online with open data licenses (see the dataset attribution for details). All the datasets in t…

Thain is a distributed flow schedule platform.

Python based Wikidata framework for easy dataframe extraction

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

DataX 是异构数据广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

Blog post on ETL pipelines with Airflow

Cloud Optimized GeoJSON spec

Enterprise-grade, production-hardened, serverless data lake on AWS