All Projects → hellobike → Amazonriver

hellobike / Amazonriver

Licence: apache-2.0
amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务

Programming Languages

go
31211 projects - #10 most used programming language

Projects that are alternatives of or similar to Amazonriver

Tunnel
PG数据同步工具(Java实现)
Stars: ✭ 122 (-38.38%)
Mutual labels:  stream, kafka, sync, postgresql, elasticsearch, postgres, replication
Zombodb
Making Postgres and Elasticsearch work together like it's 2021
Stars: ✭ 3,781 (+1809.6%)
Mutual labels:  postgresql, elasticsearch, postgres
Wal E
Continuous Archiving for Postgres
Stars: ✭ 3,313 (+1573.23%)
Mutual labels:  postgresql, postgres, replication
Supabase
The open source Firebase alternative. Follow to stay updated about our public Beta.
Stars: ✭ 25,142 (+12597.98%)
Mutual labels:  postgresql, postgres, realtime
Synch
Sync data from the other DB to ClickHouse(cluster)
Stars: ✭ 200 (+1.01%)
Mutual labels:  kafka, postgresql, replication
Postgresql cluster
PostgreSQL High-Availability Cluster (based on "Patroni" and "DCS(etcd)"). Automating deployment with Ansible.
Stars: ✭ 294 (+48.48%)
Mutual labels:  postgresql, postgres, replication
Monstache
a go daemon that syncs MongoDB to Elasticsearch in realtime
Stars: ✭ 736 (+271.72%)
Mutual labels:  sync, elasticsearch, realtime
Pg chameleon
MySQL to PostgreSQL replica system
Stars: ✭ 274 (+38.38%)
Mutual labels:  postgresql, postgres, replication
Repmgr
A lightweight replication manager for PostgreSQL (Postgres) - latest version 5.2.1 (2020-12-07)
Stars: ✭ 1,207 (+509.6%)
Mutual labels:  postgresql, postgres, replication
Testgres
Testing framework for PostgreSQL and its extensions
Stars: ✭ 85 (-57.07%)
Mutual labels:  postgresql, postgres, replication
Firecamp
Serverless Platform for the stateful services
Stars: ✭ 194 (-2.02%)
Mutual labels:  kafka, postgresql, elasticsearch
Realtime
Listen to your to PostgreSQL database in realtime via websockets. Built with Elixir.
Stars: ✭ 4,278 (+2060.61%)
Mutual labels:  postgresql, postgres, realtime
Gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Stars: ✭ 398 (+101.01%)
Mutual labels:  kafka, elasticsearch, postgres
Pgtools
Gui application to monitor postgres database events in real time
Stars: ✭ 42 (-78.79%)
Mutual labels:  postgresql, postgres, realtime
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (-47.47%)
Mutual labels:  kafka, postgresql, elasticsearch
Samsara
Samsara is a real-time analytics platform
Stars: ✭ 132 (-33.33%)
Mutual labels:  kafka, elasticsearch, realtime
Usaspending Api
Server application to serve U.S. federal spending data via a RESTful API
Stars: ✭ 166 (-16.16%)
Mutual labels:  postgresql, elasticsearch
Bigdata practice
大数据分析可视化实践
Stars: ✭ 166 (-16.16%)
Mutual labels:  stream, kafka
Magda
A federated, open-source data catalog for all your big data and small data
Stars: ✭ 193 (-2.53%)
Mutual labels:  postgresql, elasticsearch
Tlog
Terminal I/O logger
Stars: ✭ 170 (-14.14%)
Mutual labels:  stream, elasticsearch

amazonriver CircleCI Actions Status

English doc

Go Report Card golang GoDoc GitHub release

amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务

版本支持

  • Postgresql 9.4 or later
  • Kafka 0.8 or later
  • ElasticSearch 5.x

架构图

架构图

原理

amazonriver 利用pg内部的逻辑复制功能,通过在pg创建逻辑复制槽,接收数据库的逻辑变更,通过解析test_decoding特定格式的消息,得到逻辑数据

安装使用

安装

$git clone https://github.com/hellobike/amazonriver
$cd amazonriver
$go install

使用

amazonriver -config config.json

PG 配置

PG数据库需要预先开启逻辑复制pg配置

amazonriver 配置

监控

amazonriver支持使用prometheus来监控同步数据状态,配置Grafana监控

同步到 elasticsearch

同步到elasticsearch

同步到 kafka

同步到kafka

性能测试

性能测试

许可

amazonriver 使用 Apache License 2 许可

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].