hellobike / Amazonriver
Licence: apache-2.0
amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务
Stars: ✭ 198
Programming Languages
go
31211 projects - #10 most used programming language
Projects that are alternatives of or similar to Amazonriver
Tunnel
PG数据同步工具（Java实现）
Stars: ✭ 122 (-38.38%)
Mutual labels: stream, kafka, sync, postgresql, elasticsearch, postgres, replication
Zombodb
Making Postgres and Elasticsearch work together like it's 2021
Stars: ✭ 3,781 (+1809.6%)
Mutual labels: postgresql, elasticsearch, postgres
Wal E
Continuous Archiving for Postgres
Stars: ✭ 3,313 (+1573.23%)
Mutual labels: postgresql, postgres, replication
Supabase
The open source Firebase alternative. Follow to stay updated about our public Beta.
Stars: ✭ 25,142 (+12597.98%)
Mutual labels: postgresql, postgres, realtime
Synch
Sync data from the other DB to ClickHouse(cluster)
Stars: ✭ 200 (+1.01%)
Mutual labels: kafka, postgresql, replication
Postgresql cluster
PostgreSQL High-Availability Cluster (based on "Patroni" and "DCS(etcd)"). Automating deployment with Ansible.
Stars: ✭ 294 (+48.48%)
Mutual labels: postgresql, postgres, replication
Monstache
a go daemon that syncs MongoDB to Elasticsearch in realtime
Stars: ✭ 736 (+271.72%)
Mutual labels: sync, elasticsearch, realtime
Pg chameleon
MySQL to PostgreSQL replica system
Stars: ✭ 274 (+38.38%)
Mutual labels: postgresql, postgres, replication
Repmgr
A lightweight replication manager for PostgreSQL (Postgres) - latest version 5.2.1 (2020-12-07)
Stars: ✭ 1,207 (+509.6%)
Mutual labels: postgresql, postgres, replication
Testgres
Testing framework for PostgreSQL and its extensions
Stars: ✭ 85 (-57.07%)
Mutual labels: postgresql, postgres, replication
Firecamp
Serverless Platform for the stateful services
Stars: ✭ 194 (-2.02%)
Mutual labels: kafka, postgresql, elasticsearch
Realtime
Listen to your to PostgreSQL database in realtime via websockets. Built with Elixir.
Stars: ✭ 4,278 (+2060.61%)
Mutual labels: postgresql, postgres, realtime
Gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Stars: ✭ 398 (+101.01%)
Mutual labels: kafka, elasticsearch, postgres
Pgtools
Gui application to monitor postgres database events in real time
Stars: ✭ 42 (-78.79%)
Mutual labels: postgresql, postgres, realtime
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (-47.47%)
Mutual labels: kafka, postgresql, elasticsearch
Samsara
Samsara is a real-time analytics platform
Stars: ✭ 132 (-33.33%)
Mutual labels: kafka, elasticsearch, realtime
Usaspending Api
Server application to serve U.S. federal spending data via a RESTful API
Stars: ✭ 166 (-16.16%)
Mutual labels: postgresql, elasticsearch
Magda
A federated, open-source data catalog for all your big data and small data
Stars: ✭ 193 (-2.53%)
Mutual labels: postgresql, elasticsearch
amazonriver 
amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务
版本支持
- Postgresql 9.4 or later
- Kafka 0.8 or later
- ElasticSearch 5.x
架构图
原理
amazonriver 利用pg内部的逻辑复制功能,通过在pg创建逻辑复制槽,接收数据库的逻辑变更,通过解析test_decoding特定格式的消息,得到逻辑数据
安装使用
安装
$git clone https://github.com/hellobike/amazonriver
$cd amazonriver
$go install
使用
amazonriver -config config.json
PG 配置
PG数据库需要预先开启逻辑复制pg配置
amazonriver 配置
监控
amazonriver支持使用prometheus来监控同步数据状态,配置Grafana监控
同步到 elasticsearch
同步到 kafka
性能测试
许可
amazonriver 使用 Apache License 2 许可
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].