hellobike / Tunnel
Licence: apache-2.0
PG数据同步工具（Java实现）
Stars: ✭ 122
Programming Languages
java
68154 projects - #9 most used programming language
Projects that are alternatives of or similar to Tunnel
Amazonriver
amazonriver 是一个将postgresql的实时数据同步到es或kafka的服务
Stars: ✭ 198 (+62.3%)
Mutual labels: stream, kafka, sync, postgresql, elasticsearch, postgres, replication
Firecamp
Serverless Platform for the stateful services
Stars: ✭ 194 (+59.02%)
Mutual labels: kafka, postgresql, elasticsearch
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (-14.75%)
Mutual labels: kafka, postgresql, elasticsearch
Synch
Sync data from the other DB to ClickHouse(cluster)
Stars: ✭ 200 (+63.93%)
Mutual labels: kafka, postgresql, replication
Pg chameleon
MySQL to PostgreSQL replica system
Stars: ✭ 274 (+124.59%)
Mutual labels: postgresql, postgres, replication
Repmgr
A lightweight replication manager for PostgreSQL (Postgres) - latest version 5.2.1 (2020-12-07)
Stars: ✭ 1,207 (+889.34%)
Mutual labels: postgresql, postgres, replication
Postgresql cluster
PostgreSQL High-Availability Cluster (based on "Patroni" and "DCS(etcd)"). Automating deployment with Ansible.
Stars: ✭ 294 (+140.98%)
Mutual labels: postgresql, postgres, replication
Zombodb
Making Postgres and Elasticsearch work together like it's 2021
Stars: ✭ 3,781 (+2999.18%)
Mutual labels: postgresql, elasticsearch, postgres
Wal E
Continuous Archiving for Postgres
Stars: ✭ 3,313 (+2615.57%)
Mutual labels: postgresql, postgres, replication
Gnomock
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻
Stars: ✭ 398 (+226.23%)
Mutual labels: kafka, elasticsearch, postgres
Testgres
Testing framework for PostgreSQL and its extensions
Stars: ✭ 85 (-30.33%)
Mutual labels: postgresql, postgres, replication
Activerecord Clean Db Structure
Automatic cleanup for the Rails db/structure.sql file (ActiveRecord/PostgreSQL)
Stars: ✭ 101 (-17.21%)
Mutual labels: postgresql, postgres
Rsyslog
a Rocket-fast SYStem for LOG processing
Stars: ✭ 1,385 (+1035.25%)
Mutual labels: kafka, elasticsearch
Pg flame
A flamegraph generator for Postgres EXPLAIN ANALYZE output.
Stars: ✭ 1,391 (+1040.16%)
Mutual labels: postgresql, postgres
Quiz
Example real time quiz application with .NET Core, React, DDD, Event Sourcing, Docker and built-in infrastructure for CI/CD with k8s, jenkins and helm
Stars: ✭ 100 (-18.03%)
Mutual labels: postgresql, postgres
Whatsmars
Java生态研究(Spring Boot + Redis + Dubbo + RocketMQ + Elasticsearch)🔥🔥🔥🔥🔥
Stars: ✭ 1,389 (+1038.52%)
Mutual labels: kafka, elasticsearch
Illuminati
This is a Platform that collects all the data accuring in your Application and shows the data in real time by using Kibana or other tools.
Stars: ✭ 106 (-13.11%)
Mutual labels: stream, kafka
Grafana
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
Stars: ✭ 45,930 (+37547.54%)
Mutual labels: elasticsearch, postgres
Haproxy Configs
80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, Hue, ZooKeeper, SSH, RabbitMQ, Redis, Riak, Cloudera, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, Rancher etc.
Stars: ✭ 106 (-13.11%)
Mutual labels: postgresql, elasticsearch
Springboot Templates
springboot和dubbo、netty的集成，redis mongodb的nosql模板， kafka rocketmq rabbit的MQ模板， solr solrcloud elasticsearch查询引擎
Stars: ✭ 100 (-18.03%)
Mutual labels: kafka, elasticsearch
PG数据同步工具（Java实现）
Tunnel 是一个将postgresql的实时数据同步到es或kafka的服务
版本支持
- Postgresql 9.4 or later
- Kafka 0.8 or later
- ElasticSearch 5.x
架构图
原理
tunnel 利用pg内部的逻辑复制功能,通过在pg创建逻辑复制槽,接收数据库的逻辑变更,通过解析test_decoding特定格式的消息,得到逻辑数据
安装使用
打包
$ git clone https://github.com/hellobike/tunnel
$ cd tunnel
$ mvn clean package -Dmaven.test.skip=true
使用
$ cd target
$ unzip AppTunnelService.zip
$ cd AppTunnelService
$ java -server -classpath conf/*:lib/* com.hellobike.base.tunnel.TunnelLauncher -u false -c cfg.properties
PG 配置
PG数据库需要预先开启逻辑复制pg配置
Tunnel 配置
配置文件
监控
Tunnel支持使用prometheus来监控同步数据状态,配置Grafana监控
同步到 elasticsearch
同步到 kafka
许可
Tunnel 使用 Apache License 2 许可
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].