iftech-engineering / Mongo Es
Licence: mpl-2.0
A MongoDB to Elasticsearch connector
Stars: ✭ 185
Programming Languages
typescript
32286 projects
Projects that are alternatives of or similar to Mongo Es
Monstache
a go daemon that syncs MongoDB to Elasticsearch in realtime
Stars: ✭ 736 (+297.84%)
Mutual labels: etl, mongodb, elasticsearch, connector
Mongolastic
🚥 A dataset migration tool from MongoDB to Elasticsearch and vice versa.
Stars: ✭ 131 (-29.19%)
Mutual labels: mongodb, elasticsearch, connector
Stream Reactor
Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.
Stars: ✭ 753 (+307.03%)
Mutual labels: mongodb, elasticsearch, connector
Transporter
Sync data between persistence engines, like ETL only not stodgy
Stars: ✭ 1,175 (+535.14%)
Mutual labels: etl, mongodb, elasticsearch
Spring Boot 2.x Examples
Spring Boot 2.x code examples
Stars: ✭ 104 (-43.78%)
Mutual labels: mongodb, elasticsearch
Rsyslog
a Rocket-fast SYStem for LOG processing
Stars: ✭ 1,385 (+648.65%)
Mutual labels: mongodb, elasticsearch
Nodejs Ecommerce Store
An ecommerce store built in NodeJS
Stars: ✭ 114 (-38.38%)
Mutual labels: mongodb, elasticsearch
Transformalize
Configurable Extract, Transform, and Load
Stars: ✭ 125 (-32.43%)
Mutual labels: etl, elasticsearch
Vagrant Elastic Stack
Giving the Elastic Stack a try in Vagrant
Stars: ✭ 131 (-29.19%)
Mutual labels: mongodb, elasticsearch
Springboot Templates
springboot和dubbo、netty的集成,redis mongodb的nosql模板, kafka rocketmq rabbit的MQ模板, solr solrcloud elasticsearch查询引擎
Stars: ✭ 100 (-45.95%)
Mutual labels: mongodb, elasticsearch
Adminer Custom
Customizations for Adminer, the best database management tool written in PHP.
Stars: ✭ 99 (-46.49%)
Mutual labels: mongodb, elasticsearch
Mall
mall项目是一套电商系统,包括前台商城系统及后台管理系统,基于SpringBoot+MyBatis实现,采用Docker容器化部署。 前台商城系统包含首页门户、商品推荐、商品搜索、商品展示、购物车、订单流程、会员中心、客户服务、帮助中心等模块。 后台管理系统包含商品管理、订单管理、会员管理、促销管理、运营管理、内容管理、统计报表、财务管理、权限管理、设置等模块。
Stars: ✭ 54,797 (+29520%)
Mutual labels: mongodb, elasticsearch
Dataengineeringproject
Example end to end data engineering project.
Stars: ✭ 82 (-55.68%)
Mutual labels: mongodb, elasticsearch
Aus Search
A collection of Node JS scripts to create an Elasticsearch index of Australian addresses.
Stars: ✭ 76 (-58.92%)
Mutual labels: mongodb, elasticsearch
Pifpaf
Python fixtures and daemon managing tools for functional testing
Stars: ✭ 161 (-12.97%)
Mutual labels: mongodb, elasticsearch
Spring Boot Leaning
Spring Boot 2.X 最全课程代码
Stars: ✭ 2,008 (+985.41%)
Mutual labels: mongodb, elasticsearch
Graylog Plugin Metrics Reporter
Graylog Metrics Reporter Plugins
Stars: ✭ 71 (-61.62%)
Mutual labels: mongodb, elasticsearch
Usaspending Api
Server application to serve U.S. federal spending data via a RESTful API
Stars: ✭ 166 (-10.27%)
Mutual labels: etl, elasticsearch
Mongo-ES
A MongoDB to Elasticsearch connector
Installation
npm i -g mongo-es
Usage
Command Line
# normal mode
mongo-es ./config.json
# debug mode, with debug info printed
NODE_ENV=dev mongo-es ./config.json
Programmatically
const fs = require('fs')
const Redis = require('ioredis')
const { Config, Task, run } = require('mongo-es')
const redis = new Redis('localhost')
Task.onSaveCheckpoint((name, checkpoint) => {
return redis.set(`mongo-es:${name}`, JSON.stringify(checkpoint))
})
// this will overwrite task.from in config file
Task.onLoadCheckpoint((name) => {
return redis.get(`mongo-es:${name}`).then(JSON.parse)
})
run(new Config(fs.readFileSync('config.json', 'utf8')))
Concepts
Scan phase
scan entire database for existed documents
Tail phase
tail the oplog for documents' create, update or delete
Configuration
Structure:
{
"controls": {},
"mongodb": {},
"elasticsearch": {},
"tasks": [
{
"extract": {},
"transform": {},
"load": {}
}
]
}
controls
-
mongodbReadCapacity
- Max docs read per second (default:10000
). (optional) -
elasticsearchBulkInterval
- Max bluk interval per request (default:5000
). (optional) -
elasticsearchBulkSize
- Max bluk size per request (default:5000
). (optional) -
indexNameSuffix
- Index name suffix, for index version control. (optional)
mongodb
-
url
- The connection URI string, eg:mongodb://user:[email protected]:27017/db?replicaSet=rs0
. notice: must use aadmin
user to access oplog. -
options
- Connection settings, see: MongoClient. (optional)
elasticsearch
-
options
- Elasticsearch Config Options, see: Configuration. -
indices
- If set, auto create indices when program start, see: Indeces Create. (optional)
task.from
-
phase
-scan
ortail
-
time
- tail oplog with query:{ ts: { $gte: new Timestamp(0, new Date(time).getTime() / 1000) } }
-
id
- scan collection with query{ _id: { $gte: id }}
task.extract
-
db
- Database name. -
collection
- Collection name in database. -
projection
- Projection selector, see Projection.
task.transform
-
mapping
- The field mapping from mongodb's collection to elasticsearch's index. -
parent
- The field in mongodb's collection to use as the_parent
in elasticsearch's index. (optional)
task.load
-
index
- The name of the index. -
type
- The name of the document type. -
body
- The request body, see Put Mapping.
License
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].