nikepan / Clickhouse Bulk
Licence: apache-2.0
Collects many small inserts to ClickHouse and send in big inserts
Stars: ✭ 269
Programming Languages
go
31211 projects - #10 most used programming language
Labels
Projects that are alternatives of or similar to Clickhouse Bulk
awesome-clickhouse
A curated list of awesome ClickHouse software.
Stars: ✭ 71 (-73.61%)
Mutual labels: clickhouse
jaeger-clickhouse
Jaeger ClickHouse storage plugin implementation
Stars: ✭ 103 (-61.71%)
Mutual labels: clickhouse
dbal-clickhouse
Doctrine DBAL driver for ClickHouse database
Stars: ✭ 77 (-71.38%)
Mutual labels: clickhouse
dbt-clickhouse
The Clickhouse plugin for dbt (data build tool)
Stars: ✭ 77 (-71.38%)
Mutual labels: clickhouse
litemall-dw
基于开源Litemall电商项目的大数据项目,包含前端埋点(openresty+lua)、后端埋点;数据仓库(五层)、实时计算和用户画像。大数据平台采用CDH6.3.2(已使用vagrant+ansible脚本化),同时也包含了Azkaban的workflow。
Stars: ✭ 36 (-86.62%)
Mutual labels: clickhouse
appmetrica-logsapi-loader
A tool for automatic data loading from AppMetrica LogsAPI into (local) ClickHouse
Stars: ✭ 18 (-93.31%)
Mutual labels: clickhouse
Clickhouse sinker
Easily load data from kafka to ClickHouse
Stars: ✭ 256 (-4.83%)
Mutual labels: clickhouse
chclient
Fast http client for SELECT queries in clickhouse
Stars: ✭ 44 (-83.64%)
Mutual labels: clickhouse
Addax
Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.
Stars: ✭ 615 (+128.62%)
Mutual labels: clickhouse
clickhouse hadoop
Import data from clickhouse to hadoop with pure SQL
Stars: ✭ 26 (-90.33%)
Mutual labels: clickhouse
radondb-clickhouse-kubernetes
Open Source,High Availability Cluster,based on ClickHouse
Stars: ✭ 54 (-79.93%)
Mutual labels: clickhouse
clickhouse-maxmind-geoip
A demonstration how to use ClickHouse with MaxMind GeoIP2 databases for geolocaiton
Stars: ✭ 82 (-69.52%)
Mutual labels: clickhouse
onelinerhub
2.5k code solutions with clear explanation @ onelinerhub.com
Stars: ✭ 645 (+139.78%)
Mutual labels: clickhouse
zenit
Zenit is a daemon collector for metrics and log parsers for any MySQL variant and ProxySQL
Stars: ✭ 24 (-91.08%)
Mutual labels: clickhouse
Clickhouse Mysql Data Reader
utility to read mysql data
Stars: ✭ 257 (-4.46%)
Mutual labels: clickhouse
growthbook
Open Source Feature Flagging and A/B Testing Platform
Stars: ✭ 2,342 (+770.63%)
Mutual labels: clickhouse
horgh-replicator
Golang binlog replication from MySQL to MySQL, PostgreSQL, Vertica, Clickhouse
Stars: ✭ 46 (-82.9%)
Mutual labels: clickhouse
ClickHouse-Bulk
Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers.
Installation
Download binary for you platorm
or
or from sources (Go 1.13+):
git clone https://github.com/nikepan/clickhouse-bulk
cd clickhouse-bulk
go build
Features
- Group n requests and send to any of ClickHouse server
- Sending collected data by interval
- Tested with VALUES, TabSeparated formats
- Supports many servers to send
- Supports query in query parameters and in body
- Supports other query parameters like username, password, database
- Supports basic authentication
For example:
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')
INSERT INTO table3 (c1, c2, c3) VALUES ('v4', 'v5', 'v6')
sends as
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')('v4', 'v5', 'v6')
Options
- -config - config file (json); default config.json
Configuration file
{
"listen": ":8124",
"flush_count": 10000, // check by \n char
"flush_interval": 1000, // milliseconds
"clean_interval": 0, // how often cleanup internal tables - e.g. inserts to different temporary tables, or as workaround for query_id etc. milliseconds
"remove_query_id": true, // some drivers sends query_id which prevents inserts to be batched
"dump_check_interval": 300, // interval for try to send dumps (seconds); -1 to disable
"debug": false, // log incoming requests
"dump_dir": "dumps", // directory for dump unsended data (if clickhouse errors)
"clickhouse": {
"down_timeout": 60, // wait if server in down (seconds)
"connect_timeout": 10, // wait for server connect (seconds)
"tls_server_name": "", // override TLS serverName for certificate verification (e.g. in cases you share same "cluster" certificate across multiple nodes)
"insecure_tls_skip_verify": false, // INSECURE - skip certificate verification at all
"servers": [
"http://127.0.0.1:8123"
]
}
}
Environment variables (used for docker image)
-
CLICKHOUSE_BULK_DEBUG
- enable debug logging -
CLICKHOUSE_SERVERS
- comma separated list of servers -
CLICKHOUSE_FLUSH_COUNT
- count of rows for insert -
CLICKHOUSE_FLUSH_INTERVAL
- insert interval -
CLICKHOUSE_CLEAN_INTERVAL
- internal tables clean interval -
DUMP_CHECK_INTERVAL
- interval of resend dumps -
CLICKHOUSE_DOWN_TIMEOUT
- wait time if server is down -
CLICKHOUSE_CONNECT_TIMEOUT
- clickhouse server connect timeout -
CLICKHOUSE_TLS_SERVER_NAME
- server name for TLS certificate verification -
CLICKHOUSE_INSECURE_TLS_SKIP_VERIFY
- skip certificate verification at all
Quickstart
./clickhouse-bulk
and send queries to :8124
Metrics
manual check main metrics
curl -s http://127.0.0.1:8124/metrics | grep "^ch_"
-
ch_bad_servers 0
- actual count of bad servers -
ch_dump_count 0
- dumps saved from launch -
ch_queued_dumps 0
- actual dump files id directory -
ch_good_servers 1
- actual good servers count -
ch_received_count 40
- received requests count from launch -
ch_sent_count 1
- sent request count from launch
Tips
For better performance words FORMAT and VALUES must be uppercase.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].