All Projects → vahid-sohrabloo → chconn

vahid-sohrabloo / chconn

Licence: MIT license
Low-level ClickHouse database driver for Golang

Programming Languages

go
31211 projects - #10 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to chconn

chclient
Fast http client for SELECT queries in clickhouse
Stars: ✭ 44 (-71.05%)
Mutual labels:  fast, clickhouse
ClickhouseBuilder
Fluent queries builder for Clickhouse. Also has integration with Laravel / Lumen.
Stars: ✭ 155 (+1.97%)
Mutual labels:  clickhouse
Pareto.js
An extremely small, intuitive and fast functional utility library for JavaScript
Stars: ✭ 254 (+67.11%)
Mutual labels:  fast
muparsersse
muparsersse a math parser for windows using just in time compilations of the expression
Stars: ✭ 14 (-90.79%)
Mutual labels:  fast
prometheus to clickhouse
Prometheus to Clickhouse Adapter
Stars: ✭ 14 (-90.79%)
Mutual labels:  clickhouse
MashaRoBot
MashaRoBot : 📑Editor's choice
Stars: ✭ 39 (-74.34%)
Mutual labels:  fast
Superstring
A fast and memory-optimized string library for C++
Stars: ✭ 252 (+65.79%)
Mutual labels:  fast
jazzle
An Innovative, Fast Transpiler for ECMAScript 2015 and later
Stars: ✭ 65 (-57.24%)
Mutual labels:  fast
aurum
Fast and concise declarative DOM rendering library for javascript
Stars: ✭ 17 (-88.82%)
Mutual labels:  fast
iso8601
A fast ISO8601 date parser for Go
Stars: ✭ 122 (-19.74%)
Mutual labels:  fast
docker-swoole-webapp
Simple Docker-image to build your applications based on Async PHP extensions.
Stars: ✭ 26 (-82.89%)
Mutual labels:  fast
OneSTools.EventLog
Библиотеки и готовые инструменты для чтения и экспорта журнала регистрации 1С в ClickHouse и ElasticSearch
Stars: ✭ 96 (-36.84%)
Mutual labels:  clickhouse
docker-clickhouse-grafana
Baseline Compose for Clickhouse + Grafana
Stars: ✭ 24 (-84.21%)
Mutual labels:  clickhouse
mini-async-log-c
Mini async log C port. Now with C++ wrappers.
Stars: ✭ 69 (-54.61%)
Mutual labels:  fast
DFPlayerMini Fast
Fast and easy to understand Arduino library to use the DFPlayer Mini MP3 module from DFRobot.com. This is a huge improvement (both in terms of execution speed and simplicity) to the standard library provided by DFRobot.com.
Stars: ✭ 164 (+7.89%)
Mutual labels:  fast
Geokdbush
The fastest spatial index for geographic locations in JavaScript
Stars: ✭ 251 (+65.13%)
Mutual labels:  fast
PyGLM
Fast OpenGL Mathematics (GLM) for Python
Stars: ✭ 167 (+9.87%)
Mutual labels:  fast
one-ck
php tcp client for clickhouse
Stars: ✭ 61 (-59.87%)
Mutual labels:  clickhouse
fast-speedtest-api
fast.com API / CLI tool
Stars: ✭ 138 (-9.21%)
Mutual labels:  fast
aioch
aioch - is a library for accessing a ClickHouse database over native interface from the asyncio
Stars: ✭ 145 (-4.61%)
Mutual labels:  clickhouse

Go Reference codecov Go Report Card Actions Status FOSSA Status

chconn - ClickHouse low level Driver

chconn is a pure generic Go (1.18+) driver for ClickHouse that use Native protocol chconn aims to be low-level, fast, and performant.

For comparison with other libraries, please see https://github.com/vahid-sohrabloo/go-ch-benchmark and https://github.com/go-faster/ch-bench#benchmarks

If you have any suggestion or comment, please feel free to open an issue

Example Usage

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/vahid-sohrabloo/chconn/v2/chpool"
	"github.com/vahid-sohrabloo/chconn/v2/column"
)

func main() {
	conn, err := chpool.New(os.Getenv("DATABASE_URL"))
	if err != nil {
		panic(err)
	}

	defer conn.Close()

	// to check if the connection is alive
	err = conn.Ping(context.Background())
	if err != nil {
		panic(err)
	}

	err = conn.Exec(context.Background(), `DROP TABLE IF EXISTS example_table`)
	if err != nil {
		panic(err)
	}

	err = conn.Exec(context.Background(), `CREATE TABLE  example_table (
		uint64 UInt64,
		uint64_nullable Nullable(UInt64)
	) Engine=Memory`)
	if err != nil {
		panic(err)
	}

	col1 := column.New[uint64]()
	col2 := column.New[uint64]().Nullable()
	rows := 1_000_0000 // Ten million rows - insert in 10 times
	numInsert := 10
	col1.SetWriteBufferSize(rows)
	col2.SetWriteBufferSize(rows)
	startInsert := time.Now()
	for i := 0; i < numInsert; i++ {
		for y := 0; y < rows; y++ {
			col1.Append(uint64(i))
			if i%2 == 0 {
				col2.Append(uint64(i))
			} else {
				col2.AppendNil()
			}
		}

		ctxInsert, cancelInsert := context.WithTimeout(context.Background(), time.Second*30)
		// insert data
		err = conn.Insert(ctxInsert, "INSERT INTO example_table (uint64,uint64_nullable) VALUES", col1, col2)
		if err != nil {
			cancelInsert()
			panic(err)
		}
		cancelInsert()
	}
	fmt.Println("inserted 10M rows in ", time.Since(startInsert))

	// select data
	col1Read := column.New[uint64]()
	col2Read := column.New[uint64]().Nullable()

	ctxSelect, cancelSelect := context.WithTimeout(context.Background(), time.Second*30)
	defer cancelSelect()

	startSelect := time.Now()
	selectStmt, err := conn.Select(ctxSelect, "SELECT uint64,uint64_nullable FROM  example_table", col1Read, col2Read)
	if err != nil {
		panic(err)
	}

	// make sure the stmt close after select. but it's not necessary
	defer selectStmt.Close()

	var col1Data []uint64
	var col2DataNil []bool
	var col2Data []uint64
	// read data block by block
	// for more information about block, see: https://clickhouse.com/docs/en/development/architecture/#block
	for selectStmt.Next() {
		col1Data = col1Data[:0]
		col1Data = col1Read.Read(col1Data)

		col2DataNil = col2DataNil[:0]
		col2DataNil = col2Read.ReadNil(col2DataNil)

		col2Data = col2Data[:0]
		col2Data = col2Read.Read(col2Data)
	}

	// check errors
	if selectStmt.Err() != nil {
		panic(selectStmt.Err())
	}
	fmt.Println("selected 10M rows in ", time.Since(startSelect))
}
inserted 10M rows in  1.206666188s
selected 10M rows in  880.505004ms

For moe information, please see the documentation

Features

  • Generics (go1.18) for column types
  • Connection pool with after-connect hook for arbitrary connection setup similar to pgx (thanks @jackc)
  • Support DSN and Query connection string (thanks @jackc)
  • Support All ClickHouse data types
  • Read and write data in column-oriented (like ClickHouse)
  • Do not use interface{} , reflect
  • Batch select and insert
  • Full TLS connection control
  • Read raw binary data
  • Supports profile and progress
  • database url connection very like pgx (thanks @jackc)
  • Code generator for Insert
  • Support LZ4 and ZSTD compression protocol
  • Support execution telemetry streaming profiles and progress

Supported types

  • UInt8, UInt16, UInt32, UInt64, UInt128, UInt256
  • Int8, Int16, Int32, Int64, Int128, Int256
  • Date, Date32, DateTime, DateTime64
  • Decimal32, Decimal64, Decimal128, Decimal256
  • IPv4, IPv6
  • String, FixedString(N)
  • UUID
  • Array(T)
  • Enums
  • LowCardinality(T)
  • Map(K, V)
  • Tuple(T1, T2, ..., Tn)
  • Nullable(T)
  • Point, Ring, Polygon, MultiPolygon

Benchmarks

the source code of this benchmark here https://github.com/vahid-sohrabloo/go-ch-benchmark

name \ time/op           chconn              chgo       go-clickhouse         uptrace
TestSelect100MUint64-16   150ms             154ms 	       8019ms 	       3045ms 	
TestSelect10MString-16    271ms 	    447ms 	        969ms 	        822ms 	
TestInsert10M-16          198ms 	    514ms 	        561ms 	        304ms 	

name \ alloc/op          chconn              chgo       go-clickhouse         uptrace
TestSelect100MUint64-16   111kB 	    262kB 	    3202443kB 	     800941kB 	
TestSelect10MString-16   1.63MB 	   1.79MB 	    1626.51MB 	     241.03MB 	
TestInsert10M-16         26.0MB 	  283.7MB 	     1680.4MB 	      240.2MB 	

name \ allocs/op         chconn              chgo       go-clickhouse         uptrace
TestSelect100MUint64-16    35.0 	   6683.0 	  200030937.0 	  100006069.0 	
TestSelect10MString-16     49.0 	   1748.0 	   30011991.0 	   20001120.0 	
TestInsert10M-16           26.0 	     80.0 	        224.0 	         50.0 	

License

FOSSA Status

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].