All Projects → FriendsOfDoctrine → dbal-clickhouse

FriendsOfDoctrine / dbal-clickhouse

Licence: GPL-3.0 license
Doctrine DBAL driver for ClickHouse database

Programming Languages

PHP
23972 projects - #3 most used programming language

Projects that are alternatives of or similar to dbal-clickhouse

stefano-tree
Framework agnostic Nested Set (MPTT) implementation for PHP
Stars: ✭ 24 (-68.83%)
Mutual labels:  doctrine-dbal, doctrine2
Proton
High performance Pinba server
Stars: ✭ 27 (-64.94%)
Mutual labels:  clickhouse
click house
Modern Ruby database driver for ClickHouse
Stars: ✭ 133 (+72.73%)
Mutual labels:  clickhouse
uptrace
Open source APM: OpenTelemetry traces, metrics, and logs
Stars: ✭ 1,187 (+1441.56%)
Mutual labels:  clickhouse
fense
Fense is a database proxy written in Java, which can connect DB of different engines at the same time. The key features are: authority management, query cache, audit security, current limiting fuse, onesql and so on
Stars: ✭ 22 (-71.43%)
Mutual labels:  clickhouse
ClickHouseTools
Инструменты обслуживания и разработки для Yandex ClickHouse, а также другие интересности
Stars: ✭ 16 (-79.22%)
Mutual labels:  clickhouse
spark-clickhouse-connector
Spark ClickHouse Connector build on DataSourceV2 API
Stars: ✭ 100 (+29.87%)
Mutual labels:  clickhouse
awesome-clickhouse
A curated list of awesome ClickHouse software.
Stars: ✭ 71 (-7.79%)
Mutual labels:  clickhouse
clickhouse-ast-parser
AST parser and visitor for ClickHouse SQL
Stars: ✭ 60 (-22.08%)
Mutual labels:  clickhouse
dbal-rds-data
A driver to use the aws aurora serverless rds data api in the doctrine database abstraction layer
Stars: ✭ 24 (-68.83%)
Mutual labels:  doctrine-dbal
openreplay
📺 OpenReplay is developer-friendly, open-source session replay.
Stars: ✭ 6,131 (+7862.34%)
Mutual labels:  clickhouse
yabr.os
Чтение скобочного формата файлов 1С (oscript)
Stars: ✭ 33 (-57.14%)
Mutual labels:  clickhouse
ClickHouseMigrator
Help to migrate data to ClickHouse, create database and table auto.
Stars: ✭ 58 (-24.68%)
Mutual labels:  clickhouse
sql exporter
Database agnostic SQL exporter for Prometheus
Stars: ✭ 72 (-6.49%)
Mutual labels:  clickhouse
appmetrica-logsapi-loader
A tool for automatic data loading from AppMetrica LogsAPI into (local) ClickHouse
Stars: ✭ 18 (-76.62%)
Mutual labels:  clickhouse
doctrine-extensions
Doctrine2 behavioral extension Transformable
Stars: ✭ 14 (-81.82%)
Mutual labels:  doctrine2
chtable
Grafana's table plugin for ClickHouse
Stars: ✭ 26 (-66.23%)
Mutual labels:  clickhouse
cds
Data syncing in golang for ClickHouse.
Stars: ✭ 839 (+989.61%)
Mutual labels:  clickhouse
onelinerhub
2.5k code solutions with clear explanation @ onelinerhub.com
Stars: ✭ 645 (+737.66%)
Mutual labels:  clickhouse
trickster
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator
Stars: ✭ 1,753 (+2176.62%)
Mutual labels:  clickhouse

If you want to become a maintainer of this package -- please, contact me via [email protected]

Build Status Total Downloads Latest Stable Version License

Doctrine DBAL ClickHouse Driver

Doctrine DBAL driver for ClickHouse -- an open-source column-oriented database management system by Yandex (https://clickhouse.yandex/)

Driver is suitable for Symfony or any other framework using Doctrine.

Installation

composer require friendsofdoctrine/dbal-clickhouse

Initialization

Custom PHP script

$connectionParams = [
    'host' => 'localhost',
    'port' => 8123,
    'user' => 'default',
    'password' => '',
    'dbname' => 'default',
    'driverClass' => 'FOD\DBALClickHouse\Driver',
    'wrapperClass' => 'FOD\DBALClickHouse\Connection',
    'driverOptions' => [
        'extremes' => false,
        'readonly' => true,
        'max_execution_time' => 30,
        'enable_http_compression' => 0,
        'https' => false,
    ],
];
$conn = \Doctrine\DBAL\DriverManager::getConnection($connectionParams, new \Doctrine\DBAL\Configuration());

driverOptions are special smi2/phpclickhouse client settings

Symfony

configure...

# app/config/config.yml
doctrine:
    dbal:
        connections:
            clickhouse:
                host:     localhost
                port:     8123
                user:     default
                password: ""
                dbname:   default
                driver_class: FOD\DBALClickHouse\Driver
                wrapper_class: FOD\DBALClickHouse\Connection
                options:
                    enable_http_compression: 1
                    max_execution_time: 60
            #mysql:
            #   ...

...and get from the service container

$conn = $this->get('doctrine.dbal.clickhouse_connection');

Usage

Create new table

// ***quick start***
$fromSchema = $conn->getSchemaManager()->createSchema();
$toSchema = clone $fromSchema;


// create new table object
$newTable = $toSchema->createTable('new_table');

// add columns
$newTable->addColumn('id', 'integer', ['unsigned' => true]);
$newTable->addColumn('payload', 'string', ['notnull' => false]);
// *option 'notnull' in false mode allows you to insert NULL into the column; 
//                   in this case, the column will be represented in the ClickHouse as Nullable(String)
$newTable->addColumn('hash', 'string', ['length' => 32, 'fixed' => true]);
// *option 'fixed' sets the fixed length of a string column as specified; 
//                 if specified, the type of the column is FixedString

//set primary key
$newTable->setPrimaryKey(['id']);


// execute migration SQLs to create table in ClickHouse
$sqlArray = $fromSchema->getMigrateToSql($toSchema, $conn->getDatabasePlatform());
foreach ($sqlArray as $sql) {
    $conn->exec($sql);
}
// ***more options (optional)***

//specify table engine
$newTable->addOption('engine', 'MergeTree');
// *if not specified -- default engine 'ReplacingMergeTree' will be used


// add Date column for partitioning
$newTable->addColumn('event_date', 'date', ['default' => 'toDate(now())']);
$newTable->addOption('eventDateColumn', 'event_date');
// *if not specified -- default Date column named EventDate will be added
$newTable->addOption('eventDateProviderColumn', 'updated_at');
// *if specified -- event date column will be added with default value toDate(updated_at); 
//    if the type of the provider column is `string`, the valid format of provider column values must be either `YYYY-MM-DD` or `YYYY-MM-DD hh:mm:ss`
//    if the type of provider column is neither `string`, nor `date`, nor `datetime`, provider column values must contain a valid UNIX Timestamp
$newTable->addOption('samplingExpression', 'intHash32(id)');
// samplingExpression -- a tuple that defines the table's primary key, and the index granularity

//specify index granularity
$newTable->addOption('indexGranularity', 4096);
// *if not specified -- default value 8192 will be used

Insert

// 1
$conn->exec("INSERT INTO new_table (id, payload) VALUES (1, 'dummyPayload1')");
// 2
$conn->insert('new_table', ['id' => 2, 'payload' => 'dummyPayload2']);
// INSERT INTO new_table (id, payload) VALUES (?, ?) [2, 'dummyPayload2']
// 3 via QueryBuilder
$qb = $conn->createQueryBuilder();

$qb
    ->insert('new_table')
    ->setValue('id', ':id')
    ->setValue('payload', ':payload')
    ->setParameter('id', 3, \PDO::PARAM_INT) // need to explicitly set param type to `integer`, because default type is `string` and ClickHouse doesn't like types mismatchings
    ->setParameter('payload', 'dummyPayload3');

$qb->execute();

Select

echo $conn->fetchColumn('SELECT SUM(views) FROM articles');

Select via Dynamic Parameters and Prepared Statements

$stmt = $conn->prepare('SELECT authorId, SUM(views) AS total_views FROM articles WHERE category_id = :categoryId AND publish_date = :publishDate GROUP BY authorId');

$stmt->bindValue('categoryId', 123);
$stmt->bindValue('publishDate', new \DateTime('2017-02-29'), 'datetime');
$stmt->execute();

while ($row = $stmt->fetch()) {
    echo $row['authorId'] . ': ' . $row['total_views'] . PHP_EOL;
}

Additional types

If you want to use Array(T) type, register additional DBAL types in your code:

// register all custom DBAL Array types
ArrayType::registerArrayTypes($conn->getDatabasePlatform());
// register one custom DBAL Array(Int8) type
Type::addType('array(int8)', 'FOD\DBALClickHouse\Types\ArrayInt8Type');

or register them in Symfony configuration file:

# app/config/config.yml
doctrine:
    dbal:
        connections:
        ...
        types:
            array(int8): FOD\DBALClickHouse\Types\ArrayInt8Type
            array(int16): FOD\DBALClickHouse\Types\ArrayInt16Type
            array(int32): FOD\DBALClickHouse\Types\ArrayInt32Type
            array(int64): FOD\DBALClickHouse\Types\ArrayInt64Type
            array(uint8): FOD\DBALClickHouse\Types\ArrayUInt8Type
            array(uint16): FOD\DBALClickHouse\Types\ArrayUInt16Type
            array(uint32): FOD\DBALClickHouse\Types\ArrayUInt32Type
            array(uint64): FOD\DBALClickHouse\Types\ArrayUInt64Type
            array(float32): FOD\DBALClickHouse\Types\ArrayFloat32Type
            array(float64): FOD\DBALClickHouse\Types\ArrayFloat64Type
            array(string): FOD\DBALClickHouse\Types\ArrayStringType
            array(datetime): FOD\DBALClickHouse\Types\ArrayDateTimeType
            array(date): FOD\DBALClickHouse\Types\ArrayDateType

Additional type BigIntType helps you to store bigint values as Int64/UInt64 value type in ClickHouse. You can override DBAL type in your code:

Type::overrideType(Type::BIGINT, 'FOD\DBALClickHouse\Types\BigIntType');

or use custom mapping types in Symfony configuration:

# app/config/config.yml
doctrine:
    dbal:
        types:
            bigint:  FOD\DBALClickHouse\Types\BigIntType
            ...

More information in Doctrine DBAL documentation:

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].