spark-druid-connector
A library for querying Druid data sources with Apache Spark.
Compatability
This libaray is compatable with Spark-2.x and Druid-0.9.0+
Usage
Compile
sbt clean assembly
Using with spark-shell
bin/spark-shell --jars spark-druid-connector-assembly-0.1.0-SNAPSHOT.jar
In spark-shell, a temp table could be created like this:
val df = spark.read.format("org.rzlabs.druid").
option("druidDatasource", "ds1").
option("zkHost", "localhost:2181").
option("hyperUniqueColumnInfo", """[{"column":"city", "hllMetric": "unique_city"}]""").load
df.createOrReplaceTempView("ds")
spark.sql("select time, sum(event) from ds group by time").show
or you can create a hive table:
spark.sql("""
create table ds1 using org.rzlabs.druid options (
druidDatasource "ds1",
zkHost "localhost:2181",
hyperUniqueColumnInfo, "[{\"column\": \"city\", \"hllMetric\": \"unique_city\"}]"
)
""")
Options
| option | required | default value | descrption |
|---|---|---|---|
| druidDatasource | yes | none | data source name in Druid |
| zkHost | no | localhost | zookeeper server Druid use, e.g., localhost:2181 |
| zkSessionTimeout | no | 30000 | zk server connection timeout |
| zkEnableCompression | no | true | zk enbale compression or not |
| zkDruidPath | no | /druid | The druid metadata root path in zk |
| zkQualifyDiscoveryNames | no | true | |
| queryGranularity | no | all | The query granularity of the Druid datasource |
| maxConnectionsPerRoute | no | 20 | The max simultaneous live connections per Druid server |
| maxConnections | no | 100 | The max simultaneous live connnections of the Druid cluster |
| loadMetadataFromAllSegments | no | true | Fetch metadata from all available segments or not |
| debugTransformations | no | false | Log debug informations about the transformations or not |
| timeZoneId | no | UTC | |
| useV2GroupByEngine | no | false | Use V2 groupby engine or not |
| useSmile | no | true | Use smile binary format as the data format exchanged between client and Druid servers |
Major features
Currently
- Direct table creating in Spark without requiring of base table.
- Support Aggregate and Project & Filter operators pushing down and transform to GROUPBY and SCAN query against Druid accordingly.
- Support majority of primitive filter specs, aggregation specs and extraction functions.
- Lightweight datasource metadata updating.
In the future
- Support Join operator.
- Support Limit and Having operators pushing down.
- Suport more primitive specs and extraction functions.
- Support more Druid query specs according to query details.
- Suport datasource creating and metadata lookup.
- ...