RedisStorer

A UDF StoreFunc for Apache Pig designed to bulk-load data into Redis. Inspired by wonderdog, the Infochimps bulk-loader for elasticsearch.

Compiling and running

Compile:

Dependencies are automatically retrieved using Ivy.

$ ant hadoop

Use:

$ pig
grunt> REGISTER dist/pig-redis.jar;
grunt> a = LOAD 'somefile.tsv' USING PigStorage('\t');
grunt> STORE a INTO 'dummy-filename-is-ignored' USING com.hackdiary.pig.RedisStorer('kv', 'localhost');

Bulkloading strategy

RedisStorer runs in four modes: kv, set, hash and list (specified as the first argument to RedisStorer). If no mode is specified, kv is the default.

In kv mode, it takes the first field of the stored tuple as the key, and the second field as the value, and issues SET key value. Any further fields are ignored.

In set mode, it takes the first field of the stored tuple as the key, and issues SADD key value once for each subsequent field value in the tuple.

In hash mode, it takes the first field of the stored tuple as the key, and issues HSET key fieldname value once for each subsequent field value, using the same key for each, and taking the fieldname from the tuple's schema fieldnames. This means that it will fail unless the stored tuple has a schema with named fields.

In list mode, it takes the first field of the stored tuple as the key, and issues LPUSH key value once for each subsequent field value in the tuple.