Apache DataFu

Follow @apachedatafu

Apache DataFu is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.

It consists of two libraries:

• Apache DataFu Pig: a collection of user-defined functions for Apache Pig
• Apache DataFu Hourglass: an incremental processing framework for Apache Hadoop in MapReduce

DataFu is currently undergoing incubation with Apache. A mirror of the official git repository can be found on GitHub at https://github.com/apache/incubator-datafu.

For more information please visit the website:

• http://datafu.incubator.apache.org/

If you'd like to jump in and get started, check out the corresponding guides for each library:

• Apache DataFu Pig - Getting Started
• Apache DataFu Hourglass - Getting Started

Blog Posts

• Introducing DataFu
• DataFu: The WD-40 of Big Data
• DataFu 1.0
• DataFu's Hourglass: Incremental Data Processing in Hadoop

Presentations

• A Brief Tour of DataFu
• Building Data Products at LinkedIn with DataFu
• Hourglass: a Library for Incremental Processing on Hadoop (IEEE BigData 2013)
• DataFu @ ApacheCon 2014

Videos

• Introduction to Apache DataFu @ ApacheCon 2014

Other Resources

An interesting example of using Quantile from DataFu can be found in the Hadoop Real-World Solutions Cookbook.

From Around the Web

• DataFu Enters Incubation Status at Apache
• DataFu: Open Source Apache Pig UDFs by LinkedIn
• LinkedIn Opens DataFu: A Library for Working with Hadoop and Pig

Papers

• Hourglass: a Library for Incremental Processing on Hadoop (IEEE BigData 2013)

Getting Help

Please visit the website:

• http://datafu.incubator.apache.org/