isolation-forest

This is a Scala/Spark implementation of the Isolation Forest unsupervised outlier detection algorithm. This library was created by James Verbus from the LinkedIn Anti-Abuse AI team.

This library supports distributed training and scoring using Spark data structures. It inherits from the Estimator and Model classes in Spark's ML library in order to take advantage of machinery such as Pipelines. Model persistence on HDFS is supported.

Copyright

Copyright 2019 LinkedIn Corporation All Rights Reserved.

Licensed under the BSD 2-Clause License (the "License"). See License in the project root for license information.

How to use

Building the library

It is recommended to use Scala 2.11.8 and Spark 2.3.0. To build, run the following:

./gradlew build

This will produce a jar file in the ./isolation-forest/build/libs/ directory.

If you want to use the library with Spark 2.4 (and the Scala 2.11.8 default), you can specify this when running the build command.

./gradlew build -PsparkVersion=2.4.3

You can also build an artifact with Spark 2.4 (or 3.0) and Scala 2.12.

./gradlew build -PsparkVersion=3.0.0 -PscalaVersion=2.12.11

Add an isolation-forest dependency to your project

Please check Bintray for the latest artifact versions.

Gradle example

The artifacts are available in JCenter, so you can specify the JCenter repository in the top-level build.gradle file.

repositories {
    jcenter()
}

Add the isolation-forest dependency to the module-level build.gradle file. Here are some examples for multiple recent Spark/Scala version combinations.

dependencies {
    compile 'com.linkedin.isolation-forest:isolation-forest_2.3.0_2.11:1.0.1'
}

dependencies {
    compile 'com.linkedin.isolation-forest:isolation-forest_2.4.3_2.11:1.0.1'
}

dependencies {
    compile 'com.linkedin.isolation-forest:isolation-forest_2.4.3_2.12:1.0.1'
}

dependencies {
    compile 'com.linkedin.isolation-forest:isolation-forest_3.0.0_2.12:1.0.1'
}

Maven example

If you are using the Maven Central repository, declare the isolation-forest dependency in your project's pom.xml file. Here are some examples for multiple recent Spark/Scala version combinations.

<dependency>
  <groupId>com.linkedin.isolation-forest</groupId>
  <artifactId>isolation-forest_2.3.0_2.11</artifactId>
  <version>1.0.1</version>
</dependency>

<dependency>
  <groupId>com.linkedin.isolation-forest</groupId>
  <artifactId>isolation-forest_2.4.3_2.11</artifactId>
  <version>1.0.1</version>
</dependency>

<dependency>
  <groupId>com.linkedin.isolation-forest</groupId>
  <artifactId>isolation-forest_2.4.3_2.12</artifactId>
  <version>1.0.1</version>
</dependency>

<dependency>
  <groupId>com.linkedin.isolation-forest</groupId>
  <artifactId>isolation-forest_3.0.0_2.12</artifactId>
  <version>1.0.1</version>
</dependency>

Model parameters

Training and scoring

Here is an example demonstrating how to import the library, create a new IsolationForest instance, set the model hyperparameters, train the model, and then score the training data.data is a Spark DataFrame with a column named features that contains a org.apache.spark.ml.linalg.Vector of the attributes to use for training. In this example, the DataFrame data also has a labels column; it is not used in the training process, but could be useful for model evaluation.

import com.linkedin.relevance.isolationforest._
import org.apache.spark.ml.feature.VectorAssembler

/**
  * Load and prepare data
  */

// Dataset from http://odds.cs.stonybrook.edu/shuttle-dataset/
val rawData = spark.read
  .format("csv")
  .option("comment", "#")
  .option("header", "false")
  .option("inferSchema", "true")
  .load("isolation-forest/src/test/resources/shuttle.csv")

val cols = rawData.columns
val labelCol = cols.last
 
val assembler = new VectorAssembler()
  .setInputCols(cols.slice(0, cols.length - 1))
  .setOutputCol("features")
val data = assembler
  .transform(rawData)
  .select(col("features"), col(labelCol).as("label"))

// scala> data.printSchema
// root
//  |-- features: vector (nullable = true)
//  |-- label: integer (nullable = true)

/**
  * Train the model
  */

val contamination = 0.1
val isolationForest = new IsolationForest()
  .setNumEstimators(100)
  .setBootstrap(false)
  .setMaxSamples(256)
  .setMaxFeatures(1.0)
  .setFeaturesCol("features")
  .setPredictionCol("predictedLabel")
  .setScoreCol("outlierScore")
  .setContamination(contamination)
  .setContaminationError(0.01 * contamination)
  .setRandomSeed(1)

val isolationForestModel = isolationForest.fit(data)
 
/**
  * Score the training data
  */

val dataWithScores = isolationForestModel.transform(data)

// scala> dataWithScores.printSchema
// root
//  |-- features: vector (nullable = true)
//  |-- label: integer (nullable = true)
//  |-- outlierScore: double (nullable = false)
//  |-- predictedLabel: double (nullable = false)

The output DataFrame, dataWithScores, is identical to the input data DataFrame but has two additional result columns appended with their names set via model parameters; in this case, these are named predictedLabel and outlierScore.

Saving and loading a trained model

Once you've trained an isolationForestModel instance as per the instructions above, you can use the following commands to save the model to HDFS and reload it as needed.

val path = "/user/testuser/isolationForestWriteTest"

/**
  * Persist the trained model on disk
  */

// You can ensure you don't overwrite an existing model by removing .overwrite from this command
isolationForestModel.write.overwrite.save(path)

/**
  * Load the saved model from disk
  */

val isolationForestModel2 = IsolationForestModel.load(path)

Validation

The original 2008 "Isolation forest" paper by Liu et al. published the AUROC results obtained by applying the algorithm to 12 benchmark outlier detection datasets. We applied our implementation of the isolation forest algorithm to the same 12 datasets using the same model parameter values used in the original paper. We used 10 trials per dataset each with a unique random seed and averaged the result. The quoted uncertainty is the one-sigma error on the mean.

Our implementation provides AUROC values that are in very good agreement the results in the original Liu et al. publication. There are a few very small discrepancies that are likely due the limited precision of the AUROC values reported in Liu et al.

Contributions

If you would like to contribute to this project, please review the instructions here.

References

• F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in 2008 Eighth IEEE International Conference on Data Mining, 2008, pp. 413–422.
• F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation-based anomaly detection,” ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 6, no. 1, p. 3, 2012.
• Shebuti Rayana (2016). ODDS Library [http://odds.cs.stonybrook.edu]. Stony Brook, NY: Stony Brook University, Department of Computer Science.