zillow / Luminaire

A hands-off Anomaly Detection Library

Table of contents

• What is Luminaire
• Quick Start
• Time Series Outlier Detection Workflow
• Anomaly Detection for High Frequency Time Series
• Contributing
• Acknowledgements
• Development Team

What is Luminaire

Luminaire is a python package that provides ML-driven solutions for monitoring time series data. Luminaire provides several anomaly detection and forecasting capabilities that incorporate correlational and seasonal patterns as well as uncontrollable variations in the data over time.

Quick Start

Install Luminaire from PyPI using pip

pip install luminaire

Import luminaire module in python

import luminaire

Check out Luminaire documentation for detailed description of methods and usage.

Time Series Outlier Detection Workflow

Luminaire outlier detection workflow can be divided into 3 major components:

Data Preprocessing and Profiling Component

This component can be called to prepare a time series prior to training an anomaly detection model on it. This step applies a number of methods that make anomaly detection more accurate and reliable, including missing data imputation, identifying and removing recent outliers from training data, necessary mathematical transformations, and data truncation based on recent change points. It also generates profiling information (historical change points, trend changes, etc.) that are considered in the training process.

Profiling information for time series data can be used to monitor data drift and irregular long-term swings.

Modeling Component

This component performs time series model training based on the user-specified configuration OR optimized configuration (see Luminaire hyperparameter optimization). Luminaire model training is integrated with different structural time series models as well as filtering based models. See Luminaire outlier detection for more information.

The Luminaire modeling step can be called after the data preprocessing and profiling step to perform necessary data preparation before training.

Configuration Optimization Component

Luminaire's integration with configuration optimization enables a hands-off anomaly detection process where the user needs to provide very minimal configuration for monitoring any type of time series data. This step can be combined with the preprocessing and modeling for any auto-configured anomaly detection use case. See fully automatic outlier detection for a detailed walkthrough.

Anomaly Detection for High Frequency Time Series

Luminaire can also monitor a set of data points over windows of time instead of tracking individual data points. This approach is well-suited for streaming use cases where sustained fluctuations are of greater concern than individual fluctuations. See anomaly detection for streaming data for detailed information.

Contributing

Want to help improve Luminaire? Check out our contributing documentation.

Citing

Please cite the following article if Luminaire is used for any research purpose or scientific publication:

Chakraborty, S., Shah, S., Soltani, K., Swigart, A., Yang, L., & Buckingham, K. (2020). Building an Automated and Self-Aware Anomaly Detection System. arXiv preprint arXiv:2011.05047. (arxiv link)

Other Useful Resources

1. Chakraborty, S., Shah, S., Soltani, K., & Swigart, A. (2019, December). Root Cause Detection Among Anomalous Time Series Using Temporal State Alignment. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA) (pp. 523-528). IEEE. (arxiv link)

Acknowledgements

This project has leveraged methods described in the following scientific publications:

1. Soule, Augustin, Kavé Salamatian, and Nina Taft. "Combining filtering and statistical methods for anomaly detection. " Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement. 2005.

Development Team

Luminaire is developed and maintained by Sayan Chakraborty, Smit Shah, Kiumars Soltani, Luyao Yang, Anna Swigart, Kyle Buckingham and many other contributors from the Zillow Group A.I. team.