Marco's SysML reading list

A curated reading list of computer science research for work at the intersection of machine learning and systems. PR are welcome.

Review

A Berkeley View of Systems Challenges for AI https://arxiv.org/pdf/1712.05855.pdf

Strategies and Principles of Distributed Machine Learning on Big Data https://arxiv.org/abs/1512.09295

Background

Deep learning Nature volume 521, 2015 https://www.nature.com/articles/nature14539

Deep learning reading list http://deeplearning.net/reading-list

Measurement

Multi-tenant GPU Clusters for Deep LearningWorkloads: Analysis and Implications https://www.microsoft.com/en-us/research/uploads/prod/2018/05/gpu_sched_tr.pdf

Frameworks

TensorFlow: A System for Large-Scale Machine Learning OSDI 2016 https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf

Ray: A Distributed Framework for Emerging AI Applications OSDI 2018 https://www.usenix.org/system/files/osdi18-moritz.pdf

Tuning

HyperDrive: Exploring Hyperparameters with POP Scheduling MiddleWare 2017 https://dl.acm.org/citation.cfm?id=3135994

Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads VLDB 2018 http://www.vldb.org/pvldb/vol11/p607-li.pdf

Automating Model Search for Large Scale Machine Learning SoCC 2015 http://dl.acm.org/authorize?N91362

Google Vizier: A Service for Black-Box Optimization KDD 2017 https://dl.acm.org/citation.cfm?id=3098043

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization Journal of Machine Learning Research 18 (2018) https://arxiv.org/pdf/1603.06560.pdf

Hyperopt: a Python library for model selection and hyperparameter optimization Computational Science & Discovery, 8(1) 2015 http://iopscience.iop.org/article/10.1088/1749-4699/8/1/014008

Auto-Keras: Efficient Neural Architecture Search with Network Morphism https://arxiv.org/pdf/1806.10282v2.pdf

Runtime execution

Cavs: An Efficient Runtime System for Dynamic Neural Networks ATC 2018 https://www.usenix.org/system/files/conference/atc18/atc18-xu-shizhen.pdf

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning OSDI 2018 https://www.usenix.org/system/files/osdi18-chen.pdf

PipeDream: Fast and Efficient Pipeline Parallel DNN Training https://arxiv.org/pdf/1806.03377.pdf

STRADS: A Distributed Framework for Scheduled Model Parallel Machine Learning EuroSys 2016 https://dl.acm.org/citation.cfm?id=2901331

Dynamic Control Flow in Large-Scale Machine Learning EuroSys 2018 https://dl.acm.org/citation.cfm?id=3190551

Improving the Expressiveness of Deep Learning Frameworks with Recursion EuroSys 2018 https://dl.acm.org/citation.cfm?id=3190530

Continuum: A Platform for Cost-Aware, Low-Latency Continual Learning SoCC 2018 https://dl.acm.org/citation.cfm?id=3267817

KeystoneML: Optimizing Pipelines for Large-ScaleAdvanced Analytics ICDE 2017 https://amplab.cs.berkeley.edu/wp-content/uploads/2017/01/ICDE_2017_CameraReady_475.pdf

Owl: A General-Purpose Numerical Library in OCaml https://arxiv.org/pdf/1707.09616.pdf

Distributed learning

Large Scale Distributed Deep Networks NIPS 2012 https://ai.google/research/pubs/pub40565.pdf

Managed Communication and Consistency for Fast Data-Parallel Iterative Analytics SoCC 2015 http://dl.acm.org/authorize?N91363

Ako: Decentralised Deep Learning with Partial Gradient Exchange SOCC 2016 https://lsds.doc.ic.ac.uk/sites/default/files/ako-socc16.pdf

Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters ATC 2017 https://www.usenix.org/system/files/conference/atc17/atc17-zhang.pdf

Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training SoCC 2018 https://dl.acm.org/citation.cfm?id=3267840

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems ML Systems Workshop at NIPS 2016 https://arxiv.org/pdf/1512.01274.pdf

Scaling Distributed Machine Learning with the Parameter Server OSDI 2014 https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-li_mu.pdf

Project Adam: Building an Efficient and Scalable Deep Learning Training System OSDI 2014 https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chilimbi.pdf

Orpheus: Efficient Distributed Machine Learning via System and Algorithm Co-design SoCC 2018 https://dl.acm.org/citation.cfm?id=3267810

Petuum: A New Platform for Distributed Machine Learning on Big Data KDD 2015 https://arxiv.org/pdf/1312.7651.pdf

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism https://arxiv.org/pdf/1811.06965.pdf

Serving systems and inference

DeepCPU: Serving RNN-based Deep Learning Models 10x Faster ATC 2018 https://www.usenix.org/system/files/conference/atc18/atc18-zhang-minjia.pdf

Clipper: A Low-Latency Online Prediction Serving System NSDI 2017 https://www.usenix.org/system/files/conference/nsdi17/nsdi17-crankshaw.pdf

Research for Practice: Prediction-Serving Systems ACM Queue 16(1), 2018 https://queue.acm.org/detail.cfm?id=3210557

InferLine: ML Inference Pipeline Composition https://arxiv.org/pdf/1812.01776.pdf

PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems OSDI 2018 https://www.usenix.org/system/files/osdi18-lee.pdf

Olympian: Scheduling GPU Usage in a Deep Neural Network Model Serving System MiddleWare 2018 https://dl.acm.org/citation.cfm?id=3274813

Low Latency RNN Inference with Cellular Batching EuroSys 2018 https://dl.acm.org/citation.cfm?id=3190541

SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism SC 2016 https://ieeexplore.ieee.org/document/7877104

NoScope: Optimizing Neural Network Queries over Video at Scale VLDB 2017 https://dl.acm.org/citation.cfm?id=3137664

Scheduling Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters EuroSys 2018 https://dl.acm.org/citation.cfm?id=3190517

SLAQ: Quality-Driven Scheduling for Distributed Machine Learning SoCC 2017 https://dl.acm.org/authorize?N46878

Proteus: agile ML elasticity through tiered reliability in dynamic resource markets EuroSys 2017 https://dl.acm.org/citation.cfm?id=3064182

Gandiva: Introspective Cluster Scheduling for Deep Learning OSDI 2018 https://www.usenix.org/system/files/osdi18-xiao.pdf

Topology-Aware GPU Scheduling for Learning Workloads in Cloud Environments SC 2017 https://dl.acm.org/citation.cfm?id=3126933

Algorithmic aspects in scalable ML

Hemingway: Modeling Distributed Optimization Algorithms ML Systems Workshop at NIPS 2016 https://arxiv.org/pdf/1702.05865.pdf

Asynchronous Methods for Deep Reinforcement Learning ICML 2016 http://proceedings.mlr.press/v48/mniha16.pdf

Don't Use Large Mini-Batches, Use Local SGD https://arxiv.org/pdf/1808.07217.pdf

GeePS: Scalable deep learning on distributed GPUs with a GPU-specialized parameter server EuroSys 2016 https://dl.acm.org/citation.cfm?id=2901323

ImageNet Training in Minutes ICPP 2018 https://dl.acm.org/citation.cfm?id=3225069

Semantics-Preserving Parallelization of Stochastic Gradient Descent IPDPS 2018 https://ieeexplore.ieee.org/abstract/document/8425176

HOGWILD!: A Lock-Free Approach to ParallelizingStochastic Gradient Descent NIPS 2011 https://papers.nips.cc/paper/4390-hogwild-a-lock-free-approach-to-parallelizing-stochastic-gradient-descent.pdf

QSGD: Communication-Efficient SGD via Randomized Quantization NIPS 2017 https://papers.nips.cc/paper/6768-qsgd-communication-efficient-sgd-via-gradient-quantization-and-encoding.pdf

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent NIPS 2017 https://papers.nips.cc/paper/7117-can-decentralized-algorithms-outperform-centralized-algorithms-a-case-study-for-decentralized-parallel-stochastic-gradient-descent.pdf

Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD AIStats 2018 https://arxiv.org/pdf/1803.01113.pdf

Probabilistic Synchronous Parallel https://arxiv.org/pdf/1709.07772.pdf

AI Testing and Verification

DeepXplore: Automated Whitebox Testing of Deep Learning Systems SOSP 2017 https://dl.acm.org/authorize?N47145

Programmatically Interpretable Reinforcement Learning ICML 2018 https://arxiv.org/pdf/1804.02477.pdf

AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation SP 2018 https://ieeexplore.ieee.org/document/8418593

Interpretability and Explainability

“Why Should I Trust You?”Explaining the Predictions of Any Classifier KDD 2016 https://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf

Learning to Explain: An Information-Theoretic Perspective on Model Interpretation ICML 2018 https://arxiv.org/pdf/1802.07814.pdf

A Unified Approach to Interpreting Model Predictions NIPS 2017 https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf

The Mythos of Model Interpretability WHI 2016 https://arxiv.org/pdf/1606.03490.pdf

Model Management

MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis SIGMOD 2018 https://dl.acm.org/citation.cfm?id=3196934

MODELDB: A System for Machine Learning Model Management HILDA 2016 https://mitdbg.github.io/modeldb/papers/hilda_modeldb.pdf

Model Governance: Reducing the Anarchy of Production ML ATC 2018 https://www.usenix.org/system/files/conference/atc18/atc18-sridhar.pdf

The Missing Piece in Complex Analytics: Low Latency,Scalable Model Management and Serving with Velox CIDR 2015 http://www.bailis.org/papers/velox-cidr2015.pdf

Bandana: Using Non-volatile Memory for Storing Deep Learning Models SysML 2019 https://arxiv.org/abs/1811.05922

Hardware

Deep learning with limited numerical precision ICML 2015 http://proceedings.mlr.press/v37/gupta15.pdf

In-Datacenter Performance Analysis of a Tensor Processing Unit ISCA 2017 https://dl.acm.org/citation.cfm?id=3080246

Serving DNNs in Real Timeat Datacenter Scale with Project Brainwave IEEE MICRO 38(2), Mar./Apr. 2018 https://ieeexplore.ieee.org/document/8344479

Security aspects

Efficient Deep Learning on Multi-Source Private Data https://arxiv.org/pdf/1807.06689.pdf

Chiron: Privacy-preserving Machine Learning as a Service https://arxiv.org/pdf/1803.05961.pdf

MLCapsule: Guarded Offline Deployment of Machine Learning as a Service https://arxiv.org/pdf/1808.00590.pdf

Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware https://arxiv.org/pdf/1806.03287.pdf

Privado: Practical and Secure DNN Inference https://arxiv.org/pdf/1810.00602.pdf

ML Platforms (Applied)

Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective HPCA 2018 https://research.fb.com/publications/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/

Machine Learning at Facebook: Understanding Inference at the Edge HPCA 2019 https://research.fb.com/publications/machine-learning-at-facebook-understanding-inference-at-the-edge/

Meet Michelangelo: Uber’s Machine Learning Platform https://eng.uber.com/michelangelo/

Introducing FBLearner Flow: Facebook’s AI backbone https://code.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/

TFX: A TensorFlow-Based Production-Scale Machine LearningPlatform http://dl.acm.org/authorize?N33328

Horovod: fast and easy distributed deep learning in TensorFlow https://arxiv.org/pdf/1802.05799v3.pdf

ML for Systems

Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms SOSP 2017 https://dl.acm.org/authorize?N47144

Adaptive Execution of Continuous and Data-intensive Workflows with Machine Learning MiddleWare 2018 https://dl.acm.org/citation.cfm?id=3274827

AuTO: Scaling Deep Reinforcement Learning to Enable Datacenter-Scale Automatic Traffic Optimization SIGCOMM 2018 https://dl.acm.org/citation.cfm?id=3230551

Neural Adaptive Video Streaming with Pensieve SIGCOMM 2017 https://dl.acm.org/citation.cfm?id=3098843

Neural Adaptive Content-aware Internet Video Delivery OSDI 2018 https://www.usenix.org/system/files/osdi18-yeo.pdf

Workshops

Systems for ML and Open Source Software Workshop at NeurIPS 2018 http://learningsys.org/nips18/acceptedpapers.html

SysML 2018 http://www.sysml.cc/2018/index.html

Engineering Dependable and Secure Machine Learning Systems 2019 https://sites.google.com/view/edsmls2019/program

Engineering Dependable and Secure Machine Learning Systems 2018 https://sites.google.com/edu.haifa.ac.il/edsmls/program

Workshop on Distributed Machine Learning 2017 https://distributedml2017.wordpress.com/schedule/

ML Systems Workshop at NIPS 2016 https://sites.google.com/site/mlsysnips2016/accepted-papers

Upcoming 2019

ColumnML: Column Store Machine Learning with On The Fly Data Transformation VLDB 2019

Continuous Integration of Machine Learning Models: A Rigorous Yet Practical Treatment SysML 2019

Seer: Leveraging Big Data to Navigate the Complexity of Performance Debugging in Cloud Microservices ASPLOS 2019

RLgraph: Flexible Computation Graphs for Deep Reinforcement Learning SysML 2019 https://arxiv.org/pdf/1810.09028.pdf

For adding/updating the list

1. Fork the repository
2. Update this file
3. Send a pull request