Benchmark ML Experiment Tracking Tools

This repo is a benchmark for ML experiment tracking tools. We build some ML projects from scratch and upgrade them with different experiment tracking tools. The goal is to provide a detailed comparison of different experiment tracking tools, so users can choose the best one for their projects.

• Training and Inference are supported.

• Experiment management with

  • hydra
  • tensorboard
  • neptune.ai
  • wandb
  • mlflow

• Various Frameworks and Models

  • PyTorch Vision for Scene Classification
  • TIMM for Image Classification
  • HuggingFace for NLP

• Model Zoo with pretrained models

🚀 Installation

# Download the code
git clone [email protected]:MLSysOps/ml_exp_tracking_benchmark.git
cd ml_exp_tracking_benchmark

# Create a conda environment
conda create -n ml_track_benchmark python=3.8
conda activate ml_track_benchmark

# Install dependencies
pip install - r requirements.txt

🏃‍♀️ Quick Start

Please download the data from [Place2 Data]

# 1. Download and unzip the data
sh download_data_pytorch.sh

# 2. Train a model
export PYTHONPATH=$PYTHONPATH:$(pwd)
python benchmark/main_tensorboard.py

🔨 Results

🔥 Neptune.ai

😀 Model Zoo (Pretrained Models)

Please refer [Model Zoo]

🎉 Acknowledge

The dataset and basic code comes from [MIT Place365]

Thanks for the great work!