Federated-Benchmark: A Benchmark of Real-world Images Dataset for Federated Learning

Overview

We present a real-world image dataset, reflecting the characteristic real-world federated learning scenarios, and provide provided an extensive benchmark on model performance, efficiency, and communication in a federated learning setting.

Resources

• Dataset: dataset.fedai.org
• Paper: "Real-World Image Datasets for Federated Learning"

Street_Dataset

• Overview: Image Dataset
• Details: 7 different classes, 956 images with pixels of 704 by 576, 5 or 20 devices
• Task: Object detection for federated learning
• Dataset_description.md

Getting Started

We implemented two mainstream object detection algorithms (YOLOv3 and Faster R-CNN). Code for YOLOv3 is borrowed from PyTorch-YOLOv3 and Faster R-CNN from simple-faster-rcnn-pytorch.

Install dependencies

• requires PyTorch with GPU (code are GPU only)
• install cupy, you can install via pip install cupy-cuda80 or (cupy-cuda90, cupy-cuda91, etc)
• install other dependencies, pip install -r requirements.txt
• Optional but strongly recommended: build cython code nms_gpu_post:

  cd model/utils/nms/
  python build.py build_ext --inplace
  cd -
  

Prepare data

1. Download the dataset, refer to dataset.fedai
2. It should have the basic structure for faster r-cnn

   Federated-Benchmark/data/street_5/$DEVICE_ID$/ImageSets
   Federated-Benchmark/data/street_5/$DEVICE_ID$/JPEGImages
   Federated-Benchmark/data/street_5/$DEVICE_ID$/Annotations
   

3. Generate config file for federated learning

   cd data
   python3 generate_task_json.py
   

Train

1. Start server

   sh ./run_server.sh street_5 yolo 1234
   

2. Start clients

   sh ./run.sh street_5 5 yolo 1234
   

3. Stop training

   sh ./stop.sh street_5 yolo