ImVisible: Pedestrian Traffic Light Dataset, LytNet Neural Network, and Mobile Application for the Visually Impaired

The implementation of the work done in the following two papers: https://arxiv.org/abs/1907.09706, https://arxiv.org/abs/1909.09598.

Introduction

This project consists of three sections. First, we provide an image dataset of street intersections, labelled with the color of the corresponding pedestrian traffic light and the position of the zebra crossing in the image. Second, we provide a neural network adapted off of MobileNet v2 (LytNet) that accepts a larger input size while still running at near real-time speeds on an IPhone 7. Third, we provide a demo iOS application that is able to run LytNet and output the appropriate information onto the phone screen.

Pedestrian-Traffic-Lights (PTL) is a high-quality image dataset of street intersections, created for the detection of pedestrian traffic lights and zebra crossings. Images have variation in weather, position and orientation in relation to the traffic light and zebra crossing, and size and type of intersection.

Stats

Use these stats for image normalization:
mean = [120.56737612047593, 119.16664454573734, 113.84554638827127]
std=[66.32028460114392, 65.09469952002551, 65.67726614496246]

Labels

Each row of the csv files in the annotations folder contain a label for an image in the form:

[file_name, class, x1, y1, x2, y2, blocked tag].

An example is:

['IMG_2178.JPG', '2', '1040', '1712', '3210', '3016', 'not_blocked'].

Note that all labels are in String format, so it is neccessary to cast the coordinates to integers in python.

Classes are as follows:

0: Red

1: Green

2: Countdown Green

3: Countdown Blank

4: None

With the following distribution:

Images may contain multiple pedestrian traffic lights, in which the intended "main" traffic light was chosen.

The coordinates represent the start and endpoint of the midline of the zebra crossing. They are labelled as the position on the original 4032x3024 sized image, so if a different resolution is used it is important to convert the image coordinates to the appropriate values or normalize the coordinates to be between a range of [0, 1].

Download

Annotations can be downloaded from the annotations folder in this repo. There are three downloadable versions of the dataset. With our network, the 876x657 resolution images was used during training to accomodate random cropping. The 768x576 version was used during validation and testing without a random crop.

The 4032x3024 full resolution dataset is out! Download here: Part 1, Part 2.

Model

LytNet V1

We created our own pytorch neural network LytNet that can be accessed from the Model folder in this repo. The folder contains both the code and the weights after running the code with the dataset. Given and input image, our model will return the appropriate color of the traffic light, and two image coordinates representing the predicted endpoints of the zebra crossing.

Here are the precisions and recalls for each class:

Here are the endpoint errors:

Our network is adapted from MobileNet, with a larger input size of 768x576 designed for image classification tasks that involve a smaller object within the image (such as a traffic light). Certain layers from MobileNet v2 were removed for the network to run at a near real-time frame rate (21 fps), but still maintain high accuracy.

This is the structure of LytNet V1:

LytNet V2

Our network has been updated, achieving better accuracy. Below is a comparison between our two networks:

Application

A demo iOS application is also provided. Requirements are iOS 11 and above. The application continuously iterates through the flowchart below:

To use the application, open the LYTNet demo xcode project and add a developer team. Build the application on a device with iOS 12.0 or above.

Citations

Please consider citing our papers in your publications if this project helped with your research. The BibTeX references are as follows:

@InProceedings{yu2019lytnet,
  title = {LYTNet: A Convolutional Neural Network for Real-Time Pedestrian Traffic Lights and Zebra Crossing Recognition for the Visually Impaired},
  author = {Yu, Samuel and Lee, Heon and Kim, John},
  booktitle = {Computer Analysis of Images and Patterns (CAIP)},
  month = {Aug},
  year = {2019}
}
@InProceedings{yu2019lytnetv2,
  author = {Yu, Samuel and Lee, Heon and Kim, Junghoon},
  title = {Street Crossing Aid Using Light-Weight CNNs for the Visually Impaired},
  booktitle = {The IEEE International Conference on Computer Vision (ICCV) Workshops},
  month = {Oct},
  year = {2019}
}