MankaranSingh / Auto-Birds-Eye

Bird's Eye view generation and Mapping through end-to-end deep learning.

First of all, this project wouldn't had been possible without Maciek Dziubiński.

The repository works on building a model that takes in input the front camera image and generates the top-down view as well as simultaneous map generation (Can be thought of 2D SLAM).

Tesla's autopilot results:

Our results (In CARLA Simulator):

Training:

3 Models were trained:

• U-Net
• Autoencoder
• Deeper Autoencoder

Model was trained using following losses:

• SSIM
• Dice Loss
• Cross Entropy Loss

Front view images:

Predicted Bird's eye view:

Ground Truth:

Testing on real world data (Comma10k dataset):

The model was trained on a simulator, but I was able to obtain good results after finetuning the model and some further image processing. See how the car is correctly localized to the right and th road curve is correctly detected.

Map Generation (In Progress) :

This is based on image stitching using map coordinate metadata. I noticed that SIFT/SURF, etc based image feature extractors and matchers do not work in this case since the images produced are very symmetrical and sparse in colors, therefore, no good/unique features can be extracted from these images to perform stitching based on pixels.

Hence, we use location coordintes based stitching. Images are simply overlapped along with correct rotation.

Dataset:

Dataset can be found here.

Although, the dataset contains images from five camera:

• Front
• Top
• Left
• Right
• Rear

Note: For training the model, only front images were used to predict the half top-down image.

Try this out on Colab:

References:

https://mono.software/2018/03/14/Image-stitching/

https://medium.com/asap-report/from-semantic-segmentation-to-semantic-birds-eye-view-in-the-carla-simulator-1e636741af3f