Stereo Magnification: Learning View Synthesis using Multiplane Images

This code accompanies the paper

Stereo Magnification: Learning View Synthesis using Multiplane Images
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, Noah Snavely
SIGGRAPH 2018

Please note that this is not an officially supported Google product.

Training the MPI model

The entry python script for training is train.py. The input flags are specified in two places: 1) train.py and 2) stereomag/loader.py.

The input flag which_color_pred specifies how to predict the color image at each MPI plane:

bg [default] - Our default model. The network predicts: 1) weights for blending the background and foreground (reference source image) color images at each plane, 2) the alphas at each plane. 3) a background color image fgbg - Instead of using the reference source as the foreground image, the network predicts an extra foreground image for blending with the background alpha_only - No color image (or blending weights) is predicted by the network. The reference source image is used as the color image at each MPI plane. single - The network predicts a single color image shared for all MPI planes. all - The network directly outputs the color image at each MPI plane.

You can also specify which loss to use for training: pixel or vgg (i.e., the perceptual loss as measured by differences in VGG features). Note that when using the VGG loss, you will need to download the pre-trained VGG model imagenet-vgg-verydeep-19.mat available at

http://www.vlfeat.org/matconvnet/pretrained/#downloading-the-pre-trained-models

The path to this file can be set by the vgg_model_file flag in train.py.

Testing the MPI model

The entry python script for testing the models is test.py.

One could specify what to output to disk by concatenating one or more of the following (e.g. with '_'): src_images, ref_image, tgt_image, psv, fgbg, poses, intrinsics, blend_weights, rgba_layers.

psv - the plane sweep volume used as input to the network. fgbg - foreground and background color images (only valid when which_color_pred is either fgbg or bg) blend_weights - weights for blending foreground and backgroud color images (only valid when which_color_pred is either fgbg or bg)

Quantitative evaluation

evaluate.py contains sample code for evaluating the view synthesis performance based on the SSIM and PSNR metrics. It assumes that each scene result folder contains a ground-truth target image tgt_image_*.png and the synthesized image output_image_*.png. The script will output a text file summarizing the metrics inside the folder FLAGS.result_root.

Pre-trained models from SIGGRAPH'18 paper

Our pre-trained model can be downloaded into the models subdirectory by running the script bash scripts/download_model.sh.

Running the model on a single image pair

To run a trained model on a single image pair to generate an MPI, use mpi_from_images.py. This tool assumes images with the same orientation (as with a rectified stereo pair), but allows for specifying the (x, y, z) offset between the images.

You can find example input stereo pairs and command lines for generating results in the examples directory.

(You must first download the pretrained model or train your own model and place in the models/ subdirectory)

Reference examples and results

For reference, you can find additional example input stereo pairs, as well as corresponding output multi-plane images and view synthesis results used in the paper in this Google drive link (772 MB).

RealEstate10K dataset

We have released the RealEstate10K dataset suitable for training and testing the MPI model. Note that due to data restrictions, this is not the same version used in our SIGGRAPH'18 paper. However, we are working on updating the results using this public version.