All Projects → simonmeister → Unflow

simonmeister / Unflow

Licence: mit
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Unflow

deepOF
TensorFlow implementation for "Guided Optical Flow Learning"
Stars: ✭ 26 (-89.12%)
Mutual labels:  optical-flow, unsupervised-learning
back2future
Unsupervised Learning of Multi-Frame Optical Flow with Occlusions
Stars: ✭ 39 (-83.68%)
Mutual labels:  optical-flow, unsupervised-learning
GuidedNet
Caffe implementation for "Guided Optical Flow Learning"
Stars: ✭ 28 (-88.28%)
Mutual labels:  optical-flow, unsupervised-learning
Joint-Motion-Estimation-and-Segmentation
[MICCAI'18] Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences
Stars: ✭ 45 (-81.17%)
Mutual labels:  optical-flow, unsupervised-learning
Ddflow
DDFlow: Learning Optical Flow with Unlabeled Data Distillation
Stars: ✭ 101 (-57.74%)
Mutual labels:  unsupervised-learning, optical-flow
Selflow
SelFlow: Self-Supervised Learning of Optical Flow
Stars: ✭ 319 (+33.47%)
Mutual labels:  unsupervised-learning, optical-flow
PCLNet
Unsupervised Learning for Optical Flow Estimation Using Pyramid Convolution LSTM.
Stars: ✭ 29 (-87.87%)
Mutual labels:  optical-flow, unsupervised-learning
Arflow
The official PyTorch implementation of the paper "Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation".
Stars: ✭ 134 (-43.93%)
Mutual labels:  unsupervised-learning, optical-flow
Voxelmorph
Unsupervised Learning for Image Registration
Stars: ✭ 1,057 (+342.26%)
Mutual labels:  unsupervised-learning, optical-flow
Cc
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation
Stars: ✭ 348 (+45.61%)
Mutual labels:  unsupervised-learning, optical-flow
Back2future.pytorch
Unsupervised Learning of Multi-Frame Optical Flow with Occlusions
Stars: ✭ 104 (-56.49%)
Mutual labels:  unsupervised-learning, optical-flow
Hidden Two Stream
Caffe implementation for "Hidden Two-Stream Convolutional Networks for Action Recognition"
Stars: ✭ 179 (-25.1%)
Mutual labels:  unsupervised-learning, optical-flow
Fleetx
Paddle Distributed Training Extended. 飞桨分布式训练扩展包
Stars: ✭ 196 (-17.99%)
Mutual labels:  unsupervised-learning
Pytorch Byol
PyTorch implementation of Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Stars: ✭ 213 (-10.88%)
Mutual labels:  unsupervised-learning
Liteflownet2
A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization, TPAMI 2020
Stars: ✭ 195 (-18.41%)
Mutual labels:  optical-flow
Pixelnet
The repository contains source code and models to use PixelNet architecture used for various pixel-level tasks. More details can be accessed at <http://www.cs.cmu.edu/~aayushb/pixelNet/>.
Stars: ✭ 194 (-18.83%)
Mutual labels:  unsupervised-learning
Contrastive Predictive Coding Pytorch
Contrastive Predictive Coding for Automatic Speaker Verification
Stars: ✭ 223 (-6.69%)
Mutual labels:  unsupervised-learning
Gan Sandbox
Vanilla GAN implemented on top of keras/tensorflow enabling rapid experimentation & research. Branches correspond to implementations of stable GAN variations (i.e. ACGan, InfoGAN) and other promising variations of GANs like conditional and Wasserstein.
Stars: ✭ 210 (-12.13%)
Mutual labels:  unsupervised-learning
Free Ai Resources
🚀 FREE AI Resources - 🎓 Courses, 👷 Jobs, 📝 Blogs, 🔬 AI Research, and many more - for everyone!
Stars: ✭ 192 (-19.67%)
Mutual labels:  unsupervised-learning
Df Net
[ECCV 2018] DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency
Stars: ✭ 190 (-20.5%)
Mutual labels:  optical-flow

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

This repository contains the TensorFlow implementation of the paper

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss (AAAI 2018)

Simon Meister, Junhwa Hur, and Stefan Roth.

Slides

Download slides from AAAI 2018 talk.

Citation

If you find UnFlow useful in your research, please consider citing:

@inproceedings{Meister:2018:UUL,
  title  = {{UnFlow}: Unsupervised Learning of Optical Flow
            with a Bidirectional Census Loss},
  author = {Simon Meister and Junhwa Hur and Stefan Roth},
  address = {New Orleans, Louisiana},
  booktitle = {AAAI},
  month = feb,
  year = {2018}
}

License

UnFlow is released under the MIT License (refer to the LICENSE file for details).

Unofficial PyTorch code

There is a unofficial third party PyTorch implementation by Simon Niklaus.

Contents

  1. Introduction
  2. Usage
  3. Replicating our Models
  4. Navigating the Code

Introduction

Our paper describes a method to train end-to-end deep networks for dense optical flow without the need for ground truth optical flow.

NOTE (January 2020): There may be some hiccups with more recent versions of tensorflow due to the unstable custom op compilation API used for the correlation operation. If you experience these issues, please try one of the older tensorflow versions (see list of releases, e.g. 1.2 or 1.7) until I find time to fix these issues. I'm currently too busy with new projects to upgrade the code, and would be happy about any contributions.

This implementation supports all training and evaluation styles described in the paper. This includes

The supported network architectures are FlowNetS, FlowNetC, as well as stacked variants of these networks as introduced in FlowNet 2.0.

All datasets required for training and evaluation are downloaded on-demand by the code (except Cityscapes, which has to be downloaded manually if needed). Please ensure that the data directory specified in the configuration file has enough space to hold at least 150 GB (when using SYNTHIA and KITTI only).

Usage

Please refer to the configuration file template (config_template/config.ini) for a detailed description of the different operating modes.

Hardware requirements

  • at least one NVIDIA GPU (multi-GPU training is supported). We used the Titan X Pascal with 12GB memory.
  • for best performance, at least 8GB of GPU memory is recommended, for the stacked variants 11-12GB may be best
  • to run the code with less GPU memory, take a look at https://github.com/openai/gradient-checkpointing. One user reported successfully running the full CSS model with a GTX 960 with 4GB memory. Note that training will take longer in that case.

Software requirements

  • python 3
  • gcc4
  • RAR backend tool for rarfile (see https://pypi.python.org/pypi/rarfile/)
  • python packages: matplotlib pypng rarfile pillow and tensorflow-gpu (at least version 1.7)
  • CUDA and CuDNN. You should use the installer downloaded from the NVIDIA website and install to /usr/local/cuda. Please make sure that the versions you install are compatible with the version of tensorflow-gpu you are using (see e.g. https://github.com/tensorflow/tensorflow/releases).

Prepare environment

  • copy config_template/config.ini to ./ and modify settings in the [dir], [run] and [compile] sections for your environment (see comments in the file).

Run & evaluate experiments

  • adapt settings in ./config.ini for your experiment
  • cd src
  • train with python run.py --ex my_experiment. Evaluation is run during training as specified in config.ini, but no visualizations are saved.
  • evaluate (multiple) experiments with python eval_gui.py --ex experiment_name_1[, experiment_name_2 [...]] and visually compare results side-by-side.

You can use the --help flag with run.py or eval_gui.py to view all available flags.

View tensorboard logs

  • view logs for all experiments with tensorboard --logdir=<log_dir>/ex

Pre-trained models

We provide checkpoints for the C_SYNTHIA, CS_SYNTHIA, CSS_SYNTHIA, C, CS, CSS and CSS_ft models (see "Replicating our models" for a description). To use them,

  • download this file and extract the contents to <log_dir>/ex/.

Now, you can evaluate and compare different models, e.g.

  • python eval_gui.py --ex C_SYNTHIA,C,CSS_ft.

Replicating our models

In the following, each list item gives an experiment name and parameters to set in ./config.ini. For each experiment, first modify the configuration parameters as specified, and then run python run.py --ex experiment_name.

First, create a series of experiments for the models pre-trained on SYNTHIA:

  • C_SYNTHIA: dataset = synthia, flownet = C,
  • CS_SYNTHIA: dataset = synthia, flownet = CS, finetune = C_SYNTHIA,
  • CSS_SYNTHIA: dataset = synthia, flownet = CSS, finetune = C_SYNTHIA,CS_SYNTHIA.

Next, create a series of experiments for the models trained on KITTI raw:

  • C: dataset = kitti, flownet = C, finetune = C_SYNTHIA,
  • CS: dataset = kitti, flownet = CS, finetune = C,CS_SYNTHIA,
  • CSS: dataset = kitti, flownet = CSS, finetune = C,CS,CSS_SYNTHIA.

Then, train the final fine-tuned model on KITTI 2012 / 2015:

  • CSS_ft: dataset = kitti_ft, flownet = CSS, finetune = C,CS,CSS, train_all = True. Please monitor the eval logs and stop the training as soon as the validation error starts to increase (for CSS, it should be at about 70K iterations).

If you want to train on Cityscapes:

  • C_Cityscapes: dataset = cityscapes, flownet = C, finetune = C_SYNTHIA.

Note that all models but CSS_ft were trained without ground truth optical flow, using our unsupervised proxy loss only.

Navigating the code

The core implementation of our method resides in src/e2eflow/core. The key files are:

  • losses.py: Proxy losses for unsupervised training,
  • flownet.py: FlowNet architectures with support for stacking,
  • image_warp.py: Backward-warping via differentiable bilinear image sampling,
  • supervised.py: Supervised loss and network computation,
  • unsupervised.py: Unsupervised (bi-directional) multi-resolution network and loss computation,
  • train.py: Implementation of training with online evaluation,
  • augment.py: Geometric and photometric data augmentations.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].