image-registration-resources

image registration related books, papers, videos, and toolboxes

Image registration is the process of transforming different sets of data into one coordinate system. Data may be multiple photographs, and from different sensors, times, depths, or viewpoints.

It is used in computer vision, medical imaging, military automatic target recognition, compiling and analyzing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from different measurements.

[toc]

Paper Lists

Many thanks to Yochengliu awesome-point-cloud-analysis , hoya012 deep_learning_object_detection and Amusi awesome-object-detection

A paper list of image registration.

Keywords

medi.: medical image | nat.: natural image | rs.: remote sensing | pc.: point cloud

data.: dataset  |   dep.: deep learning

oth.: other, including correspondence, mapping, matching, alignment...

Statistics: 🔥 code is available & stars >= 100  |  ⭐ citation >= 50

Update log

Last updated: 2020/04/20

2020/04/20 - update all of recent papers (2017-2020) about point cloud registration and make some diagram about history of image registration.

2020

[CVPR] Learning multiview 3D point cloud registration. [code] [pc.]

2019

[CVPR] DeepMapping: Unsupervised Map Estimation From Multiple Point Clouds. [code] [pc.]

[CVPR] PointNetLK: Point Cloud Registration using PointNet. [pytorch] [pc.]

[CVPR] SDRSAC: Semidefinite-Based Randomized Approach for Robust Point Cloud Registration without Correspondences. [matlab] [pc.]

[CVPR] FilterReg: Robust and Efficient Probabilistic Point-Set Registration using Gaussian Filter and Twist Parameterization. [code] [pc.]

[CVPR] PointNetLK: Robust & Efficient Point Cloud Registration using PointNet. [pytorch] [pc.]

[CVPR] 3D Local Features for Direct Pairwise Registration. [pc.]

[ICCV] DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration. [pc.]

[ICCV] Accelerated Gravitational Point Set Alignment with Altered Physical Laws. [pc.]

[ICCV] Deep Closest Point: Learning Representations for Point Cloud Registration. [pc.]

[ICCV] Efficient Learning on Point Clouds with Basis Point Sets. [code] [ pc.]

[ICCV] Robust Variational Bayesian Point Set Registration. [pc.]

[ICRA] Robust low-overlap 3-D point cloud registration for outlier rejection. [matlab] [pc.]

[ICRA] Robust Generalized Point Set Registration Using Inhomogeneous Hybrid Mixture Models Via Expectation. [pc.]

[ICRA] CELLO-3D: Estimating the Covariance of ICP in the Real World. [pc.]

2018

[3DV] PCN: Point Completion Network. [tensorflow] [pc. ] 🔥

[CVPR] Density Adaptive Point Set Registration. [code] [pc.]

[CVPR] Inverse Composition Discriminative Optimization for Point Cloud Registration. [pc.]

[ECCV] Learning and Matching Multi-View Descriptors for Registration of Point Clouds. [pc.]

[ECCV] 3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration. [tensorflow] [pc.]

[ECCV] Efficient Global Point Cloud Registration by Matching Rotation Invariant Features Through Translation Search. [pc.]

[ECCV] HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration. [pc.]

[ICRA] Robust Generalized Point Cloud Registration Using Hybrid Mixture Model. [pc.]

[ICRA] A General Framework for Flexible Multi-Cue Photometric Point Cloud Registration. [pc.]

[IROS] Dynamic Scaling Factors of Covariances for Accurate 3D Normal Distributions Transform Registration. [pc.]

[IROS] Robust Generalized Point Cloud Registration with Expectation Maximization Considering Anisotropic Positional Uncertainties. [pc.]

[IROS] PCAOT: A Manhattan Point Cloud Registration Method Towards Large Rotation and Small Overlap. [pc.]

2017

[CVPR] 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions. [code] [pc. data. ] 🔥 ⭐

[CVPR] Discriminative Optimization: Theory and Applications to Point Cloud Registration. [pc.]

[CVPR] 3D Point Cloud Registration for Localization using a Deep Neural Network Auto-Encoder. [git] [pc.]

[ICCV] Colored Point Cloud Registration Revisited. [pc.]

[ICRA] Using 2 point+normal sets for fast registration of point clouds with small overlap. [pc.]

Learning Resources

Many thanks to yzhao062 Anomaly Detection Learning Resources. I followed his style to collect resources

This resources collect:

• Books & Academic Papers
• Datasets
• Open-source and Commercial Libraries/Toolkits
• On-line Courses and Videos
• Key Conferences & Journals

1. Papers

1.1. Overview & Survey Papers

Medical Image

[1] A. Sotiras, et.al., “Deformable medical image registration: A survey,” 2013.

[2] N. J. Tustison, et.al., “Learning image-based spatial transformations via convolutional neural networks : A review,” 2019.

[3] G. Haskins,et.al. “Deep Learning in Medical Image Registration: A Survey,” 2019.

[4] N. Tustison, et.al., “Learning image-based spatial transformations via convolutional neural networks: A review,”2019.

1.2. Key Algorithms

2. Datasets & Competitions

2.1. Datasets

Medical Image

Natural image

Indoor LiDAR-RGBD Scan Dataset

ETH3D SLAM & Stereo Benchmarks

EuRoC MAV Dataset

ViViD : Vision for Visibility Dataset

Apolloscape: Scene Parsing

KITTI Visual Odometry dataset

NCLT Dataset

Oxford Robotcar Dataset

Remote Sensing

ISPRS Benchmarks

HPatches: The HPatches dataset was used as the basis for the local descriptor evaluation challenge that was presented in the Local Features: State of the Art, Open Problems and Performance Evaluation workshop during ECCV 2016.

The Zurich Urban Micro Aerial Vehicle Dataset

Zurich Summer Dataset

Inria Aerial Image Labeling DataSet

LANDSAT

NWPU-RESISC45

DOTA

MUUFLGulfport

Point Cloud

The Stanford 3D Scanning Repository（斯坦福大学的3d扫描存储库）

http://graphics.stanford.edu/data/3Dscanrep/

这应该是做点云数据最初大家用最多的数据集，其中包含最开始做配准的Bunny、Happy Buddha、Dragon等模型。

[Stanford 3D] The Stanford 3D Scanning Repository. [pc.]

Shapenet

ShapeNet是一个丰富标注的大规模点云数据集，其中包含了55中常见的物品类别和513000个三维模型。

The KITTI Vision Benchmark Suite

链接：http://www.cvlibs.net/datasets/kitti/

这个数据集来自德国卡尔斯鲁厄理工学院的一个项目，其中包含了利用KIT的无人车平台采集的大量城市环境的点云数据集（KITTI），这个数据集不仅有雷达、图像、GPS、INS的数据，而且有经过人工标记的分割跟踪结果，可以用来客观的评价大范围三维建模和精细分类的效果和性能。

Robotic 3D Scan Repository

链接：http://kos.informatik.uni-osnabrueck.de/3Dscans/

这个数据集比较适合做SLAM研究，包含了大量的 Riegl 和 Velodyne 雷达数据

佐治亚理工大型几何模型数据集

链接：https://www.cc.gatech.edu/projects/large_models/

PASCAL3D+

链接：http://cvgl.stanford.edu/projects/pascal3d.html

包含了12中刚体分类，每一类超过了3000个实例。并且包含了对应的imageNet中每一类的图像。

其他总结

链接：https://github.com/timzhang642/3D-Machine-Learning

Other

awesome-point-cloud-analysis

[UWA Dataset] . [pc.]

[ASL Datasets Repository(ETH)] This site is dedicated to provide datasets for the Robotics community with the aim to facilitate result evaluations and comparisons. [ pc. ]

[3D Match] Keypoint Matching Benchmark, Geometric Registration Benchmark, RGB-D Reconstruction Datasets. [pc. ]

2.2. Competitions

All Challenges

2019

CuRIOUS:2019 | Official solution

1 Register pre-operative MRI to iUS before tumor resection
2 Register iUS after tumor resection to iUS before tumor resection

ANHIR:2019 | Official solution

IEEE International Symposium on Biomedical Imaging (ISBI) 2019
High-resolution (up to 40x magnification) whole-slide images of tissues (lesions, lung-lobes, mammary-glands) were acquired - the original size of our images is up to 100k x 200k pixels. The acquired images are organized in sets of consecutive sections where each slice was stained with a different dye and any two images within a set can be meaningfully registered.

2018

iChallenges

Continuous Registration Challenge

Multi-shell Diffusion MRI Harmonisation Challenge 2018 (MUSHAC)

2010

EMPIRE10

3. Toolbox

Natural image

[C++] [Python] OpenCV: OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.

[C++] PCL: Point Cloud Library. The Point Cloud Library (PCL) is a standalone, large scale, open project for 2D/3D image and point cloud processing.

[C++] Ceres Solver: Ceres Solver is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems.

[C++] Open3D: Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization.

Medical Image

[c++] ITK: Segmentation & Registration Toolkit

An open-source, cross-platform system that provides developers with an extensive suite of software tools for image analysis. Developed through extreme programming methodologies. ITK employs leading-edge algorithms for registering and segmenting multidimensional data.

[c++] [Python] [Java] SimpleITK: a simplified layer built on top of ITK.

[c++] ANTs: Advanced Normalization Tools.

Image registration with variable transformations (elastic, diffeomorphic, diffeomorphisms, unbiased) and similarity metrics (landmarks, cross-correlation, mutual information, etc.). Image segmentation with priors & nonparametric, multivariate models.

[c++] Elastix: open source software, based on the well-known ITK .

The software consists of a collection of algorithms that are commonly used to solve (medical) image registration problems. [manual]

[C++] [Python] [Java] [R] [Ruby] [Lua] [Tcl] [C#] SimpleElastix: a medical image registration library that makes state-of-the-art image registration really easy to do in languages like Python, Java and R.

3D slicer : an open source software platform for medical image informatics, image processing, and three-dimensional visualization. Built over two decades through support from the National Institutes of Health and a worldwide developer community, Slicer brings free, powerful cross-platform processing tools to physicians, researchers, and the general public.

Github repository for deep learning medical image registration:

[Keras] VoxelMorph

[Keras] FAIM

[Tensorflow] Weakly-supervised CNN

[Tensorflow] RegNet3D

[Tensorflow] Recursive-Cascaded-Networks

[Pytorch] Probabilistic Dense Displacement Network

[Pytorch] Linear and Deformable Image Registration

[Pytorch] Inverse-Consistent Deep Networks

[Pytorch] Non-parametric image registration

[Pytorch] One Shot Deformable Medical Image Registration

[Pytorch] Image-and-Spatial Transformer Networks

Remote Sensing

[C++] OTB: Orfeo ToolBox (OTB) is an open-source project for state-of-the-art remote sensing. Built on the shoulders of the open-source geospatial community, it can process high resolution optical, multispectral and radar images at the terabyte scale. A wide variety of applications are available: from ortho-rectification or pansharpening, all the way to classification, SAR processing, and much more!

[C++] [Python] OpenCV: OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code.

[C++] ITK: Insight Toolkit (ITK) an open-source, cross-platform system that provides developers with an extensive suite of software tools for image analysis. Developed through extreme programming methodologies, ITK employs leading-edge algorithms for registering and segmenting multidimensional data.

[Python] Spectral Python (SPy): Spectral Python (SPy) is a pure Python module for processing hyperspectral image data (imaging spectroscopy data). It has functions for reading, displaying, manipulating, and classifying hyperspectral imagery.

Post Processing Tools

[C++] enblend: Enblend blends away the seams in a panoramic image mosaic using a multi-resolution spline. Enfuse merges different exposures of the same scene to produce an image that looks much like a tone-mapped image.

[C++] maxflow: An implementation of the maxflow algorithm which can be used to detect the optimal seamline.

[C++] [Matlab] gco-v3.0: Multi-label optimization library by Olga Veksler and Andrew Delong.

Source Code

APAP

AANAP

NISwGSP

SPHP

Parallax-tolerant image stitching

Point Cloud

MeshLab

简介：是一款开源、可移植和可扩展的三维几何处理系统。主要用于处理和编辑3D三角网格，它提供了一组用于编辑、清理、修复、检查、渲染、纹理化和转换网格的工具。提供了处理由3D数字化工具/设备生成的原始数据以及3D打印功能，功能全面而且丰富。MeshLab支持多数市面上常见的操作系统，包括Windows、Linux及Mac OS X，支持输入/输出的文件格式有：STL 、OBJ 、 VRML2.0、U3D、X3D、COLLADA MeshLab可用于各种学术和研究环境，如微生物学、文化遗产及表面重建等。

ICP开源库

SLAM6D

Libicp

libpointmatcher

g-icp

n-icp

4. Books & Tutorials

4.1. Books

Natural image

Multiple view geometry in computer vision by Richard Hartley and Andrew Zisserman, 2004: Mathematic and geometric basis for 2D-2D and 2D-3D registration. A must-read for people in the field of registration. E-book

Computer Vision: A Modern Approach by David A. Forsyth, Jean Ponce: for upper-division undergraduate- and graduate-level courses in computer vision found in departments of Computer Science, Computer Engineering and Electrical Engineering.

Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Engineering by Jean Gallier and Jocelyn Quaintance. The latest book from upenn about the algebra and optimization theory.

Three-Dimensional Computer vision-A Geometric Viewpoint Classical 3D computer vision textbook.

An invitation to 3D vision a self-contained introduction to the geometry of three-dimensional (3-D) vision.

Medical Image

Zhenhuan Zhou, et.al: A software guide for medical image segmentation and registration algorithm. 医学图像分割与配准(ITK实现分册) Part Ⅱ introduces the most basic network and architecture of medical registration algorithms (Chinese Version).

2-D and 3-D Image Registration for Medical, Remote Sensing, and Industrial Applications by A. Ardeshir Goshtasby

医学图像配准技术与应用 by 吕晓琪

Intensity-based 2D-3D Medical Image Registration by Russakoff, Daniel

Biomedical Image Registration by Fischer, Dawant, Lorenz

Medical Image Registration by Hajnal, Joseph V.

Deep Learning for Medical Image Analysis (part IV)

Point Cloud

14 lectures on visual SLAM By Xiang Gao and Tao Zhang and Yi Liu and Qinrui Yan. 视觉SLAM十四讲 视觉配准方向较易懂的入门教材。通俗讲述视觉匹配的物理模型， 数学几何基础，优化过程等。 新手必读。 [github] [Videos]

点云数据配准及曲面细分技术 by 薛耀红, 赵建平, 蒋振刚, 等 书籍内容比较过时，仅适合零基础读者阅读。推荐自行查找相关博客学习。

Remote Sensing

Image Registration for Remote Sensing

2-D and 3-D Image Registration: For Medical, Remote Sensing, and Industrial Applications by A. A. Goshtasby, 2005.

航空遥感图像配准技术

基于特征的光学与SAR遥感图像配准

基于特征的航空遥感图像配准及部件检测技术

Introduction to Remote Sensing

Remote Sensing and Image Interpretation

Remote Sensing: Models and Methods for Image Processing

4.2. Tutorials

Natural image

Medical Image

• Medical Image Registration

• [MICCAI2019] learn2reg PDF

Big thanks to Yipeng Hu organizing the excellent tutorial.

Description:

Medical image registration has been a cornerstone in the research fields of medical image computing and computer assisted intervention, responsible for many clinical applications. Whilst machine learning methods have long been important in developing pairwise algorithms, recently proposed deep-learning-based frameworks directly infer displacement fields without iterative optimization for unseen image pairs, using neural networks trained from large population data. These novel approaches promise to tackle several most challenging aspects previously faced by classical pairwise methods, such as high computational cost, robustness for generalization and lack of inter-modality similarity measures.

Output from several international research groups working in this area include award-winning conference presentations, high-impact journal publications, well-received open-source implementations and industrial-partnered translational projects, generating significant interests to all levels of world-wide researchers. Accessing to the experience and expertise in this inherently multidisciplinary topic can be beneficial to many in our community, especially for the next generation of young scientists, engineers and clinicians who often have only been exposed to a subset of these methodologies and applications.

We organize a tutorial including both theoretical and practical sessions, inviting expert lectures and tutoring coding for real-world examples. Three hands-on sessions guiding participants to understand and implement published algorithms using clinical imaging data. This aims to provide an opportunity for the participants to bridge the gap between expertises in medical image registration and deep learning, as well as to start a forum to discuss know-hows, challenges and future opportunities in this area.

• [kaggle:2016] Image registration, the R way, (almost) from scratch

There are some packages in R for image manipulation and after some test I select “imager” , based on the CImg C++, fast and providing several image processing tools.

• [kaggle:2018] X-Ray Patient Scan Registration

SimpleITK, ITK, scipy, OpenCV, Tensorflow and PyTorch all offer tools for registering images, we explore a few here to see how well they work when applied to the fairly tricky problem of registering from the same person at different time and disease points.

• [MICCAI2019] Autograd Image Registration Laboratory

• [MIT] HST.582J Biomedical Signal and Image Processing PDF

Remote Sensing

• Image Alignment and Stitching: A Tutorial

• Image Registration for Remote Sensing

• Image Stitching

• The Remote Sensing Tutorial 1

• The Remote Sensing Tutorial 2

Point Cloud

• 点云配准算法说明与流程介绍

• 点云配准算法介绍与比较

• 机器学习方法处理三维点云

• 一个例子详细介绍点云配准的过程

4.3. Blogs

图像配准指北

图像配准综述

基于深度学习的医学图像配准综述

基于深度学习和图像引导的医学图像配准

图像配准：从SIFT到深度学习

点云配准综述

图像配准会议介绍@ MICCAI2019 / CVPR2019 / ICCV2019 / NeurIPS2019

Image Registration: From SIFT to Deep Learning

点云配准

点云配准算法的说明与流程介绍

几种点云配准算法的方法的介绍与比较

三维点云用机器学习的方法进行处理

一个例子详细介绍了点云配准的过程

5. Courses/Seminars/Videos

Courses

16-822: Geometry-based Methods in Vision

[VALSE 2018] Talk: 2017以来的2D to 3D by 吴毅红

Workshops

WBIR - International Workshop on Biomedical Image Registration

WBIR2018, Leiden, Netherlands WBIR2016, Las Vegas NV WBIR2014, London, UK

Seminars

Videos

• Definition and Introduction to Image Registration Pre Processing Overview

• 仿射变换与图像配准（科普性视频， 比较简陋）

Remote Sensing

• Registration of images of different modalities in Remote Sensing

6. Key Conferences/Workshops/Journals

6.1. Conferences & Workshops

CVPR: IEEE International Conference on Computer Vision and Pattern Recognition

ICCV: IEEE International Conference on Computer Vision

ECCV: European Conference on Computer Vision

NeurIPS: Conference on Neural Information Processing Systems

AAAI: Association for the Advancement of Artificial Intelligence

ICML: International Conference on Machine Learning

ICPR: International Conference on Pattern Recognition

IJCNN: International Joint Conference on Neural Networks

ICIP: IEEE International Conference on Image Processing

IJCAI: International Joint Conferences on Artificial Intelligence

ICRA: IEEE International Conference on Robotics and Automation

International Conference on 3D Vision

WACV: Winter Conference on Applications of Computer Vision

Biomedical image

MICCAI: International Conference on Medical Image Computing and Computer Assisted Intervention

IPMI: Information Processing in Medical Imaging

ISBI: International Symposium on Biomedical Imaging

Medical Imaging SPIE

Remote Sensing

ISPRS-2020

Point Cloud

点云配准主要应用于工业制造业的逆向工程、古文物修复、医学三维图像构建等领域。研究内容是属于计算机视觉领域的研究范畴。国际上的会议如计算机视觉三大顶会ICCV、CVPR、ECCV等都会有相关技术，除此之外，还有ACCV、BMVC、SSVM等认可度也比较高。

6.2. Journals

IEEE Transactions on Pattern Analysis and Machine Intelligence

International Journal of Computer Vision

ISPRS Journal of Photogrammetry and Remote Sensing

Biomedical image

TMI: IEEE Transactions on Medical Imaging

MIA: Medical Image Analysis

TIP: IEEE Transactions on Image Processing

TBME: IEEE Transactions on Biomedical Engineering

BOE: Biomedical Optics Express

JHBHI: Journal of Biomedical and Health Informatics

Remote Sensing

Remote Sensing of Environment

ISPRS Journal of Photogrammetry And Remote Sensing

IEEE Transactions on Geoscience And Remote Sensing

International Journal of Applied Earth Observation and Geoinformation

IEEE Geoscience and Remote Sensing Letters

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Remote sensing

GIScience & Remote Sensing

Photogrammetric engineering and remote sensing

International journal of remote sensing

Remote Sensing Letters

Journal of Applied Remote Sensing

Point Cloud

IEEE旗下的TPAMI，TIP等，还有SIAM Journal Image Sciences，Springer那边有IJCV

How to contact us

We have QQ Group 【配准萌新交流群】 （群号 869211738）

and Wechat Group 【配准交流群】（已满员） for comunications.

More items will be added to the repository. Please feel free to suggest other key resources by opening an issue report, submitting a pull request, or dropping me an email @ ([email protected]). Enjoy reading!