Deep Transfer Learning

Contributed by Ningyu Zhang.

When Transfer Learning Meets Deep Learning

Survey

• Domain Adaptation for Visual Applications: A Comprehensive Survey

New Trends

• GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations

• Learning to Transfer (ICML-18)

• Label Efficient Learning of Transferable Representations across Domains and Tasks (NIPS-17)

Unbalanced Transfer

• Re-weighted Adversarial Adaptation Network for Unsupervised Domain Adaptation (CVPR-18)

• Deep Asymmetric Transfer Network for Unbalanced Domain Adaptation (AAAI-18)

• Importance Weighted Adversarial Nets for Partial Domain Adaptation (CVPR-18)

• Partial Transfer Learning with Selective Adversarial Networks (CVPR-18)

• Partial Adversarial Domain Adaptation (ECCV-18)

Distilling Aproach

• Distilling the Knowledge in a Neural Network (NIPS-14)

• Born Again Neural Networks (ICML-18)

• NITE : A Neural Inductive Teaching Framework for Domain-Specific NER (EMNLP-17)

Target Data: labelled, Source Data: labelled

• DAN/JAN (Deep Adaptation Network/Joint Adaptation Network, ICML-15,17)
  • Caffe
• Learning Multiple Tasks with Multilinear Relationship Networks (NIPS-17)
  • Pytorch
• Multi-Adversarial Domain Adaptation (AAAI-18)
  • Caffe
• Partial Transfer Learning with Selective Adversarial Networks (arXiv-17)
  • Caffe
• Gradient Episodic Memory for Continual Learning (NIPS-17)
  • Pytorch
• Unified deep supervised domain adaptation and generalization (ICCV-17)
  • Keras
• Semi-supervised learning knowledge transfer for deep learning from private training data (ICLR-17)
  • Tensorflow
• Net2Net: Accelerating Learning via Knowledge Transfer(ICLR-16)
  • Tensorflow|Torch|Numpy

Evolution based

• Overcoming Catastrophic Forgetting in Neural Networks (PNAS-17)
  • Tensorflow
• Progressive Neural Networks (arXiv-16)
  • Tensorflow
• Evolution Channels Gradient Descent in Super Neural Networks (arXiv-17)
  • Tensorflow
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks (arxiv-17)
  • Tensorflow|Pytorch

One-shot learning

• One-shot Learning with Memory-Augmented Neural Networks (arXiv-16)
  • Theano
• Siamese Neural Networks for One-Shot Image Recognition (ICML-15)
  • Keras
• Learning to Compare: Relation Network for Few-Shot Learning (arXiv-17)
  • Pytorch

Target Data: labelled, Source Data: unlabelled

Self Taught Learning

Target Data: unlabelled, Source Data: labelled

• RTN (Unsupervised Domain Adaptation with Residual Transfer Networks, NIPS-16)
  • Caffe
• Associative Domain Adaptation (ICCV-17)
  • Tensorflow
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation (ECCV-16)
  • PyTorch|Matlab
• Domain Separation Networks (NIPS-16)
  • Tensorflow
• Deep Hashing Network for Unsupervised Domain Adaptation (CVPR-17)
  • Matlab
• Deep Transfer Network: Unsupervised Domain Adaptation (arXiv-16)
  • Theano
• Joint distribution optimal transportation for domain adaptation (NIPS-17)
  • Sklearn
• When Unsupervised Domain Adaptation Meets Tensor Representations (ICCV-17)
  • Matlab
• Self-ensembling for visual domain adaptation (ICLR-18)
  • Theano
• AutoDIAL: Automatic DomaIn Alignment Layers (ICCV-17)
  • Caffe
• Minimal-Entropy Correlation Alignment for Unsupervised Deep Domain Adaptation (ICLR-18)
  • Tensorflow
• Asymmetric Tri-training for Unsupervised Domain Adaptation (arXiv-17)
  • Pytorch

GAN based

• Learning Semantic Representations for Unsupervised Domain Adaptation (ICML-18)

• Unsupervised Domain Adaptation by Backpropagation (ICML-15)

  • Caffe(from author)|Tensorflow(third party)

• Domain-Adversarial Training of Neural Networks (JMLR-16)

  • Tensorflow

• Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (CVPR-17)

  • Tensorflow

• ADDA (Adversarial Discriminative Domain Adaptation, arXiv-17)

  • Tensorflow

• Coupled Generative Adversarial Networks (NIPS-16)

  • Tensorflow|caffe|Pytorch

• Wasserstein Distance Guided Representation Learning for Domain Adaptation (AAAI-18)

  • Pytorch

• Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery (CVPR-18)

  • Pytorch

RNN based

• Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks (ICLR-17)
  • Theano

Zero-shot learning

Target Data: unlabelled, Source Data: unlabelled

Self Taught Clustering

• Self-Taught Convolutional Neural Networks for Short Text Clustering (Neural Networks-17)
  • Matlab

Concept Drift

• Dynamic Weighted Majority for Incremental Learning of Imbalanced Data Streams with Concept Drift (IJCAI-17)
  • Matlab

Transferability Analysis

• How transferable are features in deep neural networks?

Transfer Learning for NLP

• Adversarial Multi-task Learning for Text Classification (ACL-17)

• Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks (IJCAI-18)

• Cross-Domain Sentiment Classification with Target Domain Specific Information (ACL-18)

• Transfer Learning for Context-Aware Question Matching in Information-seeking Conversations in E-commerce (ACL-18)

• Universal Language Model Fine-tuning for Text Classificatione (ACL-18)

• Improving Language Understanding by Generative Pre-Training

• Universal Sentence Encoder

Appliactions

• TransNets: Learning to Transform for Recommendation (RecSys-17)
  • Tensorflow
• Empower Sequence Labeling with Task-Aware Neural Language Model (AAAI-18)
  • Pytorch
• Adversarial Learning For Semi-Supervised Semantic Segmentation (ICLR 2018)
  • Pytorch
• Cross-Domain Sentiment Classification via Topic-Related TrAdaBoost (AAAI-17)
  • Keras