Domain Adaptation (Pytorch-Lightning)
This repository contains unofficial pytorch (pytorch-lightning) version source code introduced by domain adaptation papers:
- DANN (2015) [paper, repo]
- CDAN (2017) [paper, repo]
- MSTN (2018) [paper, repo]
- BSP (2019) [paper, repo]
- DSBN (2019) [paper, repo]
- RSDA-MSTN (2020) [paper, repo]
- SHOT (2020) [paper, repo]
- TransDA (2021) [paper, repo]
- FixBi (2021) [paper ,repo]
Tutorial (use standard dataset: Office-31)
-
Clone repository and install dependency
git clone https://github.com/hankyul2/DomainAdaptation.git pip3 install -r requirements.txt
-
Train Model (If you don't use neptune, just remove it from here). Check configuration in configs
python3 main.py fit --config=configs/cdan_e.yaml -d 'amazon_webcam' -g '0,'
Closed world Benchmark Result
Office-31 Dataset Benchmark reported in original papers
| A>D | A>W | D>A | D>W | W>A | W>D | Avg | |
|---|---|---|---|---|---|---|---|
| source only | 80.8 | 76.9 | 60.3 | 95.3 | 63.6 | 98.7 | 79.3 |
| DANN (2015) | 79.7 | 82.0 | 68.2 | 96.9 | 67.4 | 99.1 | 82.2 |
| CDAN (2017) | 89.8 | 93.1 | 70.1 | 98.2 | 68.0 | 100.0 | 86.6 |
| CDAN+E (2017) | 92.9 | 94.1 | 71.0 | 98.6 | 69.3 | 100.0 | 87.7 |
| MSTN (2018) | 90.4 | 91.3 | 72.7 | 98.9 | 65.6 | 100.0 | 86.5 |
| BSP+DANN (2019) | 90.0 | 93.0 | 71.9 | 98.0 | 73.0 | 100.0 | 87.7 |
| BSP+CDAN+E (2019) | 93.0 | 93.3 | 73.6 | 98.2 | 72.6 | 100.0 | 88.5 |
| DSBN+MSTN (2019) | 90.8 | 93.3 | 72.7 | 99.1 | 73.9 | 100.0 | 88.3 |
| RSDA+MSTN (2020) | 95.8 | 96.1 | 77.4 | 99.3 | 78.9 | 100.0 | 91.1 |
| SHOT (2020) | 94.0 | 90.1 | 74.7 | 98.4 | 74.3 | 99.9 | 88.6 |
| TransDA (2021) | 97.2 | 95.0 | 73.7 | 99.3 | 79.3 | 99.6 | 90.7 |
| FixBi (2021) | 95.0 | 96.1 | 78.7 | 99.3 | 79.4 | 100.0 | 91.4 |
In this work
| A>D [tf.dev] |
A>W [tf.dev] |
D>A [tf.dev] |
D>W [tf.dev] |
W>A [tf.dev] |
W>D [tf.dev] |
Avg | |
|---|---|---|---|---|---|---|---|
| source only [code, config] |
82.3 weight |
77.9 | 63.0 weight |
94.5 | 64.7 weight |
98.3 | 80.1 |
| source only ViT [code, config] |
88.0 weight |
87.9 | 76.7 weight |
97.7 | 77.1 weight |
99.7 | 87.8 |
| DANN (2015) [code, config] |
87.2 weight |
90.4 weight |
70.6 weight |
97.8 weight |
73.7 weight |
99.7 weight |
86.6 |
| CDAN (2017) [code, config] |
92.4 | 95.1 | 75.8 | 98.6 | 74.4 | 99.9 | 89.4 |
| CDAN+E (2017) [code, config] |
93.2 | 95.6 | 75.1 | 98.7 | 75.0 | 100.0 | 89.6 |
| MSTN (2018) [code, config] |
89.0 | 92.7 | 71.4 | 97.9 | 74.1 | 99.9 | 87.5 |
| BSP+DANN (2019) [code, config] |
86.3 | 89.1 | 71.4 | 97.7 | 73.4 | 100.0 | 86.3 |
| BSP+CDAN+E (2019) [code, config] |
92.6 | 94.7 | 73.8 | 98.7 | 74.7 | 100.0 | 89.1 |
| DSBN+MSTN Stage1 (2019) [code, config] |
87.8 | 92.3 | 72.2 | 98.0 | 73.2 | 99.9 | 87.2 |
| DSBN+MSTN Stage1 (2019) [code, config] |
90.6 | 93.5 | 74.0 | 98.0 | 73.1 | 99.5 | 88.1 |
| RSDA+MSTN (2020) [Not Implemented] |
- | - | - | - | - | - | - |
| SHOT (2020) [code, config] |
93.2 | 92.5 | 74.3 | 98.2 | 75.9 | 100.0 | 89.0 |
| SHOT (CDAN+E) (2020) [code, config] |
93.2 | 95.7 | 77.7 | 98.9 | 76.0 | 100.0 | 90.2 |
| MIXUP (CDAN+E) (2021) [code, config] |
92.9 | 96.1 | 76.2 | 98.9 | 77.7 | 100.0 | 90.3 |
| TransDA (2021) [code, config] |
94.4 | 95.8 | 82.3 | 99.2 | 82.0 | 99.8 | 92.3 |
| FixBi (2021) [code, config] |
90.8 | 95.7 | 72.6 | 98.7 | 74.8 | 100.0 | 88.8 |
Note
- Reported scores are from SHOT, FixBi paper
- Evaluation datasets are:
valid=test=target. For me, this looks weird, but there are no other way to reproduce results in paper. But, source only model's evaluation is a bit different:valid=source,test=target - In this works, scores are 3 times averaged scores.
- If you want to use pretrained model weight, you should add loading pretrained model weights.
- Optimizer and learning rate scheduler are same to all model(SGD) except
mstn,dsbn+mstn(Adam) SHOTcan results lower accuracy than reported scores. To reproduce reported score, I recommend you to use provided source only model weights. I don't know why...BSP,DSBN+MSTN,FixBi: Fails to reproduce scores reported in paper
Experiment
Combination of several methods or models. (No weights and tf.dev)
| A>D | A>W | D>A | D>W | W>A | W>D | Avg | |
|---|---|---|---|---|---|---|---|
| DANN ViT (2015) [code, config] |
90.0 | 91.0 | 79.0 | 98.9 | 78.8 | 99.9 | 89.6 |
| CDAN ViT (2017) [code, config] |
94.7 | 96.3 | 80.0 | 98.9 | 80.4 | 100.0 | 91.7 |
| CDAN+E ViT (2017) [code, config] |
97.1 | 96.7 | 80.1 | 99.2 | 79.8 | 99.9 | 92.1 |
| SHOT (CDAN+E) (2020) [code, config] |
93.2 | 95.7 | 77.7 | 98.9 | 76.0 | 100.0 | 90.2 |
| MIXUP (CDAN+E) (2021) [code, config] |
92.9 | 96.1 | 76.2 | 98.9 | 77.7 | 100.0 | 90.3 |
Future Updates
- Add weight parameter
- Add ViT results
- Check Fixbi code
- Add office-home dataset results
- Add digits results
Some Notations
- We use
pytorch-lightningin the code. So if you are unfamiliar withpytorch-lightning, I recommend you to read quick-start of pytorch-lightning. (quick start is enough to read this code) - To avoid duplication of code, we use class inheritance and just add changes proposed in papers. We try to keep code simple and nice to read. So if you think code is difficult to read, please leave it as issue or PR.
- Only 8 papers are now implemented. If there are any request for certain paper, we will try to re-implement it.
- There are some problems in backbone code. (I could not find where) So performance can be lower than reported table. I recommend to use standard library model. (timm, torchvision, etc)