[EMNLP'20 (Long)] SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning

A PyTorch implementation of SSCR

Paper | Slide

Overview

SSCR is an implementation of
"SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning"
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang
in Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020 (Long)

The iterative editor is a conditional generator that modifies an image based on current instruction and editing history. Counterfactual reasoning allows the model to think about the expected, resulting images under various counterfactual instructions. In this way, the editor can consider more diverse instructions than the original data to improve the generalizability, even if under data scarcity. With the help of this cross-task matching, we can not only provide a detailed token-level training signal to train the editor better but also supply training loss for counterfactual reasoning in a selfsupervised scenario.

Requirements

This code is implemented under Python3 and PyTorch.
Following libraries are also required:

• PyTorch >= 1.2
• Visdom

Usage

• Dataset:
  • Follow GeNeVA or download pre-processed here.
• Run Vidsom

visdom

• Train

python -m geneva.inference.train @example_args/iclevr-d-subtract.args

• Test

python -m geneva.inference.test @example_args/iclevr-d-subtract.args

• Trained Models
  • Edit "--load_snapshot" in args

Resources

• GeNeVA
• This project

Citation

@inproceedings{fu2020sscr, 
  author = {Tsu-Jui Fu and Xin Eric Wang and Scott Grafton and Miguel Eckstein and William Yang Wang}, 
  title = {SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning}, 
  booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP)}, 
  year = {2020} 
}