hellochick / Ficm

Status: Archive.

Flow-based Intrinsic Curiosity Module (FICM)

[Paper] [Demo Video] [Reddit Discuss]

This is a TensorFlow-based implementation for our paper "Flow-based Intrinsic Curiosity Module".

FICM is used for evaluating the novelty of observations, it generates intrinsic rewards based on the prediction errors of optical flow estimation since the rapid change part in consecutive frames usually serve as important signals.

Without any external reward, FICM can help RL agent to play SuperMario successfully.

This repo is largely inherited from large-scale-curiosity, and we also borrowed correlation layer from flownet2_tf.

Flow-based Intrinsic Curiosity Module
Hsuan-Kung Yang*, Po-Han Chiang*, Min-Fong Hong, and Chun-Yi Lee (* equal contributions)
ElsaLab, National Tsing Hua University
In IJCAI'20 main track and ICLR'19 TARL workshop

Dependencies

• Python3.5

Installation

pip install -r requirement.txt
pip install git+https://github.com/openai/[email protected]

FICM-C (Optional)

If you want to use FICM-C architecture, you'll need to compile for correlation layer additionally.

cd correlation_layer
make all

Note: You might encounter some errors raised from the source codes of tensorflow, you can easily change the header file of 'cuda_kernel_helper.h', 'cuda_device_functions.h', and 'cuda_launch_config.h'

SuperMario (Optional)

If you want to train an agent to play SuperMario, you need to install retro and import SuperMarioBros-Nes game.

pip install retro

Read the official guide here

Example usage

Mario

python run.py --feat_learning flowS --env_kind mario

Atari

python run.py --feat_learning flowS --env SeaquestNoFrameskip-v4 --seed 666

Citation

If you find this paper or code implementation helpful, please consider to cite:

@inproceedings{flowbasedcuriosity2020,
  title     = {Flow-based Intrinsic Curiosity Module},
  author    = {Hsuan-Kung Yang, Po-Han Chiang, Min-Fong Hong and Chun-Yi Lee},
  booktitle = {Proceedings of the Twenty-Ninth International Joint Conference on
               Artificial Intelligence, {IJCAI-20}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},             
  editor    = {Christian Bessiere}	
  pages     = {2065--2072},
  year      = {2020},
  month     = {7},
  note      = {Main track}
  doi       = {10.24963/ijcai.2020/286},
  url       = {https://doi.org/10.24963/ijcai.2020/286},
}

Reference

@inproceedings{largeScaleCuriosity2018,
    Author = {Burda, Yuri and Edwards, Harri and
              Pathak, Deepak and Storkey, Amos and
              Darrell, Trevor and Efros, Alexei A.},
    Title = {Large-Scale Study of Curiosity-Driven Learning},
    Booktitle = {arXiv:1808.04355},
    Year = {2018}
}