World Models Library

World Models is a platform-agnostic library to facilitate visual based agents for planning. This notebook (run it in colab) shows how to use World Models library and its different components.

To run locally, use the following command:

python3 -m world_models.bin.train_eval  \
    --config_path=/path/to/config  \
    --output_dir=/path/to/output_dir  \
    --logtostderr

Experiment Results

Below is a summary of our findings. For full discussion please see our paper: Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning

Is predicting future rewards sufficient for achieving success in visual model-based reinforcement learning? We experimentally demonstrate that this is usually not the case in the online settings and the key is to predict future images too.

Amazingly, this also means there is a weak correlation between reward prediction accuracy and performance of the agent. However, we show that there is a much stronger correlation between image reconstruction error and the performance of the agent.

We show how this phenomenon is directly related to exploration: models that fit the data better usually perform better in an offline setup. Surprisingly, these are often not the same models that perform the best when learning and exploring from scratch!

How to Cite

If you use this work, please cite the following paper where it was first introduced:

   @article{2020worldmodels,
     title   = {Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning},
     author  = {Mohammad Babaeizadeh and Mohammad Taghi Saffar and Danijar Hafner and Harini Kannan and Chelsea Finn and Sergey Levine and Dumitru Erhan},
     year    = {2020},
     url     = {https://arxiv.org/abs/2012.04603}
   }

You can reach us at [email protected]

Dependencies

• absl
• gin-config
• TensorFlow==1.15
• TensorFlow probability==0.7
• gym
• dm_control
• MuJoCo

Disclaimer: This is not an official Google product.