Awesome Deep Reinforcement Learning

Landscape of Deep RL

This project is built for people who are learning and researching on latest deep reinforcement learning methods.

Content

• General guidances
• Foundations and theory
• General benchmark frameworks
• Value based
• Policy gradient
• Explorations
• Actor-Critic
• Model-based
• Model-free + Model-based
• Hierarchical
• Option
• Connection with other methods
• Connecting value and policy
• Reward design
• Unifying
• Faster DRL
• Multi-agent
• New design
• Multitask
• Observational Learning
• Meta Learning
• Distributional
• Planning
• Safety
• Inverse RL
• No reward RL
• Time
• Applications
• Adversarial learning
• Natural Language
• Generative and contrastive representation learning
• Belief
• PAC
• Applications

Illustrations:

Recommendations and suggestions are welcome.

General guidances

• Multiagent Reinforcement Learning by Marc Lanctot RLSS @ Lille 11 July 2019
• RLDM 2019 Notes by David Abel 11 July 2019
• A Survey of Reinforcement Learning Informed by Natural Language 10 Jun 2019 arxiv
• Challenges of Real-World Reinforcement Learning 29 Apr 2019 arxiv
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning 25 Apr 2019 arxiv
• Principles of Deep RL by David Silver
• University AI's General introduction to deep rl (in Chinese)
• OpenAI's spinningup
• The Promise of Hierarchical Reinforcement Learning 9 Mar 2019
• Deep Reinforcement Learning that Matters 30 Jan 2019 arxiv

Foundations and theory

• General non-linear Bellman equations 9 July 2019 arxiv
• Monte Carlo Gradient Estimation in Machine Learning 25 Jun 2019 arxiv

General benchmark frameworks

• Spriteworld: a flexible, configurable python-based reinforcement learning environment
• Chainerrl Visualizer
• Behaviour Suite for Reinforcement Learning 13 Aug 2019 arxiv | code
• Quantifying Generalization in Reinforcement Learning 20 Dec 2018 arxiv
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning 25 Sept 2018
• dopamine
• StarCraft II
• tfrl
• chainerrl
• PARL

Value based

• Harnessing Structures for Value-Based Planning and Reinforcement Learning 5 Feb 2020 arxiv | code
• Recurrent Value Functions 23 May 2019 arxiv
• Stochastic Lipschitz Q-Learning 24 Apr 2019 arxiv
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning 8 Mar 2018
• DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY 2 Mar 2018
• Rainbow: Combining Improvements in Deep Reinforcement Learning 6 Oct 2017
• Learning from Demonstrations for Real World Reinforcement Learning 12 Apr 2017
• Dueling Network Architecture
• Double DQN
• Prioritized Experience
• Deep Q-Networks

Policy gradient

• Phasic Policy Gradient 9 Sep 2020 arxiv code
• An operator view of policy gradient methods 22 Jun 2020 arxiv
• Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spaces 14 Jun 2019 arxiv
• Policy Gradient Search: Online Planning and Expert Iteration without Search Trees 7 Apr 2019 arxiv
• SUPERVISED POLICY UPDATE FOR DEEP REINFORCEMENT LEARNING 24 Dec 2018 arxiv
• PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation 5 Oct 2018 arxiv
• Clipped Action Policy Gradient 22 June 2018
• Expected Policy Gradients for Reinforcement Learning 10 Jan 2018
• Proximal Policy Optimization Algorithms 20 July 2017
• Emergence of Locomotion Behaviours in Rich Environments 7 July 2017
• Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning 1 Jun 2017
• Equivalence Between Policy Gradients and Soft Q-Learning
• Trust Region Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• Q-PROP: SAMPLE-EFFICIENT POLICY GRADIENT WITH AN OFF-POLICY CRITIC

Explorations

• Self-Supervised Exploration via Disagreement 10 Jun 2019 arxiv
• Approximate Exploration through State Abstraction 24 Jan 2019
• The Uncertainty Bellman Equation and Exploration 15 Sep 2017
• Noisy Networks for Exploration 30 Jun 2017 implementation
• Count-Based Exploration in Feature Space for Reinforcement Learning 25 Jun 2017
• Count-Based Exploration with Neural Density Models 14 Jun 2017
• UCB and InfoGain Exploration via Q-Ensembles 11 Jun 2017
• Minimax Regret Bounds for Reinforcement Learning 16 Mar 2017
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Actor-Critic

• Generalized Off-Policy Actor-Critic 27 Mar 2019
• Soft Actor-Critic Algorithms and Applications 29 Jan 2019
• The Reactor: A Sample-Efficient Actor-Critic Architecture 15 Apr 2017
• SAMPLE EFFICIENT ACTOR-CRITIC WITH EXPERIENCE REPLAY
• REINFORCEMENT LEARNING WITH UNSUPERVISED AUXILIARY TASKS
• Continuous control with deep reinforcement learning

Model-based

• When to use parametric models in reinforcement learning? 12 Jun 2019 arxiv
• Model Based Reinforcement Learning for Atari 5 Mar 2019
• Model-Based Stabilisation of Deep Reinforcement Learning 6 Sep 2018
• Learning model-based planning from scratch 19 July 2017

Model-free + Model-based

• Imagination-Augmented Agents for Deep Reinforcement Learning 19 July 2017

Hierarchical

• WHY DOES HIERARCHY (SOMETIMES) WORK SO WELL IN REINFORCEMENT LEARNING? 23 Sep 2019 arxiv
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning 18 Jun 2019 arxiv

Option

• Variational Option Discovery Algorithms 26 July 2018
• A Laplacian Framework for Option Discovery in Reinforcement Learning 16 Jun 2017

Connection with other methods

• Robust Imitation of Diverse Behaviors
• Learning human behaviors from motion capture by adversarial imitation
• Connecting Generative Adversarial Networks and Actor-Critic Methods

Connecting value and policy methods

• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Policy gradient and Q-learning

Reward design

• End-to-End Robotic Reinforcement Learning without Reward Engineering 16 Apr 2019 arxiv
• Reinforcement Learning with Corrupted Reward Channel 23 May 2017

Unifying

• Multi-step Reinforcement Learning: A Unifying Algorithm

Faster DRL

• Neural Episodic Control

Multi-agent

• No Press Diplomacy: Modeling Multi-Agent Gameplay 4 Sep 2019 arxiv
• Options as responses: Grounding behavioural hierarchies in multi-agent RL 6 Jun 2019 arxiv
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination 18 Jun 2019 arxiv
• A Regularized Opponent Model with Maximum Entropy Objective 17 May 2019 arxiv
• Deep Q-Learning for Nash Equilibria: Nash-DQN 23 Apr 2019 arxiv
• Malthusian Reinforcement Learning 3 Mar 2019 arxiv
• Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning 4 Nov 2018
• INTRINSIC SOCIAL MOTIVATION VIA CAUSAL INFLUENCE IN MULTI-AGENT RL 19 Oct 2018
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning 30 Mar 2018
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning 26 Feb 2018
• The Mechanics of n-Player Differentiable Games 15 Feb 2018
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments 10 Oct 2017
• Learning with Opponent-Learning Awareness 13 Sep 2017
• Counterfactual Multi-Agent Policy Gradients
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments 7 Jun 2017
• Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games 29 Mar 2017

New design

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures 9 Feb 2018
• Reverse Curriculum Generation for Reinforcement Learning
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Learning to Design Games: Strategic Environments in Deep Reinforcement Learning 5 July 2017

Multitask

• Kickstarting Deep Reinforcement Learning 10 Mar 2018
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning 7 Nov 2017
• Distral: Robust Multitask Reinforcement Learning 13 July 2017

Observational Learning

• Observational Learning by Reinforcement Learning 20 Jun 2017

Meta Learning

• Discovery of Useful Questions as Auxiliary Tasks 10 Sep 2019 arxiv
• Meta-learning of Sequential Strategies 8 May 2019 arxiv
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables 19 Mar 2019 arxiv
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning 11 Jan 2019 arxiv
• Meta-Gradient Reinforcement Learning 24 May 2018 arxiv
• ProMP: Proximal Meta-Policy Search 16 Oct 2018 arxiv
• Unsupervised Meta-Learning for Reinforcement Learning 12 Jun 2018

Distributional

• GAN Q-learning 20 July 2018
• Implicit Quantile Networks for Distributional Reinforcement Learning 14 Jun 2018
• Nonlinear Distributional Gradient Temporal-Difference Learning 20 May 2018
• DISTRIBUTED DISTRIBUTIONAL DETERMINISTIC POLICY GRADIENTS 23 Apr 2018
• An Analysis of Categorical Distributional Reinforcement Learning 22 Feb 2018
• Distributional Reinforcement Learning with Quantile Regression 27 Oct 2017
• A Distributional Perspective on Reinforcement Learning 21 July 2017

Planning

• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning 12 June 2019 arxiv

Safety

• Robust Reinforcement Learning for Continuous Control with Model Misspecification 18 Jun 2019 arxiv
• Verifiable Reinforcement Learning via Policy Extraction 22 May 2018 arxiv

Inverse RL

• ADDRESSING SAMPLE INEFFICIENCY AND REWARD BIAS IN INVERSE REINFORCEMENT LEARNING 9 Sep 2018

No reward RL

• Fast Task Inference with Variational Intrinsic Successor Features 2 Jun 2019 arxiv
• Curiosity-driven Exploration by Self-supervised Prediction 15 May 2017

Time

• Interval timing in deep reinforcement learning agents 31 May 2019 arxiv
• Time Limits in Reinforcement Learning

Adversarial learning

• Sample-efficient Adversarial Imitation Learning from Observation 18 Jun 2019 arxiv

Use Natural Language

• Using Natural Language for Reward Shaping in Reinforcement Learning 31 May 2019 arxiv

Generative and contrastive representation learning

• Unsupervised State Representation Learning in Atari 19 Jun 2019 arxiv

Belief

• Shaping Belief States with Generative Environment Models for RL 24 Jun 2019 arxiv

PAC

• Provably Convergent Off-Policy Actor-Critic with Function Approximation 11 Nov 2019 arxiv

Applications

• Learning Reciprocity in Complex Sequential Social Dilemmas 19 Mar 2019 arxiv
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills 9 Apr 2018
• TUNING RECURRENT NEURAL NETWORKS WITH REINFORCEMENT LEARNING