ikostrikov / Pytorch A2c Ppo Acktr Gail

Licence: mit

PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).

Programming Languages

python

139335 projects - #7 most used programming language

Jupyter Notebook

11667 projects

Projects that are alternatives of or similar to Pytorch A2c Ppo Acktr Gail

imitation learning

PyTorch implementation of some reinforcement learning algorithms: A2C, PPO, Behavioral Cloning from Observation (BCO), GAIL.

Stars: ✭ 93 (-96.47%)

Mutual labels: deep-reinforcement-learning, proximal-policy-optimization, ppo, advantage-actor-critic, a2c

Deep-Reinforcement-Learning-With-Python

Master classic RL, deep RL, distributional RL, inverse RL, and more using OpenAI Gym and TensorFlow with extensive Math

Stars: ✭ 222 (-91.57%)

Mutual labels: deep-reinforcement-learning, actor-critic, ppo, a2c

Reinforcement Learning With Tensorflow

Simple Reinforcement learning tutorials, 莫烦Python 中文AI教学

Stars: ✭ 6,948 (+163.98%)

Mutual labels: reinforcement-learning, ppo, actor-critic, proximal-policy-optimization

Torch Ac

Recurrent and multi-process PyTorch implementation of deep reinforcement Actor-Critic algorithms A2C and PPO

Stars: ✭ 70 (-97.34%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo, actor-critic

Minimalrl

Implementations of basic RL algorithms with minimal lines of codes! (pytorch based)

Stars: ✭ 2,051 (-22.07%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo, a2c

Pytorch Drl

PyTorch implementations of various Deep Reinforcement Learning (DRL) algorithms for both single agent and multi-agent.

Stars: ✭ 233 (-91.15%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo, actor-critic

Deeprl Tutorials

Contains high quality implementations of Deep Reinforcement Learning algorithms written in PyTorch

Stars: ✭ 748 (-71.58%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo, actor-critic

Tianshou

An elegant PyTorch deep reinforcement learning library.

Stars: ✭ 4,109 (+56.12%)

Mutual labels: ppo, mujoco, atari, a2c

Drq

DrQ: Data regularized Q

Stars: ✭ 268 (-89.82%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, actor-critic, mujoco

Deep reinforcement learning course

Implementations from the free course Deep Reinforcement Learning with Tensorflow and PyTorch

Stars: ✭ 3,232 (+22.8%)

Mutual labels: deep-reinforcement-learning, ppo, actor-critic, a2c

Reinforcement Learning

Learn Deep Reinforcement Learning in 60 days! Lectures & Code in Python. Reinforcement Learning + Deep Learning

Stars: ✭ 3,329 (+26.48%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo, a2c

Rainy

☔ Deep RL agents with PyTorch☔

Stars: ✭ 39 (-98.52%)

Mutual labels: deep-reinforcement-learning, ppo, a2c, acktr

Pytorch sac ae

PyTorch implementation of Soft Actor-Critic + Autoencoder(SAC+AE)

Stars: ✭ 94 (-96.43%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, actor-critic, mujoco

Lagom

lagom: A PyTorch infrastructure for rapid prototyping of reinforcement learning algorithms.

Stars: ✭ 364 (-86.17%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo, mujoco

Pytorch sac

PyTorch implementation of Soft Actor-Critic (SAC)

Stars: ✭ 174 (-93.39%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, actor-critic, mujoco

Slm Lab

Modular Deep Reinforcement Learning framework in PyTorch. Companion library of the book "Foundations of Deep Reinforcement Learning".

Stars: ✭ 904 (-65.65%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, ppo

Rl algos

Reinforcement Learning Algorithms

Stars: ✭ 14 (-99.47%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, actor-critic

Coach

Reinforcement Learning Coach by Intel AI Lab enables easy experimentation with state of the art Reinforcement Learning algorithms

Stars: ✭ 2,085 (-20.78%)

Mutual labels: reinforcement-learning, mujoco, roboschool

Pytorch A3c

PyTorch implementation of Asynchronous Advantage Actor Critic (A3C) from "Asynchronous Methods for Deep Reinforcement Learning".

Stars: ✭ 879 (-66.6%)

Mutual labels: reinforcement-learning, deep-reinforcement-learning, actor-critic

Mario rl

Stars: ✭ 60 (-97.72%)

Mutual labels: reinforcement-learning, ppo, actor-critic

View All Similar Projects ➔

pytorch-a2c-ppo-acktr

Update (April 12th, 2021)

PPO is great, but Soft Actor Critic can be better for many continuous control tasks. Please check out my new RL repository in jax.

Please use hyper parameters from this readme. With other hyper parameters things might not work (it's RL after all)!

This is a PyTorch implementation of

Advantage Actor Critic (A2C), a synchronous deterministic version of A3C
Proximal Policy Optimization PPO
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation ACKTR
Generative Adversarial Imitation Learning GAIL

Also see the OpenAI posts: A2C/ACKTR and PPO for more information.

This implementation is inspired by the OpenAI baselines for A2C, ACKTR and PPO. It uses the same hyper parameters and the model since they were well tuned for Atari games.

Please use this bibtex if you want to cite this repository in your publications:

@misc{pytorchrl,
  author = {Kostrikov, Ilya},
  title = {PyTorch Implementations of Reinforcement Learning Algorithms},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail}},
}

Supported (and tested) environments (via OpenAI Gym)

Atari Learning Environment
MuJoCo
PyBullet (including Racecar, Minitaur and Kuka)
DeepMind Control Suite (via dm_control2gym)

I highly recommend PyBullet as a free open source alternative to MuJoCo for continuous control tasks.

All environments are operated using exactly the same Gym interface. See their documentations for a comprehensive list.

To use the DeepMind Control Suite environments, set the flag --env-name dm.<domain_name>.<task_name>, where domain_name and task_name are the name of a domain (e.g. hopper) and a task within that domain (e.g. stand) from the DeepMind Control Suite. Refer to their repo and their tech report for a full list of available domains and tasks. Other than setting the task, the API for interacting with the environment is exactly the same as for all the Gym environments thanks to dm_control2gym.

Requirements

Python 3 (it might work with Python 2, but I didn't test it)
PyTorch
Stable baselines3

In order to install requirements, follow:

# PyTorch
conda install pytorch torchvision -c soumith

# Other requirements
pip install -r requirements.txt

# Gym Atari
conda install -c conda-forge gym-atari

Contributions

Contributions are very welcome. If you know how to make this code better, please open an issue. If you want to submit a pull request, please open an issue first. Also see a todo list below.

Also I'm searching for volunteers to run all experiments on Atari and MuJoCo (with multiple random seeds).

Disclaimer

It's extremely difficult to reproduce results for Reinforcement Learning methods. See "Deep Reinforcement Learning that Matters" for more information. I tried to reproduce OpenAI results as closely as possible. However, majors differences in performance can be caused even by minor differences in TensorFlow and PyTorch libraries.

TODO

Improve this README file. Rearrange images.
Improve performance of KFAC, see kfac.py for more information
Run evaluation for all games and algorithms

Visualization

In order to visualize the results use visualize.ipynb.

Training

Atari

A2C

python main.py --env-name "PongNoFrameskip-v4"

PPO

python main.py --env-name "PongNoFrameskip-v4" --algo ppo --use-gae --lr 2.5e-4 --clip-param 0.1 --value-loss-coef 0.5 --num-processes 8 --num-steps 128 --num-mini-batch 4 --log-interval 1 --use-linear-lr-decay --entropy-coef 0.01

ACKTR

python main.py --env-name "PongNoFrameskip-v4" --algo acktr --num-processes 32 --num-steps 20

MuJoCo

Please always try to use --use-proper-time-limits flag. It properly handles partial trajectories (see https://github.com/sfujim/TD3/blob/master/main.py#L123).

A2C

python main.py --env-name "Reacher-v2" --num-env-steps 1000000

PPO

python main.py --env-name "Reacher-v2" --algo ppo --use-gae --log-interval 1 --num-steps 2048 --num-processes 1 --lr 3e-4 --entropy-coef 0 --value-loss-coef 0.5 --ppo-epoch 10 --num-mini-batch 32 --gamma 0.99 --gae-lambda 0.95 --num-env-steps 1000000 --use-linear-lr-decay --use-proper-time-limits

ACKTR

ACKTR requires some modifications to be made specifically for MuJoCo. But at the moment, I want to keep this code as unified as possible. Thus, I'm going for better ways to integrate it into the codebase.

Enjoy

Atari

python enjoy.py --load-dir trained_models/a2c --env-name "PongNoFrameskip-v4"

MuJoCo

python enjoy.py --load-dir trained_models/ppo --env-name "Reacher-v2"

Results

A2C

PPO

ACKTR

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

ikostrikov / Pytorch A2c Ppo Acktr Gail

Programming Languages

Labels

Projects that are alternatives of or similar to Pytorch A2c Ppo Acktr Gail

pytorch-a2c-ppo-acktr

Update (April 12th, 2021)

Please use hyper parameters from this readme. With other hyper parameters things might not work (it's RL after all)!

Supported (and tested) environments (via OpenAI Gym)

Requirements

Contributions

Disclaimer

TODO

Visualization

Training

Atari

A2C

PPO

ACKTR

MuJoCo

A2C

PPO

ACKTR

Enjoy

Atari

MuJoCo

Results

A2C

PPO

ACKTR