Neural Fictitious Self-Play
An implementation of Neural Fictitous Self-Play [1] in the PokerRL framework.
This codebase is designed for:
- Researchers to compare new methods to the NFSP baseline.
- Anyone wanting to learn about Deep RL in imperfect information games.
Reproducing Results from the NFSP paper [1]
The run-script NFSP/example_paper.py launches NFSP in Leduc Hold'em with the same hyperparameters
(but using 3-layer networks) as the original paper [1]. In the run shown, the
100 mA/g (= 0.1A/g from the paper) line is crossed after 230k iterations (4:24hrs) and
the 60 mA/g line after 850k iterations (19hrs). We trained on a single m5.xlarge instance.
To reproduce, simply run NFSP/example_paper.py.
NFSP on your Local Machine
Install locally
First, please install Docker and download the PyCrayon container. For dependency management I recommend Miniconda. To install the dependencies, simply activate your conda environment and then run
conda install pytorch=0.4.1 -c pytorch -y
pip install PokerRL
Running experiments locally
Before starting NFSP, please spin up the log server by
docker run -d -p 8888:8888 -p 8889:8889 --name crayon alband/crayon
docker start crayon
You can now view logs at localhost:8888 in your browser. To run NFSP with custom hyperparameters in any Poker game
supported by PokerRL, build a script similar to NFSP/example_paper.py. Run-scripts define the hyperparameters, the game
to be played, and the evaluation metrics. Here is a very minimalistic example:
from TrainingProfile import TrainingProfile
from workers.driver.Driver import Driver
from PokerRL import BigLeduc # or any other game
if __name__ == '__main__':
ctrl = Driver(
t_prof=TrainingProfile(
name="NFSP_TEST",
game_cls=BigLeduc, # -- The game to play
lr_br=0.1, # -- Some Hyperparameters
lr_avg=0.005,
mini_batch_size_br=128,
mini_batch_size_avg=128,
),
eval_methods={
"br": 2000, # Evaluate Best Response every 2000 iterations
},
)
ctrl.run()
Cloud & Clusters
For deployment on AWS, whether single-core, many-core distributed, or on a cluster, please first follow the tutorial in the corresponding section of PokerRL's README.
We recommend forking this repository so you can write your own scripts but still have remote access through git.
In your run-script set either the DISTRIBUTED or the CLUSTER option of the TrainingProfile to True
(see e.g. NFSP/example_distributed.py).
Moreover, you should specify the number of LearnerActor and evaluator workers (if applicable) you want to deploy.
Note that hyperparmeters ending with "_per_la" (e.g. the batch size) are effectively multiplied by the number of
workers.
When running in DISTRIBUTED mode (i.e. one machine, many cores), simply ssh onto your AWS instance, get your code
onto it (e.g. through git cloning your forked repo) and start your run-script.
To fire up a cluster, define a .yaml cluster configuration that properly sets up your workers. Each of them
should have a copy of your forked repo as well as all dependencies on it.
Use ray up ... in an ssh session to the head of the cluster to start the job - more detailed instructions about
the underlying framework we use for distributed computing can be found at ray.
Citing
If you use this repository in your research, you can cite it by citing PokerRL as follows:
@misc{steinberger2019pokerrl,
author = {Eric Steinberger},
title = {PokerRL},
year = {2019},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/TinkeringCode/PokerRL}},
}
Authors
- Eric Steinberger
License
This project is licensed under the MIT License - see the LICENSE file for details.
References
[1] Heinrich, Johannes, and David Silver. "Deep reinforcement learning from self-play in imperfect-information games." arXiv preprint arXiv:1603.01121 (2016).