To explore Tune further:

• Tune documentation
• Tune tutorials
• Tune examples

To explore RLLib further:

• a simple tutorial where RLLib is used to train a PPO algorithm
• RLLib documentation
• RLLib tutorials
• RLLib examples

To explore the toolbox marltoolbox further, take a look at our examples.

• Constraints: With the Tune function API, you only need to provide the training function. See the Tune documentation.
• Best used: If you want to very quickly run some code from an external repository.
• Functionalities: Running several seeds in parallel and comparing their results. Easily plot values to TensorBoard and visualizing the plots in live. Tracking your experiments and hyperparameters. Hyperparameter search. Early stopping.

• Constraints: You need to provide a Trainer class with at minimum a setup method and a step method. See the Tune documentation.
• Best used: If you want to run some code from an external repository and you need checkpoints. Helpers in this toolbox (marltoolbox.utils.policy.get_tune_policy_class) will also allow you transform this class (already trained) into frozen RLLib policies. This is useful to produce evaluation against other RLLib algorithms or when using experimentation tools from marltoolbox.utils.
• Additional functionalities: Cleaner format. Checkpoints. Allow conversion to the RLLib policy API.
  The trained agents can be converted to the RLLib policy API for evaluation only. This allows you to use functionalities which rely on the RLLib API (but not training).

• Constraints: You need to use the RLLib API (trainer, policy, callbacks, etc.). For information, RLLib trainer classes are specific implementations of the Tune class API (just above). See the RLLib documentation.
• Best used: If you are creating a new training setup or policy from scratch. Or if you want a seamless integration with all RLLib components. Or if you need distributed training.
• Additional functionalities: Using easily all components from RLLib (models, environments, algorithms, exploration, schedulers, preprocessing, etc.). Using the customizable trainer and policy factories from RLLib.

If the setup you want to train already exist, has a training loop and if the cost to convert it into RLLib is too expensive, then with minimum changes you can use Tune.

When is the conversion cost to RLLib too high?

• If the algorithm has a complex unusual dataflow
• If the algorithm has an unusual training process
  • like LOLA: performing "virtual" opponent updates
  • like LTFT: nested algorithms
• If you don't need to change the algorithm
• If you don't plan to run the algorithm against policies from RLLib
• If you do not plan to work much with the algorithm. And thus, you do not want to invest time in the conversion to RLLib.
• Some points above and you are only starting to use RLLib
• etc.

Tutorials:

• Tutorial_Basics_How_to_use_the_toolbox.ipynb

Examples:

You can find such examples in marltoolbox.examples.tune_class_api and in marltoolbox.examples.tune_function_api.

Tutorials:

• Tutorial_Basics_How_to_use_the_toolbox.ipynb

a) Examples using the Tune class API:

• Using an A3C policy: amd.py with use_rllib_policy = True (toolbox example)
• Using (custom or not) environments:
  • IPD and coin game environments: amd.py (toolbox example)
  • Asymmetric coin game environment: lola_pg_official.py (toolbox example)

b) Examples using the RLLib API:

• IPD environments: pg_ipd.py (toolbox example)
• Coin game environment: ppo_coin_game.py (toolbox example)
• APEX_DDPG and the water world environment: multi_agent_independent_learning.py
• MADDPG and the two step game environment: two_step_game.py
• Policy Gradient (PG) and the rock paper scissors environment: rock_paper_scissors_multiagent.py (in the run_same_policy function)

Examples:

• Customize policy's postprocessing (processing after env.step) and trainer: inequity_aversion.py (toolbox example)
• Change the loss function of the Policy Gradient (PG) Policy: rock_paper_scissors_multiagent.py
  (in the run_with_custom_entropy_loss function)

In RLLib, customizing a policy allows to change its training and evaluation logics.

Examples:

• Hardcoded random Policy: multi_agent_custom_policy.py
• Hardcoded fixed Policy: rock_paper_scissors_multiagent.py
  (in the run_heuristic_vs_learned function)
• Policy with nested Policies: ltft_with_various_env.py (toolbox example)

Examples:

• Training 2 different policies with 2 different Trainers (less complex but less sample efficient than the 2nd method below): multi_agent_two_trainers.py
• Training 2 different policies with a custom Trainer (more complex, more sample efficient): two_trainer_workflow.py

Tutorials:

• Evaluations_Level_1_best_response_and_self_play_and_cross_play.ipynb

Examples:

• Training a level 1 best response: l1br_amtft.py (toolbox example)
• Evaluating same-play and cross-play performances: amtft_various_env.py (toolbox example)

• various matrix social dilemmas
• various coin games
• bargaining with alternating offers (Emergent Communication through Negotiation)

• AMD (Adaptive Mechanism Design)
• amTFT (Approximate Markov Tit-For-Tat)
• LTFT (Learning Tit-For-Tat, simplified version)
• LOLA-Exact, LOLA-PG, LOLA-DICE
• supervised learning
• population
  • This policy plays an episode by sampling a policy from a population of similar policies
• hierarchical
  • It is a base policy class which allows the use of nested algorithms

• exploration
  • SoftQ with temperature schedule
  • SoftQ with clustering of the Q values
• log
  • callbacks to log values from environments and policies
• lvl1_best_response
  • helper functions to train level 1 exploiters
• policy
  • helper to transform a trained Tune Trainer into frozen RLLib policies
• postprocessing
  • helpers to compute welfare functions and add this data in the evaluation batch (the batches sampled by the evaluation workers)
• restore
  • helpers to load a checkpoint only for a chosen policy (instead of for all existing policies as RLLib does)
• rollout
  • a rollout runner function which can be called from inside a RLLib policy
• self_and_cross_perf
  • a helper to evaluate the performance in self-play and cross-play.
    "self-play": playing against agents from the same training run.
    "cross-play": playing against agents from different training runs.
• plot
  • helpers to plot results

• aggregate_and_plot_tensorboard_data
  • a script to aggregate the logged values from several seeds (into mean, std, etc.) and to create summary plots

• Add unit tests for the algorithms
• Refactor the algorithm to make them more readable
• Use the logger everywhere
• Add and improve docstrings
• Set good hyper-parameters in the custom examples
• Report all results directly in Weights&Biases (saving download time from VM)

• Multi-agent adversarial IRL
• Multi-agent generative adversarial imitation learning
• Model-based RL like PETS, MPC
• Opponent modeling like k-level
• Capability to use algorithms from OpenSpiel like MCTS

• Reward uncertainty
• Full / partial observability of opponent actions
• (partial) Parameter transparency
• Easy benchmarking with metrics specific to MARL
• (more on) Exploitability evaluation
• Performance against a suite of other MARL algorithms