Multi-Agent Resource Optimization (MARO) platform is an instance of Reinforcement learning as a Service (RaaS) for real-world resource optimization. It can be applied to many important industrial domains, such as container inventory management in logistics, bike repositioning in transportation, virtual machine provisioning in data centers, and asset management in finance. Besides Reinforcement Learning (RL), it also supports other planning/decision mechanisms, such as Operations Research.

Key Components of MARO:

• Simulation toolkit: it provides some predefined scenarios, and the reusable wheels for building new scenarios.
• RL toolkit: it provides a full-stack abstraction for RL, such as agent manager, agent, RL algorithms, learner, actor, and various shapers.
• Distributed toolkit: it provides distributed communication components, interface of user-defined functions for message auto-handling, cluster provision, and job orchestration.

Contents

Install MARO from PyPI

• Mac OS / Linux

  pip install pymaro
  

• Windows

  # Install torch first, if you don't have one.
  pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
  
  pip install pymaro
  

Install MARO from Source (Editable Mode)

• Prerequisites

  • C++ Compiler
    • Linux or Mac OS X: gcc
    • Windows: Build Tools for Visual Studio 2017

• Enable Virtual Environment

  • Mac OS / Linux

    # If your environment is not clean, create a virtual environment firstly.
    python -m venv maro_venv
    source ./maro_venv/bin/activate
    

  • Windows

    # If your environment is not clean, create a virtual environment firstly.
    python -m venv maro_venv
    
    # You may need this for SecurityError in PowerShell.
    Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted
    
    # Activate the virtual environment.
    .\maro_venv\Scripts\activate
    

• Install MARO

  • Mac OS / Linux

    # Install MARO from source.
    bash scripts/install_maro.sh
    

  • Windows

    # Install MARO from source.
    .\scripts\install_maro.bat
    

• Notes: If your package is not found, remember to set your PYTHONPATH

  • Mac OS / Linux

  export PYTHONPATH=PATH-TO-MARO
  

  • Windows

  $Env:PYTHONPATH=PATH-TO-MARO
  

Quick Example

from maro.simulator import Env

env = Env(scenario="cim", topology="toy.5p_ssddd_l0.0", start_tick=0, durations=100)

metrics, decision_event, is_done = env.step(None)

while not is_done:
    metrics, decision_event, is_done = env.step(None)

print(f"environment metrics: {env.metrics}")

Environment Visualization

# Enable environment dump feature, when initializing the environment instance
env = Env(scenario="cim",
          topology="toy.5p_ssddd_l0.0",
          start_tick=0,
          durations=100,
          options={"enable-dump-snapshot": "./dump_data"})

# Inspect environment with the dump data
maro inspector env --source ./dump_data

Show Cases

• Case I - Container Inventory Management

• Case II - Citi Bike

Run Playground

• Pull from Docker Hub

  # Run playground container.
  # Redis commander (GUI for redis) -> http://127.0.0.1:40009
  # Local host docs -> http://127.0.0.1:40010
  # Jupyter lab with maro -> http://127.0.0.1:40011
  docker run -p 40009:40009 -p 40010:40010 -p 40011:40011 arthursjiang/maro:cpu
  

• Build from source

  • Mac OS / Linux

    # Build playground image.
    bash ./scripts/build_playground.sh
    
    # Run playground container.
    # Redis commander (GUI for redis) -> http://127.0.0.1:40009
    # Local host docs -> http://127.0.0.1:40010
    # Jupyter lab with maro -> http://127.0.0.1:40011
    docker run -p 40009:40009 -p 40010:40010 -p 40011:40011 maro/playground:cpu
    

  • Windows

    # Build playground image.
    .\scripts\build_playground.bat
    
    # Run playground container.
    # Redis commander (GUI for redis) -> http://127.0.0.1:40009
    # Local host docs -> http://127.0.0.1:40010
    # Jupyter lab with maro -> http://127.0.0.1:40011
    docker run -p 40009:40009 -p 40010:40010 -p 40011:40011 maro/playground:cpu
    

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Related Papers

Container Inventory Management

Wenlei Shi, Xinran Wei, Jia Zhang, Xiaoyuan Ni, Arthur Jiang, Jiang Bian, Tie-Yan Liu. "Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations". AAMAS 2021

Xihan Li, Jia Zhang, Jiang Bian, Yunhai Tong, Tie-Yan Liu. "A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Network". AAMAS 2019

Related News

MSRA Top-10 Hack-Techs in 2021

Open Source Platform MARO: Anywhere Door for Resource Optimization

AI from "Point" to "Surface"

Cite Us

License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.