Merlin: Network Magic
Simply a better way; a magical way; to collect, transform, and configure network state information from the CLI or REST API
What is Network Automation, Infrastructure as Code, and Merlin?
Watch these videos for an introduction and live demo of the Merlin, Excalibur, and Lancelot features
What Is Network Automation using Infrastructure As Code?: https://www.youtube.com/embed/Wo6GmvwdSdY
Learn Genie Learn: https://youtu.be/-Ns5sGIKRPI
Powered by Genie
And the pyATS framework
Featuring
And now
Also
And for your Asset Management needs - introducting - Excalibur
Welcome!
Merlin is an infrastructure as code and network automation tool that transforms CLI command and REST API data, using the Cisco Genie parsers, the Cisco pyATS Python library, and Python to automatically generate, at scale, better documentation from the output; send #chatbots; #voicebots; even #phonebots!
Merlin First "Learns" the following features and protocols about the device:
Then Merlin runs multiple "show" commands and creates the following:
A Nice JSON file (command_output.json)
A Nice YAML file (command_output.yaml)
A CSV spreadsheet (command_output.csv)
A Markdown file (command_output.md)
An HTML page (comand_output.html)
An interactive HTML Mind Map (command_output_mind_map.html)
An interactive HTML NETJSON NetGraph (command_output_netjson.html)
A TinyDB
Instant messages to WebEx, Slack, Discord, and others
Text-to-Speech, in over 200 languages, creating customized MP3 audio files in a human voice
Phone calls to any phone number in the world
Instantly. With the push of a button.
Excalibur
If you are here for the Excalibur utility please follow this guide
Excalibur - Automated Asset Management
Genie and pyATS
The main Genie documentation guide:
https://developer.cisco.com/docs/genie-docs/
The main pyATS documentation guide:
https://developer.cisco.com/docs/pyats/
The Cisco's Test Automation Solution
The Cisco Test Automation GitHub repository
https://github.com/CiscoTestAutomation
Here are the pyATS documentation guides on Testbed files and Device Connectivity:
Testbed and Topology Information: https://pubhub.devnetcloud.com/media/pyats/docs/topology/index.html
Device Connection: https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/quickstart/manageconnections.html
Testbed File Example: https://pubhub.devnetcloud.com/media/pyats/docs/topology/example.html
Device Connections: https://developer.cisco.com/docs/pyats/#!connection-to-devices
Secret Strings (how I encrypted the enable secret in my testbed file): https://pubhub.devnetcloud.com/media/pyats/docs/utilities/secret_strings.html
Getting Started
Requirements (instructions below)
Virtual Environment
We recommend running Merlin in a Python virtual environment while testing or developing. This will help keep your host system clean and allow you to have multiple environments to try new things. If you are not using a virtual environment, start at the download/clone step below.
You will also need Python 3, pip, and venv installed on your host system.
In your project directory, create your virtual environment
python3 -m venv envActivate (use) your new virtual environment (Linux):
source env/bin/activateDownload or clone the Merlin repository:
git clone https://github.com/automateyournetwork/merlinInstall pyATS, Rich, markmap, netjsongraph, xmltodict and TinyDB into your environment:
pip install -r requirements.txtsudo apt update
sudo apt install npm
sudo npm install markmap-cli -g
npm install netjsongraph.js --saveIf you run into any installation issues with pyATS, please see the installation guide here: https://pubhub.devnetcloud.com/media/pyats-getting-started/docs/install/installpyATS.html
When you are finished with your Merlin session, enter the deactivate command to exit the python virtual environment.
Merlin's Magic
How to update the appropriate testbed/testbed_*.yaml file to reflect your device inventory and configuration:
Devices:
4500: <-- Update to your router hostname (MUST MATCH HOSTNAME)
alias: <-- Update your alias; this can be the hostname or any friendly name you want
type: <-- This should be "router" for L3 routers with ARP tables and VRFs and such
platform: <-- Your Catalyst IOS-XE Platform
username: <-- Your Cisco device username
password: <-- Your Enable Secret; It is strongly recommended to follow the "Secret Strings" link to encrypt your secrets
ip: <-- The management IP address of your router
Copy & Paste and make each device unique to scale this up to many routers
Devices:
3850: <-- Update to your switch hostname (MUST MATCH HOSTNAME)
9300: <-- Update to your switch hostname (MUST MATCH HOSTNAME)
alias: <-- Update your alias; this can be the hostname or any friendly name you want
type: <-- This should be "switch" for L2 switches without ARP tables or VRFs but features like PoE
platform: <-- Your Catalyst IOS-XE Platform
username: <-- Your Cisco device username
password: <-- Your Enable Secret; It is strongly recommended to follow the "Secret Strings" link to encrypt your secrets
ip: <-- The management IP address of your router
Copy & Paste and make each device unique to scale this up to many routers
Take Merlin to the Cloud
Merlin now connects to the following Cisco.com APIs:
-
Recommended Software Release (using the real version of software from your devices)
-
Serial Number 2 Infomation (using the real serial numbers, for every part, from your devices)
-
Product Security Incident Response Team (PSIRT) (using the real version of software from your device)
Merlin on ISE
Merlin now supports the Cisco Identity Services Engine (ISE) External RESTful Service (ERS) and Monitoring and Troubleshooting (MnT) APIs !
Pro tip:
You can create a testbed file using an Excel spreadsheet (see testbed/SampleTestbedFile.xlsx), this is useful if you are not familiar with YAML and/or have a long list of devices where copying and pasting in Excel would be easier.
-
Edit the relevant fields in the sample file.
-
Run the following command:
pyats create testbed file --path SampleTestbedFile.xlsx --output my_testbed.yaml- That's it. You're good to go!
Ensure SSH connectivity at from your host's CLI, and run the pyATS job:
Cisco IOS-XE:
pyats run job IOS_XE_merlin_job.py --testbed-file testbed/testbed_ios_xe.yamlCisco DevNet Sandbox for CSR1000v (see more here)
pyats run job DevNet_Sandbox_CSR_merlin_job.py --testbed-file testbed/testbed_DevNet_CSR1000v_Sandbox.yamlCisco ISE:
pyats run job pyats run job ISE_merlin_job.pyJunOS 17 / 18 / 19:
pyats run job JUNOS_merlin_job.py --testbed-file testbed/testbed_juniper.yamlF5 BIG-IP:
python3 F5_merlin.pycd Camelot
ls To view the pyATS log in a web browser Locally
pyats logs viewTo view the pyATS log in a web browser remotely
pyats logs view --host 0.0.0.0 --port 8080 -v
Explore Camelot!
Here is an example of just one of the treasures you will find: the show ip route command!
Here is what a Global Routing Table looks like in JSON:
The same routing table, but in YAML:
The JSON and YAML outputs are incredible representations of the routing table and can be used for further pyATS testing or data modeling.
"Business-ready" documentation includes the incredibly powerful and versatile Comma-Separated Values (csv) spreadsheet format.
Markdown, the format this README file is written in, can also be used to express the data in a lightweight format that renders nicely in modern browsers.
What about a full-blown HTML webpage? Merlin also creates at least one of these per command.
Which renders nicely like this in your browser:
Another HTML page, an interactive mindmap, is also created from the Markdown file!
Another Interactive HTML page, this time a NETJSON NetGraph, is also crteated
To launch a Python web server and make Camelot available in a browser, where you can view the HTML pages:
Launch a local web server available on the same host:
cd Camelot
pushd; python3 -m http.server --bind 127.0.0.1 8888; popd;Launch your web browser and visit
Launch a local web server available to remote hosts:
cd Camelot
pushd; python3 -m http.server --bind 0.0.0.0 8888; popd;Launch your web browser and visit
http://{{ your server IP }}:8888/
Sharepoint 2016 Plug-in
Every artifact can also be uploaded dynamically to a Sharepoint 2016 Document Library !
The CSV files work with the Office extension and present as version controlled Excel-online files!
Add the code from the SharePoint2016_Plugin.py file. The show version command is used in the example.
Simply add the SHAREPOINT code, updating to reflect the appropriate command:
#Enter your SharePoint site and target library
APIs:
inventory:
sharePointUrl: 'https://{{ mydomain.com }}'
folderUrl: '/Inventory'
sharePointFileName: "Inventory/Inventory.csv"
uploadFileName: "Inventory.csv"
sharepoint_domain: mydomain
sharepoint_user: {{ domain_user_name }}
sharepoint_password: {{ domain_password }}
contracts:
sharePointUrl: 'https://{{ mydomain.com }}'
folderUrl: '/Inventory'
sharePointFileName: "Inventory/Contracts.csv"
uploadFileName: "Contracts.csv"
sharepoint_domain: mydomain
sharepoint_user: {{ domain_user_name }}
sharepoint_password: {{ domain_password }}Don't have your own lab gear to test with?
Test Merlin using the Cisco DevNet IOS-XE Sandbox
Test Merlin using the Cisco DevNet Nexus 9k Sandbox
Test Merlin using the Cisco DevNet Identity Services Engine 3.0 + pxGrid Sandbox
Supported Command Index
Cisco IOS-XE:
learn ACL
learn ARP
learn BGP
learn CONFIG
learn Dot1X
learn Interface
learn LLDP
learn NTP
learn OSPF
learn Routes
learn STP
learn VLAN
learn VRF
show access lists
show access session
show access session interface {{ interface }} detail
show authentication sessions
show authentication session interface {{ interface }} detail
show cdp neighbors
show cdp neighbors details
show environment all
show etherchannel summary
show interfaces
show interfaces status
show interfaces trunk
show inventory
show ip arp
show ip arp vrf {{ vrf }}
show ip interface brief
show ip ospf
show ip ospf database
show ip ospf interface
show ip ospf neighbor
show ip ospf neighbor detail
show ip route
show ip route vrf {{ vrf }}
show issu state detail
* 4500X IOS-XE in VSS
show mac address-table
show ntp associations
show power inline
show version
show vlan
show vrf
Cisco NXOS:
learn ACL
learn ARP
learn BGP
learn CONFIG
learn Dot1X
learn Interface
learn LLDP
learn NTP
learn OSPF
learn PLATFORM
learn Routes
learn STP
learn VLAN
learn VRF
show access lists
show bgp process VRF all
show bgp sessions
show interfaces status
show inventory
show ip arp
show ip arp vrf {{ vrf }}
show ip interface brief
show ip ospf
show ip route
show ip route vrf {{ vrf }}
show mac address-table
show version
show vlan
show vrf
show vrf all detail
show vrf all interface
Cisco Identity Services Engine (ISE)
External RESTful Service (ERS) APIs
Administrators
Allowed Protocols
Authorization Profiles
downloadable/dynamic Access Control Lists (dACLs)
Endpoint Groups
Identity Groups
Network Devices
Monitoring and Troubleshooting (MnT) APIs
Total Active Sessions
Active Sessions
MAC Session Details
JunOS:
show chassis hardware
show system information
F5 BIG-IP
/mgmt/tm/ltm/virtual
/mgmt/tm/sys/file/ssl-cert
Cross Platform Tests
Tested on:
Cisco:
Cisco Catalyst 4500X-16 03.11.03a.E
Cisco Catalyst 9300-48UXM Gibraltar
Cisco Catalyst 9200-24P 16.12.03 and 17.03.02a
Cisco Catalyst 3850-12X48U Gibraltar 16.12.04
Cisco CSR1000v
Cisco IOSv
Cisco vISE 2.7
Juniper:
JunOS 17, 18, 19
F5:
i2600 REST API
