ns1 / Pktvisor

This project is in active development.

Introduction  •   Get Started  •   Docs  •   Build  •   Contribute  •   Contact Us

What is pktvisor?

pktvisor (pronounced "packet visor") is an observability tool for summarizing high volume, information overloaded data streams directly at the edge. Its goal is to extract the useful signal from the less useful noise; to separate the needles from the haystacks as close to the source as possible. This results in lightweight, immediately actionable observability data at a tiny fraction of the raw data size.

It is a resource efficient, side-car style agent built from the ground up to be dynamically controlled in real time via API. Its output is useful both on-node via command line (for a localized, hyper real-time view) as well as centrally collected into industry standard observability stacks like Prometheus and Grafana.

The modular input stream system is designed to tap into data streams, and currently focuses on packet capture but will soon support additional taps such as sFlow / Netflow , dnstap, envoy taps, and eBPF.

The modular, real-time stream processor includes full application level analysis, and typically summarizes to one minute buckets of:

• Counters
• Histograms and Quantiles
• Timers and Rates
• Heavy Hitters/Frequent Items/Top N
• Set Cardinality
• GeoIP

pktvisor has its origins in observability of critical internet infrastructure, including traffic engineering and DDoS protection.

These screenshots display both the command line and centralized views of the Network and DNS stream processors, and the types of summary information provided:

Get Started

Docker

The easiest way to get started with pktvisor is to use the public docker image. The image contains the collector agent (pktvisord), the command line UI (pktvisor-cli), and the pcap file analyzer (pktvisor-pcap). When running the container, you specify which tool to run.

1. Pull the container

docker pull ns1labs/pktvisor

2. Start the collector agent

This will start in the background and stay running. Note that the final two arguments select pktvisord agent and the any ethernet interface for packet capture. You may substitute any for a known interface on your device, such as eth0. Note that this step requires docker host networking to observe traffic outside the container, and that only Linux supports host networking currently:

docker run --rm --net=host -d ns1labs/pktvisor pktvisord any

3. Run the command line UI

After the agent is running, you can observe results locally with the included command line UI. This command will run the UI (pktvisor-cli) in the foreground, and exit when Ctrl-C is pressed. It connects to the running agent locally using the built in REST API.

docker run -it --rm --net=host ns1labs/pktvisor pktvisor-cli

Other Installation Methods

There are currently no prebuilt packages besides Docker, although we are working on additional installation methods. If you have a preferred method you would like to see support for, please create an issue. Until then, you may build your own executable, please see the Build section below.

Docs

Agent Usage

A collector agent should be installed on each node to be monitored.

Current command line options are described with:

docker run --rm ns1labs/pktvisor pktvisord --help

    Usage:
      pktvisord [options] [IFACE]
      pktvisord (-h | --help)
      pktvisord --version

    pktvisord summarizes data streams and exposes a REST API control plane for configuration and metrics.

    IFACE, if specified, is either a network interface or an IP address (4 or 6). If this is specified,
    a "pcap" input stream will be automatically created, with "net" and "dns" handler modules attached.
    ** Note that this is deprecated; you should instead use --admin-api and create the pcap input stream via API.

    Base Options:
      -l HOST               Run webserver on the given host or IP [default: localhost]
      -p PORT               Run webserver on the given port [default: 10853]
      --admin-api           Enable admin REST API giving complete control plane functionality [default: false]
                            When not specified, the exposed API is read-only access to summarized metrics.
                            When specified, write access is enabled for all modules.
      -h --help             Show this screen
      -v                    Verbose log output
      --no-track            Don't send lightweight, anonymous usage metrics.
      --version             Show version
      --geo-city FILE       GeoLite2 City database to use for IP to Geo mapping (if enabled)
      --geo-asn FILE        GeoLite2 ASN database to use for IP to ASN mapping (if enabled)
    Handler Module Defaults:
      --max-deep-sample N   Never deep sample more than N% of streams (an int between 0 and 100) [default: 100]
      --periods P           Hold this many 60 second time periods of history in memory [default: 5]
    pcap Input Module Options (deprecated, use admin-api instead):
      -b BPF                Filter packets using the given BPF string
      -H HOSTSPEC           Specify subnets (comma separated) to consider HOST, in CIDR form. In live capture this /may/ be detected automatically
                            from capture device but /must/ be specified for pcaps. Example: "10.0.1.0/24,10.0.2.1/32,2001:db8::/64"
                            Specifying this for live capture will append to any automatic detection.

Command Line UI Usage

The command line UI (pktvisor-cli) connects directly to a pktvisord agent to visualize the real time stream summarization, which is by default a sliding 5 minute time window. It can also connect to a remote agent.

docker run --rm ns1labs/pktvisor pktvisor-cli --help

Usage:
  pktvisor-cli [-p PORT] [-H HOST]
  pktvisor-cli -h
  pktvisor-cli --version

  -H string
    	Query pktvisord metrics webserver on the given host (default "localhost")
  -h	Show help
  -p int
    	Query pktvisord metrics webserver on the given port (default 10853)
  -version
    	Show client version

pcap File Analysis

pktvisor-pcap is a tool that can statically analyze prerecorded packet capture files. It takes many of the same options, and does all of the same analysis, as the live agent version.

docker run --rm ns1labs/pktvisor pktvisor-pcap --help

    Usage:
      pktvisor-pcap [options] PCAP
      pktvisor-pcap (-h | --help)
      pktvisor-pcap --version

    Summarize a pcap file. The result will be written to stdout in JSON format, while console logs will be printed
    to stderr.

    Options:
      --max-deep-sample N   Never deep sample more than N% of streams (an int between 0 and 100) [default: 100]
      --periods P           Hold this many 60 second time periods of history in memory. Use 1 to summarize all data. [default: 5]
      -h --help             Show this screen
      --version             Show version
      -v                    Verbose log output
      -b BPF                Filter packets using the given BPF string
      --geo-city FILE       GeoLite2 City database to use for IP to Geo mapping (if enabled)
      --geo-asn FILE        GeoLite2 ASN database to use for IP to ASN mapping (if enabled)
      -H HOSTSPEC           Specify subnets (comma separated) to consider HOST, in CIDR form. In live capture this /may/ be detected automatically
                            from capture device but /must/ be specified for pcaps. Example: "10.0.1.0/24,10.0.2.1/32,2001:db8::/64"
                            Specifying this for live capture will append to any automatic detection.

You can use the docker container by passing in a volume referencing the directory containing the pcap file. The standard output will contain the JSON summarization output, which you can capture or pipe into other tools, for example:

$ docker run --rm -v /pktvisor/src/tests/fixtures:/pcaps ns1labs/pktvisor pktvisor-pcap /pcaps/dns_ipv4_udp.pcap | jq .

[2021-03-11 18:45:04.572] [pktvisor] [info] Load input plugin: PcapInputModulePlugin dev.vizer.module.input/1.0
[2021-03-11 18:45:04.573] [pktvisor] [info] Load handler plugin: DnsHandler dev.vizer.module.handler/1.0
[2021-03-11 18:45:04.573] [pktvisor] [info] Load handler plugin: NetHandler dev.vizer.module.handler/1.0
...
processed 140 packets
{
  "5m": {
    "dns": {
      "cardinality": {
        "qname": 70
      },
      "period": {
        "length": 6,
        "start_ts": 1567706414
      },
      "top_nxdomain": [],
      "top_qname2": [
        {
          "estimate": 140,
          "name": ".test.com"
        }
      ],
...     

Metrics Collection

The metrics are available from the agent in JSON format via the REST API.

For most use cases, you will want to collect the most recent full 1-minute bucket, once per minute:

curl localhost:10853/api/v1/metrics/bucket/1

This can be done with tools like telegraf and the standard HTTP plugin. Example telegraf config snippet:

[inputs]
[[inputs.http]]
urls = [ "http://127.0.0.1:10853/api/v1/metrics/bucket/1",]
interval = "60s"
data_format = "json"
json_query = "1m"
json_time_key = "period_start_ts"
json_time_format = "unix"
json_string_fields = [
  "dns_*",
  "packets_*",
]

[inputs.http.tags]
t = "pktvisor"
interval = "60"

Prometheus

pktvisord will have native Prometheus support in version 3.2.0. Until then, an adapter is available in the repository.

REST API

REST API documentation, including a description of the metrics that are available, is available in OpenAPI Format

Please note that the administration control plane API is currently undergoing heavy iteration and so is not yet documented. If you have a use case that requires the administration API, please contact us to discuss.

Advanced Agent Example

Starting the collector agent from Docker with MaxmindDB GeoIP/GeoASN support and using the Host option to identify ingress and egress traffic:

docker run --rm --net=host -d \
    --mount type=bind,source=/opt/geo,target=/geo \
    ns1labs/pktvisor pktvisord \
    --geo-city /geo/GeoIP2-City.mmdb \
    --geo-asn /geo/GeoIP2-ISP.mmdb \
    -H 192.168.0.54/32,127.0.0.1/32 \
    eth0

Further Documentation

We recognize the value of first class documentation, and this section is being expanded. Please contact us if you have any questions on installation, use, or development.

Contact Us

We are very interested in hearing about your use cases, feature requests, and other feedback!

• File an issue
• Use our public feature board
• Start a Discussion
• Join us on Slack
• Send mail to [email protected]

Build

The main code base is written in clean, modern C++. The pktvisor-cli command line interface is written in Go. The build system requires CMake and the Conan package manager system.

pktvisor adheres to semantic versioning.

Dependencies

• Linux or OSX
• Conan C++ package manager
• CMake >= 3.13 (cmake)
• C++ compiler supporting C++17
• MaxMind DB (libmaxmindb-dev)
• PcapPlusPlus (NS1 fork)

In addition, debugging integration tests requires:

• jq
• graphtage

Building

The general build steps are:

$ git clone https://github.com/ns1/pktvisor.git
$ cd pktvisor
$ mkdir build && cd build
$ conan install ..
$ cmake ..
$ make all test
$ bin/pktvisord --help

As development environments can vary widely, please see the Dockerfile and Continuous Integration build file for reference.

Contribute

Thanks for considering contributing! We will expand this section with more detailed information to guide you through the process.

Please open Pull Requests against the develop branch. If you are considering a larger contribution, please contact us to discuss your design.

See the NS1 Contribution Guidelines for more information.

License

This code is released under Mozilla Public License 2.0. You can find terms and conditions in the LICENSE file.