Threat Bus

A threat intelligence dissemination layer for open-source security tools.

Getting Started — Contributing Guidelines — Writing Plugins — License — Documentation

Chat with us on Matrix.

Key Features

• Connect Open-Source Security Tools: Threat Bus is a pub-sub broker for threat intelligence data. With Threat Bus you can seamlessly integrate MISP intelligence with the Zeek intel framework or report sightings from IDS deployments to some data base.

• Plugin-based Architecture: The project is plugin-based and can be extended easily. Read about the different plugin types and how to write your own. We welcome contributions to adopt new open source tools!

• Official Plugins: We maintain many plugins right in the official Threat Bus repository. Check out our integrations for MISP, Zeek, CIFv3, and generally apps that connect via ZeroMQ, like pyvast-threatbus.

• Snapshotting: The snapshot feature allows subscribers to directly request threat intelligence data for a certain time range from other applications. Threat Bus handles the point-to-point communication of all involved apps.

Getting Started

The config.yaml.example file provides a working configuration for Threat Bus with all existing application plugins enabled together with the RabbitMQ backbone.

The following example shows how to connect MISP, Zeek via Threat Bus. There are more integrations available, so make sure to check out all Threat Bus projects on PyPI.

Start Threat Bus

mv config.yaml.example config.yaml   # rename example config file
threatbus -c config.yaml

Start Zeek as Threat Bus app

zeek -i <INTERFACE> -C ./apps/zeek/threatbus.zeek

Start Zeek and request a snapshot

zeek -i <INTERFACE> -C ./apps/zeek/threatbus.zeek "Tenzir::snapshot_intel=30 days"

Threat Bus also ships as pre-built Docker image and is available on Docker Hub.

Use the Threat Bus Docker container

docker run tenzir/threatbus:latest --help

Start Threat Bus container with a custom config file

docker run -p 47661:47661 -v $PWD/my-custom-config.yaml:/opt/tenzir/threatbus/my-custom-config.yaml tenzir/threatbus:latest -c my-custom-config.yaml

Installation

Install threatbus and all plugins that you require. Optionally, use a virtual environment.

virtualenv venv                       # optional
source venv/bin/activate              # optional
pip install threatbus
pip install threatbus-inmem           # inmemory backbone plugin
pip install threatbus-rabbitmq        # RabbitMQ backbone plugin
pip install threatbus-misp[zmq]       # MISP application plugin
pip install threatbus-zeek            # Zeek application plugin
pip install threatbus-<plugin_name>

Testing

Use the Makefile to run unit and integration tests.

make unit-tests
make integration-tests

The integration tests require a local Zeek and Docker installation.

Plugin Development

Setup a virtual environment and install threatbus and some plugins with the in development mode:

virtualenv venv
source venv/bin/activate
make dev-mode

Configuration & Extension

A plugin must define a setup.py. Whenever a plugin is installed, you have to add a corresponding configuration section to threatbus' config.yaml. That section has to be named after the name in the entrypoint declaration of the plugin's setup.py file.

Please adhere to the plugin naming conventions and always prefix your plugin name with threatbus-.

Plugins can either be apps or backbones. Application plugins (apps) add new functionality to threatbus and allow communication to a threat-intelligence-enabled app (e.g., Zeek or Suricata). Backbone plugins add a new storage and distribution backend to threatbus (e.g., in-memory or Kafka).

Example:

• plugin folder structure:

  plugins
  ├── apps
  |   └── threatbus-myapp
  │       ├── setup.py
  |       └── threatbus_myapp.py
  └── backbones
      └── threatbus-inmem
          ├── setup.py
          └── threatbus_inmem.py
  

• setup.py

  from setuptools import setup
  setup(
    name="threatbus-myapp",
    install_requires="threatbus",
    entry_points={"threatbus.app": ["myapp = threatbus_myapp"]},
    py_modules=["threatbus_myapp"],
  )
  

• config.yaml entry for threatbus

  ...
  plugins:
    apps:
      myapp:
      ...
  

Threat Bus API

Plugins specifications are available in threatbus/appspecs.py and threatbus/backbonespecs.py, respectively. For any plugin, you should at least implement the run function.

App plugins are provided two callback functions to use for subscription management. Internally, Threat Bus will propagate subscription requests to all installed backbone plugins.

The subscription callback allows applications to request an optional snapshot time delta. Threat Bus will forward snapshot requests to all those apps that have implemented the snapshot feature (see threatbus/appspecs.py).

Implementation

Please use the StoppableWorker base class to model your plugin's busy work. Plugins should never block the main thread of the application. Implementing that class also facilitates a graceful shutdown.

All officially maintained Threat Bus plugins implement StoppableWorker. Refer to any of the existing plugins for an example.

License

Threat Bus comes with a 3-clause BSD license.