Viseron - Self-hosted NVR with object detection

Viseron is a self-hosted, local only NVR implemented in Python.
The goal is ease of use while also leveraging hardware acceleration for minimal system load.

Notable features

• Records videos on detected objects
• Supports multiple different object detectors:
  • YOLOv3/4 Darknet using OpenCV
  • Tensorflow via Google Coral EdgeTPU
• Motion detection
• Face recognition
• Lookback, buffers frames to record before the event actually happened
• Multiplatform, should support any x86-64 machine running Linux, aswell as RPi3.
  Builds are tested and verified on the following platforms:
  • Ubuntu 18.04 with Nvidia GPU
  • Ubuntu 18.04 running on an Intel NUC
  • RaspberryPi 3B+
• Native support for RTSP and MJPEG
• Supports hardware acceleration on different platforms
  • CUDA for systems with a supported GPU
  • OpenCL
  • OpenMax and MMAL on the RaspberryPi 3B+
  • Intel QuickSync with VA-API
• Zones to limit detection to a particular area to reduce false positives
• Masks to limit where motion detection occurs
• Stop/start cameras on-demand over MQTT
• Home Assistant integration via MQTT
• unRAID Community Application

Table of Contents

• Getting started
• Configuration Options
  • Cameras
    • Substream
    • Camera motion detection
    • Mask
    • Camera object detection
    • Zones
    • Points
  • Object detection
    • Darknet
    • EdgeTPU
    • Labels
  • Motion detection
  • Recorder
  • Post Processors
    • Face Recognition
  • MQTT
    • Topics for each camera
    • Topics for each Viseron instance
    • Home Assistant MQTT Discovery
  • Logging
  • Secrets
• Benchmarks

Getting started

Choose the appropriate docker container for your machine. Builds are published to Docker Hub

docker run --rm \
--privileged \
-v <recordings path>:/recordings \
-v <config path>:/config \
-v /etc/localtime:/etc/localtime:ro \
-v /dev/bus/usb:/dev/bus/usb \
-v /opt/vc/lib:/opt/vc/lib \
--name viseron \
--device /dev/vchiq:/dev/vchiq --device /dev/vcsm:/dev/vcsm \
roflcoopter/viseron-rpi:latest

version: "2.4"
services:
  viseron:
    image: roflcoopter/viseron-rpi:latest
    container_name: viseron
    volumes:
      - <recordings path>:/recordings
      - <config path>:/config
      - /etc/localtime:/etc/localtime:ro
      - /dev/bus/usb:/dev/bus/usb
      - /opt/vc/lib:/opt/vc/lib
    devices:
      - /dev/vchiq:/dev/vchiq
      - /dev/vcsm:/dev/vcsm
    privileged: true

docker run --rm \
-v <recordings path>:/recordings \
-v <config path>:/config \
-v /etc/localtime:/etc/localtime:ro \
--name viseron \
roflcoopter/viseron:latest

version: "2.4"

services:
  viseron:
    image: roflcoopter/viseron:latest
    container_name: viseron
    volumes:
      - <recordings path>:/recordings
      - <config path>:/config
      - /etc/localtime:/etc/localtime:ro

docker run --rm \
-v <recordings path>:/recordings \
-v <config path>:/config \
-v /etc/localtime:/etc/localtime:ro \
--name viseron \
--device /dev/dri \
roflcoopter/viseron-vaapi:latest

version: "2.4"

services:
  viseron:
    image: roflcoopter/viseron-vaapi:latest
    container_name: viseron
    volumes:
      - <recordings path>:/recordings
      - <config path>:/config
      - /etc/localtime:/etc/localtime:ro
    devices:
      - /dev/dri

docker run --rm \
-v <recordings path>:/recordings \
-v <config path>:/config \
-v /etc/localtime:/etc/localtime:ro \
--name viseron \
--runtime=nvidia \
roflcoopter/viseron-cuda:latest

version: "2.4"

services:
  viseron:
    image: roflcoopter/viseron-cuda:latest
    container_name: viseron
    volumes:
      - <recordings path>:/recordings
      - <config path>:/config
      - /etc/localtime:/etc/localtime:ro
    runtime: nvidia

VAAPI support is built into every container. To utilize it you need to add --device /dev/dri to your docker command.
EdgeTPU support is also included in all containers. To use it, add -v /dev/bus/usb:/dev/bus/usb --privileged to your docker command.

The config.yaml has to be mounted to the folder /config.
If no config is present, a default minimal one will be created.
Here you need to fill in atleast your cameras and you should be good to go.

Configuration Options

Cameras

cameras:
  - name: Front door
    mqtt_name: viseron_front_door
    host: 192.168.30.2
    port: 554
    username: user
    password: pass
    path: /Streaming/Channels/101/
    width: 1920
    height: 1080
    fps: 6 
    motion_detection:
      interval: 1
      trigger_detector: false
    object_detection:
      interval: 1
      labels:
        - label: person
          confidence: 0.9
        - label: pottedplant
          confidence: 0.9

Used to build the FFMPEG command to decode camera stream.
The command is built like this:
"ffmpeg" + global_args + input_args + hwaccel_args + codec + "-rtsp_transport tcp -i " + (stream url) + filter_args + output_args | Name | Type | Default | Supported options | Description | | -----| -----| ------- | ----------------- |------------ | | name | str | required | any string | Friendly name of the camera | | mqtt_name | str | name given above | any string | Name used in MQTT topics | | stream_format | str | rtsp | rtsp, mjpeg | FFMPEG stream format | | host | str | required | any string | IP or hostname of camera | | port | int | required | any integer | Port for the camera stream | | username | str | optional | any string | Username for the camera stream | | password | str | optional | any string | Password for the camera stream | | path | str | optional | any string | Path to the camera stream, eg /Streaming/Channels/101/ | | width | int | optional | any integer | Width of the stream. Will use OpenCV to get this information if not given | | height | int | optional | any integer | Height of the stream. Will use OpenCV to get this information if not given | | fps | int | optional | any integer | FPS of the stream. Will use OpenCV to get this information if not given | | global_args | list | optional | a valid list of FFMPEG arguments | See source code for default arguments | | input_args | list | optional | a valid list of FFMPEG arguments | See source code for default arguments | | hwaccel_args | list | optional | a valid list of FFMPEG arguments | FFMPEG decoder hardware acceleration arguments | | codec | str | optional | any supported decoder codec | FFMPEG video decoder codec, eg h264_cuvid | | rtsp_transport | str | tcp | tcp, udp, udp_multicast, http | Sets RTSP transport protocol. Change this if your camera doesnt support TCP | | filter_args | list | optional | a valid list of FFMPEG arguments | See source code for default arguments | | substream | dictionary | optional | see Substream config | Substream to perform image processing on | | motion_detection | dictionary | optional | see Camera motion detection config | Overrides the global motion_detection config | | object_detection | dictionary | optional | see Camera object detection config | Overrides the global object_detection config | | zones | list | optional | see Zones config | Allows you to specify zones to further filter detections | | publish_image | bool | false | true/false | If enabled, Viseron will publish an image to MQTT with drawn zones, objects, motion and masks.
Note: this will use some extra CPU and should probably only be used for debugging | | ffmpeg_loglevel | str | optional | quiet, panic, fatal, error, warning, info, verbose, debug, trace | Sets the loglevel for ffmpeg.
Should only be used in debugging purposes. | | ffmpeg_recoverable_errors | list | optional | a list of strings | ffmpeg sometimes print errors that are not fatal.
If you get errors like Error starting decoder pipe!, see below for details. | | logging | dictionary | optional | see Logging | Overrides the global log settings for this camera.
This affects all logs named lib.nvr.<camera name>.* and lib.*.<camera name> |

Default ffmpeg decoder command

A default ffmpeg decoder command is generated, which varies a bit depending on the Docker container you use,

ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -stimeout 5000000 -use_wallclock_as_timestamps 1 -vsync 0 -c:v h264_cuvid -rtsp_transport tcp -i rtsp://<username>:<password>@<host>:<port><path> -f rawvideo -pix_fmt nv12 pipe:1

ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -stimeout 5000000 -use_wallclock_as_timestamps 1 -vsync 0 -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -rtsp_transport tcp -i rtsp://<username>:<password>@<host>:<port><path> -f rawvideo -pix_fmt nv12 pipe:1

ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -stimeout 5000000 -use_wallclock_as_timestamps 1 -vsync 0 -c:v h264_mmal -rtsp_transport tcp -i rtsp://<username>:<password>@<host>:<port><path> -f rawvideo -pix_fmt nv12 pipe:1

This means that you do not have to set hwaccel_args unless you have a specific need to change the default command (say you need to change h264_cuvid to hevc_cuvid)

ffmpeg recoverable errors

Sometimes ffmpeg prints errors which are not fatal, such as [h264 @ 0x55b1e115d400] error while decoding MB 0 12, bytestream 114567.
Viseron always performs a sanity check on the ffmpeg decoder command with -loglevel fatal.
If Viseron gets stuck on an error that you believe is not fatal, you can add a subset of that error to ffmpeg_recoverable_errors.
So to ignore the error above you would add this to your configuration:

ffmpeg_recoverable_errors:
  - error while decoding MB

Substream

Using the substream is a great way to reduce the system load from FFmpeg.
When configured, two FFmpeg processes will spawn:
- One that reads the main stream and creates segments for recordings. Codec -c:v copy is used so practically no resources are used.
- One that reads the substream and pipes frames to Viseron for motion/object detection.

To really benefit from this you should reduce the framerate of the substream to match the lowest interval set for either motion or object detection.
As an example:

motion_detection:
  interval: 0.5
object_detection:
  interval: 1

The optimal FPS for this config would be 2, since it would output a frame every 0.5 seconds for the motion detector to consume.

You should also change the resolution to something lower than the main stream to benefit from this.

Camera motion detection

Each setting set here overrides the global motion detection config.

Mask

cameras:
  - name: name
    host: ip
    port: port
    path: /Streaming/Channels/101/
    motion_detection:
      area: 0.07
      mask:
        - points:
            - x: 0
              y: 0
            - x: 250
              y: 0
            - x: 250
              y: 250
            - x: 0
              y: 250
        - points:
            - x: 500
              y: 500
            - x: 1000
              y: 500
            - x: 1000
              y: 750
            - x: 300
              y: 750

Masks are used to exclude certain areas in the image from triggering motion.
Say you have a camera which is filming some trees. When the wind is blowing, motion will probably be detected.
Draw a mask over these trees and they will no longer trigger said motion.

Camera object detection

Zones

cameras:
  - name: name
    host: ip
    port: port
    path: /Streaming/Channels/101/
    zones:
      - name: zone1
        points:
          - x: 0
            y: 500
          - x: 1920
            y: 500
          - x: 1920
            y: 1080
          - x: 0
            y: 1080
        labels:
          - label: person
            confidence: 0.9
      - name: zone2
        points:
          - x: 0
            y: 0
          - x: 500
            y: 0
          - x: 500
            y: 500
          - x: 0
            y: 500
        labels:
          - label: cat
            confidence: 0.5

Zones are used to define areas in the cameras field of view where you want to look for certain objects(labels).

Say you have a camera facing the sidewalk and have labels setup to look for and record a person.
This would cause Viseron to start recording people who are walking past the camera on the sidewalk. Not ideal.
To remedy this you define a zone which covers only the area that you are actually interested in, excluding the sidewalk.

Points

Points are used to form a polygon for an object detection zone or a motion detection mask.

To easily genereate points you can use a tool like image-map.net.
Just upload an image from your camera and start drawing your zone.
Then click Show me the code! and adapt it to the config format.
Coordinates coords="522,11,729,275,333,603,171,97" should be turned into this:

points:
  - x: 522
    y: 11
  - x: 729
    y: 275
  - x: 333
    y: 603
  - x: 171
    y: 97

Object detection

object_detection:
  type: darknet
  interval: 6
  labels:
    - label: person
      confidence: 0.9
      height_min: 0.1481
      height_max: 0.7
      width_min: 0.0598
      width_max: 0.36
    - label: truck
      confidence: 0.8

The above options are global for all types of detectors.
If loglevel is set to DEBUG, all detected objects will be printed in a statement like this:

[2020-09-29 07:57:23] [lib.nvr.<camera name>.object ] [DEBUG   ] - Objects: [{'label': 'chair', 'confidence': 0.618, 'rel_width': 0.121, 'rel_height': 0.426, 'rel_x1': 0.615, 'rel_y1': 0.423, 'rel_x2': 0.736, 'rel_y2': 0.849}, {'label': 'pottedplant', 'confidence': 0.911, 'rel_width': 0.193, 'rel_height': 0.339, 'rel_x1': 0.805, 'rel_y1': 0.466, 'rel_x2': 0.998, 'rel_y2': 0.805}, {'label': 'pottedplant', 'confidence': 0.786, 'rel_width': 0.065, 'rel_height': 0.168, 'rel_x1': 0.522, 'rel_y1': 0.094, 'rel_x2': 0.587, 'rel_y2': 0.262}, {'label': 'pottedplant', 'confidence': 0.532, 'rel_width': 0.156, 'rel_height': 0.159, 'rel_x1': 0.644, 'rel_y1': 0.317, 'rel_x2': 0.8, 'rel_y2': 0.476}]

Darknet

The above options are specific to the type: darknet detector. The included models are placed inside /detectors/models/darknet folder.
The default model differs a bit per container:

• roflcoopter/viseron-cuda: This one uses YOLOv4 by default.
• roflcoopter/viseron-vaapi: This one uses YOLOv3 by default.
• roflcoopter/viseron: This one uses YOLOv3 by default.

The reason why not all containers are using YOLOv4 is that there are currently some issues with OpenCVs implementation of it.
As soon as this is fixed for the versions of OpenCV that Viseron is using, YOLOv4 will be the standard for all.

The containers using YOLOv3 also has YOLOv3-tiny included in the image.
YOLOv3-tiny can be used to reduce CPU usage, but will hav significantly worse accuracy. If you want to swap to YOLOv3-tiny you can change these configuration options:

object_detection:
  model_path: /detectors/models/darknet/yolov3-tiny.weights
  model_config: /detectors/models/darknet/yolov3-tiny.cfg

EdgeTPU

The above options are specific to the type: edgetpu detector.
The included models are placed inside /detectors/models/edgetpu folder.
There are two models available, one that runs on the EdgeTPU and one the runs on the CPU. If no EdgeTPU is found, Viseron will fallback to use the CPU model instead.

Labels

Labels are used to tell Viseron what objects to look for and keep recordings of.
The available labels depends on what detection model you are using.
For the built in models you can check the label_path file to see which labels that are available, see commands below.

The max/min width/height is used to filter out any unreasonably large/small objects to reduce false positives.

Motion detection

motion_detection:
  interval: 1
  trigger_detector: true
  timeout: true
  max_timeout: 30
  width: 300
  height: 300
  area: 0.1
  frames: 3

Motion detection works by creating a running average of frames, and then comparing the current frame to this average.
If enough changes have occured, motion will be detected.
By using a running average, the "background" image will adjust to daylight, stationary objects etc.
This blogpost from PyImageSearch explains this procedure quite well.

Recorder

recorder:
  lookback: 10
  timeout: 10
  retain: 7
  folder: /recordings

Viseron uses ffmpeg segments to handle recordings.
This means Viseron will write small 5 second segments of the stream to disk, and in case of any recording starting, Viseron will find the appropriate segments and concatenate them together.
The reason for using segments instead of just starting the recorder on an event, is to support to the lookback feature which makes it possible to record before an event actually happened.

ffmpeg -hide_banner -loglevel error -y -protocol_whitelist file,pipe -f concat -safe 0 -i - -c:v copy <outfile.mp4>

If you want to re-encode the video you can choose codec, filter_args and optionally hwaccel_args.
To place the segments in memory instead of writing to disk, you can mount a tmpfs disk in the container.

docker run --rm \
-v <recordings path>:/recordings \
-v <config path>:/config \
-v /etc/localtime:/etc/localtime:ro \
--tmpfs /tmp \
--name viseron \
roflcoopter/viseron:latest

version: "2.4"

services:
  viseron:
    image: roflcoopter/viseron:latest
    container_name: viseron
    volumes:
      - <recordings path>:/recordings
      - <config path>:/config
      - /etc/localtime:/etc/localtime:ro
    tmpfs:
      - /tmp

recorder:
  segments_folder: /tmp

Thumbnail

The default location for the thumbnail if save_to_disk: true is /recordings/{camera_name}/latest_thumbnail.jpg

Post Processors

post_processors:
  face_recognition:
    type: dlib
    expire_after: 10
  logging:
    level: info

Post processors are used when you want to perform some kind of action when a specific object is detected.
Right now the only implemented post processor is face recognition. In the future more of these post processors will be added (ALPR) along with the ability to create your own custom post processors.

Face Recognition

On startup images are read from face_recognition_path and a model is trained to recognize these faces.
The folder structure of the faces folder is very strict. Here is an example of the default one:

/config
|-- face_recognition
|   `-- faces
|       |-- person1
|       |   |-- image_of_person1_1.jpg
|       |   |-- image_of_person1_2.png
|       |   `-- image_of_person1_3.jpg
|       `-- person2
|       |   |-- image_of_person2_1.jpeg
|       |   `-- image_of_person2_2.jpg

MQTT

mqtt:
  broker: mqtt_broker.lan
  port: 1883
  username: user
  password: pass

Topics for each camera

{"state": "scanning_for_objects", "attributes": {"last_recording_start": <timestamp>, "last_recording_end": <timestamp>}}

{"state": "on", "attributes": {}}

{"state": "on", "attributes": {"objects": [{"label": "person", "confidence": 0.961, "rel_width": 0.196, "rel_height": 0.359, "rel_x1": 0.804, "rel_y1": 0.47, "rel_x2": 1.0, "rel_y2": 0.829}]}}

{"state": "on", "attributes": {"count": 2}}

{"state": "on", "attributes": {}}

{"state": "on", "attributes": {"objects": [{"label": "person", "confidence": 0.961, "rel_width": 0.196, "rel_height": 0.359, "rel_x1": 0.804, "rel_y1": 0.47, "rel_x2": 1.0, "rel_y2": 0.829}]}}

{"state": "on", "attributes": {"count": 2}}

Topics for each Viseron instance

{"state": "on", "attributes": {}}

All MQTT topics are largely inspired by Home Assistants way of organizing entities.

Home Assistant MQTT Discovery

Viseron integrates into Home Assistant using MQTT discovery and is enabled by default if you configure MQTT.
Viseron will create a number of entities depending on your configuration.

Cameras
A variable amount of cameras will be created based on your configuration.

1. A camera entity to debug zones, masks, objects and motion.
   Images are only sent to this topic if publish_image: true
2. If send_to_mqtt under recorder is set to true , a camera entity named camera.{client_id from mqtt config}_{mqtt_name from camera config}_latest_thumbnail is created

Sensor
A sensor entity is created for each camera which indicates the status of Viseron. The state is set to recording, scanning_for_motion or scanning_for_objects depending on the situation.

Binary Sensors
A variable amount of binary sensors will be created based on your configuration.

1. A binary sensor showing if any tracked object is in view.
2. A binary sensor for each tracked object showing if the label is in view.
3. A binary sensor for each zone showing if any tracked object is in the zone.
4. A binary sensor for each tracked object in a zone showing if the label is in the zone.
5. A binary sensor showing if motion is detected.
6. A binary sensor showing if a face is detected.

Switch
A switch entity will be created for each camera.
The switch is used to arm/disarm a camera. When disarmed, no system resources are used for the camera.

Logging

logging:
  level: debug

Secrets

Any value in config.yaml can be substituted with secrets stored in secrets.yaml.
This can be used to remove any private information from your config.yaml to make it easier to share your config.yaml with others.

The secrets.yaml is expected to be in the same folder as config.yaml.
The full path needs to be /config/secrets.yaml.

Here is a simple usage example.
Contents of /config/secrets.yaml:

camera_ip: 192.168.1.2
username: coolusername
password: supersecretpassword

Contents of /config/config.yaml:

cameras:
  - name: Front Door
    host: !secret camera_ip
    username: !secret username
    password: !secret password

Benchmarks

Here I will show you the system load on a few different machines/configs.
All examples are with one camera running 1920x1080 at 6 FPS.
Motion and object detection running at a 1 second interval.

Intel i3-9350K CPU @ 4.00GHz 4 cores with Nvidia GTX1660 Ti | Process | Load on one core | When | | ----- | -----| ---- | | ffmpeg | ~5-6% | Continously | | viseron | ~1.3-3% | Scanning for motion only | | viseron | ~7.6-9% | Scanning for objects only | | viseron | ~8.6-9.3% | Scanning for motion and objects |

Intel NUC NUC7i5BNH (Intel i5-7260U CPU @ 2.20GHz 2 cores) using VAAPI and OpenCL | Process | Load on one core | When | | ----- | -----| ---- | | ffmpeg | ~8% | Continously | | viseron | ~3.3% | Scanning for motion only | | viseron | ~7.5% | Scanning for objects only | | viseron | ~8% | Scanning for motion and objects |

Intel NUC NUC7i5BNH (Intel i5-7260U CPU @ 2.20GHz 2 cores) without VAAPI or OpenCL | Process | Load on one core | When | | ----- | -----| ---- | | ffmpeg | ~25% | Continously | | viseron | ~3.3% | Scanning for motion only | | viseron | ~23% | Scanning for objects only | | viseron | ~24% | Scanning for motion and objects |

Tips

• If you are experiencing issues with a camera, I suggest you add debug logging to it and examine the logs

Ideas and upcoming features

• UI

  • Create a UI for configuration and viewing of recordings

• Detectors

  • Pause detection via MQTT
  • Allow specified confidence to override height/width thresholds
  • Dynamic detection interval, speed up interval when detection happens for all types of detectors
  • Implement an object tracker for detected objects
  • Make it easier to implement custom detectors

• Watchdog Build a watchdog for the camera process

• Recorder

  • Weaving, If detection is triggered close to previous detection, send silent alarm and "weave" the videos together.
  • Dynamic lookback based on motion

• Docker

  • Try to reduce container footprint

https://devblogs.nvidia.com/object-detection-pipeline-gpus/

Donations are very appreciated and will go directly into more hardware for Viseron to support.