Recent Changes – January 2023

• Almost 1400 algorithms are included with an average of 31 lines of code per algorithm

• Python scripts were all moved to the end of the list of project files in Visual Studio’s Project Explorer

  • All the Python scripts begin with “z_” to separate them from the VB.Net algorithms

  • The documentation for using OpenCVB with Python scripts was updated (search below for “Python Interface”)

• OpenCVB is evolving into a “layered” set of algorithms as more algorithms incorporate other algorithms.

  • Use the TreeView button to breakdown the structure of all the contributing algorithms

  • Click on the name of an algorithm in TreeView to see the output for that “layer” of the algorithm

  • Read the section labelled “TreeView” below for the details and images.

• RCR – RedCloud Recursion creates cells within a cell to help isolate surfaces.

• A new algorithm group “<Changed recently>” for modules that have been recently modified.

• A list of core “layered” algorithms is available under the heading “Cortico” (needed something unique)

• WarpPerspective_Basics was replaced with a more targeted approach to warping an image

• A monthly history of changes is included at the bottom of this document

Profile_Derivative: A new series of algorithms was added to work with the contour of RedCloud cells. In this example some key points on the contour of a cell are explored. The upper left image outlines in yellow the selected RedCloud cell in the RGB image. The upper right image shows the RedCloud_Basics output (click to select another cell.) The lower left image shows the derivative of the contour in depth with yellow highlighting where contour points are closest to the camera and blue shows where contour points are farther away from the camera. The information in the lower right image shows the point cloud coordinates of the rightmost, leftmost, highest, lowest, closest and farthest points (see the key in the lower right image for color definitions.)

Introduction

There is no better documentation of an algorithm than a working example. This is especially true for computer vision where the output is visual and often self-explanatory. Now imagine well over 1000 OpenCV examples in a single app where each algorithm is less than a page of code and in a familiar language. And each algorithm is just the algorithm without baggage from a user interface or environment. Each algorithm is designed to be reused in other algorithms so variations can be easily built.

A full installation can take about 30-50 minutes using the 1-step “Update_All.bat” file discussed in the “Installation” section below. But there is no obligation to install needed libraries just to read the code for an algorithm. Open the OpenCVB.sln file after download and inspect the code in the C++, C#, VB.Net or Python. Each algorithm gets a standardized presentation of all the data from any of the RGBZ cameras listed below.

However, a full installation is recommended. An algorithm may fit in one page of code and reading the code is one way to review them but understanding the algorithms is a lot faster and easier when the output is visualized by running it. The output is often self-documenting or a natural representation of the algorithm’s intent.

The basic layout of OpenCVB is shown below. Any of the algorithms can be selected from the first combo box at the top of the form. The second combo box is used to select an algorithm group. The default grouping is “<All>”. There are a variety of other special groupings that select, for example, all Python or all C++ algorithms.

The “Sample Results” section below provides numerous additional examples with explanations.

The Objective

The objective is to solve many small computer vision problems and do so in a way that enables any of the solutions to be reused. The result is a toolkit for solving incrementally bigger problems. The philosophy behind this approach is that human vision is not computationally intensive but is built on many, usually trivial algorithms working together. Computer vision problems are not huge; there are just an unmanageable number of them. Combining many reusable algorithms into a single app is the motivation for OpenCVB’s organization.

OpenCVB is targeting only cameras that produce depth and color and have an IMU to detect gravity and motion. These newer cameras have prompted a review of existing vision algorithms to see how they can be improved if depth and gravity are known. To enable revisiting existing algorithms, this software provides a single application that can run OpenCV algorithms on any of the cameras listed above. Adding more cameras is a multiplier. If there are 1000 algorithms and 6 cameras, testing all of them requires 6000 tests which is the reason for the integrated regression testing. If the different resolutions are added, the multiplier and the need for regression testing is even greater.

There are many computer vision examples on the web but too often something is missing, or setup is difficult. OpenCVB is designed to collect algorithms into a single application and guarantee that each will build and run. In addition, software automation and aids simplify the process of adding variants and experiments. OpenCVB is more work to install than a typical app but it has been reduced to a few steps using standard (and free) packages.

The languages used are those often found in OpenCV projects - C++, C#, and Python - but also include VB.Net, hence the name OpenCVB. OpenCVB also provides access to multiple libraries - OpenCV, OpenCVSharp, OpenGL, Open3D, MatPlotLib, Dlib, NumPy, NAudio, and OpenMP. And OpenCVB enables many possible image representations - 3D, bitmaps, ply format, plots, bar charts, spreadsheets, and text. Examples of all of these representations are included in OpenCVB.

Making these languages and libraries available while using the same infrastructure shaped a standardized class for OpenCVB algorithms. Implementing hundreds of examples with the same reusable class structure has confirmed the approach is useful. The result is a starting point to add depth and explore its usage with OpenCV.

There are other objectives. Convolutions combined with neural nets (CNN’s) are a successful approach to computer vision. CNN’s detect differences within a set of images and identify content surprisingly well. OpenCVB is a pathway to search for more and better features than convolutions, features that are measured, objective, and essential. Depth, infrared, gravity, and camera motion are the kind of objective features that can enhance almost any imaging algorithm.

And what if all cameras had depth and an IMU? Making this assumption explains why only a few cameras from Intel, Microsoft, and others are currently supported. The data from each camera – color, depth, point cloud, and IMU data - is presented to all the algorithms in the same standardized format. More cameras with depth are expected to arrive and integration with OpenCVB is likely to follow. OpenCVB is an opportunity to experiment with the features of these cameras and apply the same algorithm to all cameras.

The algorithms are notably short, almost always less than a page of code, labelled reasonably well, easily searched, and easily combined, while often providing links in the code to online documentation and versions for other platforms. Many downloadable algorithms are encumbered by environmental considerations that can obscure the meaning or context of an algorithm. All the algorithms here contain just the algorithm separate from any camera dependencies and will work with each of the supported cameras. Isolating just the algorithm functionality enables easy adaptation to other environments or platforms.

Before You Start

Here are the pre-install requirements:

• Windows 10

• Visual Studio Community Edition (free)

• A recent Python installation (also free) and be sure it is in the path

• Any one of the following RGBZ cameras:

  • Microsoft Kinect for Azure

  • Intel RealSense D435i

  • StereoLabs ZED2

  • Mynt Eye D 1000

  • Intel RealSense D455 – the latest in the series of Intel RealSense cameras

  • Luxonis (OpenCV) Oak-D or Oak-D Lite camera (Oak-D Lite has no IMU though.)

All of the above cameras have an IMU (Inertial Measurement Unit.) The Microsoft Kinect for Azure has the best depth accuracy but requires more power and is not as portable as the Intel cameras. All the cameras use USB-C to provide data to the host platform. A brief comparison of each camera is provided in Addendum 1.

Download and install the following software. Each is free and easily downloaded for Windows 10:

• Microsoft Visual Studio Community Edition

  • Download: https://visualstudio.microsoft.com/downloads/

• CMAKE 3.0 or later

  • https://cmake.org/download/

• TortoiseGit and Git

  • https://tortoisegit.org/

  • https://git-scm.com/downloads

Installation

Installation is not as simple as opening the OpenCVB.sln file but it is not much more than that.

• Run the “Update_All.bat” script that comes with OpenCVB. It will download and run CMake for needed libraries.

  • OpenCVB tree will occupy about 25Gb of disk space – plan accordingly. The process can take 30-50 minutes.

• Download the Kinect4Azure proprietary binaries (needed even if you don’t have the Microsoft camera):

  • https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/usage.md

• Open the OpenCVB.sln and set the “OpenCVB” project as the “Startup Project” if not already set.

• There are some “.bat” files to simplify the process of updating external components of OpenCVB:

  • “Update_librealsense.bat” will download, configure, and build the librealsense library that is essential to OpenCVB.

  • “Update_OpenCV.bat” will update the OpenCV library with the latest version.

  • “Update_Azure4Kinect.bat” will update the open source portion of the Kinect4Azure camera.

  • Optionally, there are some scripts to install support for OakD cameras, Eigen, and Python.

Why are there no official releases of OpenCVB? The repository is the release. The infrastructure is solid at this point (the exceptions are rare and transitory.) Any problems that arise are easily avoided because they will be confined to an algorithm. OpenCVB regression tests are continuously running so even individual algorithm problems will be apparent shortly. If any problem is encountered, download the latest and if it is still there, submit a pull request.

Optionally Install Additional Cameras

Support for some optional cameras can be added.

• For the StereoLabs ZED 2 camera (released Q1 2020), install the StereoLabs SDK from

  • https://www.stereolabs.com/

• For the Mynt Eye D 1000 camera, download the SDK from:

  • https://mynt-eye-d-sdk.readthedocs.io/en/latest/sdk/install_win_exe.html

• For Luxonis (OpenCV) Oak-D cameras, run “Update_OakD.bat” and uncomment OPENCV_OAKD in cameraDefines.hpp.

Edit “Cameras/CameraDefines.hpp” file to add OpenCVB’s support for StereoLabs Zed 2 or Mynt Eye D 1000 support.

Trouble-Shooting a New Install

Some typical problems with new installations:

• If OpenCVB installation fails, there are simple ways to determine what needs to be changed.

  • Look in the “Build” directory and if the .sln file is missing, run CMake-gui to figure out why the component failed.

    • Kinect 4 Azure, librealsense, and OpenCV all should have a .sln file in the Build directory

      • <OpenCVB Dir>/Azure-Kinect-Sensor-SDK/Build/K4A.sln

      • <OpenCVB Dir>/librealsense/Build/librealsense2.sln

      • <OpenCVB Dir>/opencv/Build/opencv.sln

    • If not, the problem is in the CMake.

  • Review the Windows Environmental display for defined variables that may impact a CMake

    • “OpenCV_Dir” variable (once required) should be removed

    • Review the “Path” – both global and user values – and remove other Visual Studio compilers

  • Figure out which component is failing:

    • Test “Update_All.bat” runs “Update_librealsense.bat”, “Update_Azure4Kinect.bat”, and “Update_OpenCV.bat” individually

  • Post any problems encountered. The problem will have the highest priority automatically.

• Camera Failure: check the camera installation by testing the examples provided by the camera vendor. Did the Kinect4Azure support get upgraded recently? Post if some configuration problems prevent the camera from working in OpenCVB.

• “Mark of the Web” problem. Some of the .resx files are marked as coming from the web as part of the download of a release. It is not clear how to avoid this but the solution is to right click the offending file, select “Properties”, and click the “Unblock” checkbox in the General tab of the properties. Any suggestions on how to avoid this problem would be welcome.

• Python Scripts Fail: if any Python scripts fail, open a command line window and run the script. The error messages will indicate which package is missing from your Python installation.

• Link problems: the C++ code in OpenCVB relies on PragmaLibs.h which is automatically created as part of the build process. “PragmaLibs.h” defines the names of the OpenCV libraries. It should automatically be updated with the current OpenCV version that is in use. If not, run the “VersionUpdates” application included in the OpenCVB.sln file. “VersionUpdates” will update the names of the files from OpenCV to be linked into the OpenCVB interfaces. Open the “PragmaLibs.h” file to see the current version of OpenCV that is expected to be present.

Building New Experiments With Snippets

OpenCVB is a WinForms application and most of the algorithms were written using Microsoft's managed code but C++ and Python examples are provided as well. Python examples don’t require a VB.Net wrapper. There are several VB.Net examples that demonstrate how to move images to Python and get results back into the OpenCVB user interface (see “AddWeighted_PS.py” as an example that is only a few lines of code.)

For C++, C#, and VB.Net writing a new experiment requires a new class to be added in the “VB_Classes” project. OpenCVB will automatically detect the new class and present it in the user interface. The code is self-aware in this regard – the UI_Generator project is invoked in a pre-compile step for the VB_Classes project.

Code “snippets” are provided to accelerate development of new VB.Net, OpenGL, and C++ algorithms. To use any snippets, first install them in Visual Studio:

• Click the menu “Tools/Code Snippets Manager”

• Select “Basic” as the Language.

• Add the “<OpenCVB Dir>/OpenCVB.snippets” directory.

• Access the code snippets with a right-click in the VB.Net code, select “Snippet/Insert Snippet” and select “OpenCVB.snippets”.

• Even C++ algorithms can use snippets, but each C++ algorithm has a VB.Net entry that includes both the C++ and the VB.Net code in the snippet. The C++ portion is to be cut and pasted anywhere in OpenCVB’s “CPP_Classes” Visual Studio project.

Experimental Subsets

The complete list of algorithms may be grouped into smaller subsets to study some shared API. Algorithm subsets are selected through the Subset Combo Box in the toolbar (indicated below.) The list of subsets is built from all the OpenCVB algorithm names and all the OpenCV API’s referenced. For instance, selecting “Threshold” in the Subset Combo Box, will update the Algorithm Combo Box with all the algorithms that use the OpenCV “Threshold” API.

In the image above, the Subset Combo Box selects all algorithms using the “Edges_Sobel” algorithm. The Algorithm Combo Box shows the “Gradient_Depth” algorithm, one of the algorithms that uses “Edges_Sobel”. When the Subset Combo Box is set to “<All>”, the Algorithm Combo Box will contain all the algorithms in OpenCVB.

The ability to create subsets from the hundreds of algorithms makes it easier to study examples of an OpenCV API or OpenCVB algorithm usage. In addition, the Subset Combo Box has several higher-level groupings. For example, “<Python>” selects all Python algorithms. Or “<OpenGL>” selects only the OpenGL algorithms. The “<All>” entry in the Subset Combo Box will restore the complete list of algorithms.

Regression Testing All Experiments

Testing is integrated into OpenCVB. Clicking the icon below runs through a checklist of all the algorithms on all the attached depth cameras attached to the system at all the supported resolutions. The duration of each test can be selected in the Global Options dialog.

When using a subset of the algorithms, the “Test All” button will test only the algorithms in the subset. This can be useful when changing an algorithm that is reused frequently by other algorithms. For instance, if the Edges_Sobel algorithm is changed, select “Edges_Sobel” in the Subset Combo Box, then click the regression test button to visually review each algorithm using the updated Edges_Sobel.

One side benefit of the “Test All” feature is that is provides a way to visually review all the algorithms. When used in combination with the subset feature, it can make it easier to search for a desired effect.

Why VB.Net?

VB.Net is not a language associated with computer vision algorithms. But the abundance of examples in OpenCVB suggests this may be an oversight. Even the seasoned developer should recognize what is obvious to the beginner: VB.Net can keep the code simple to read and write. Papers and articles on software often use pseudo-code to present an algorithm. In many respects, VB.Net code resembles pseudo-code except it is an actual working implementation of the algorithm.

VB.Net provides a full-featured language just like C# with lambda functions and multi-threading except VB.Net uses only a subset of the special keys available on the keyboard. Contrasted with Python or C++, VB.Net need make no apologies for using real words instead of the keyboard hieroglyphics defined in Python or C++. VB.Net includes user interface tools that are flexible and complete (check boxes, radio buttons, sliders, TrueType fonts, and much more) - options missing from OpenCV's popular HighGUI library. (All existing HighGUI interfaces are still supported in OpenCVB.)

The main caution in using VB.Net is to treat it as a scripting language like Python. Most of the algorithms avoid pixel-by-pixel details – VB.Net can be detailed but it will be slower than optimized C++. Usually, OpenCVB is doing most of the real work in optimized C++ through the OpenCVSharp interface. Most algorithms run reasonably fast even in Debug mode because the release version of OpenCVSharp is active when OpenCVB is in Debug mode.

Critics will point out that a Windows 10 app using VB.Net is not easily portable to other platforms but the entire OpenCVB application does not need to be ported to other platforms. Only individual algorithms are likely to be ported after they are debugged and polished, Most OpenCVB algorithms consist almost entirely of OpenCV APIs which are available everywhere. OpenCVB’s value lies in the ability to experiment and test an OpenCV algorithm. After the prototype is complete the algorithm can be transferred to a different platform.

Camera Interface

All the camera code is isolated in the “camera” class – see cameraRS2.vb, cameraKinect.vb, cameraMynt.vb, cameraOakDPython.vb, or cameraZed2.vb. There are no references to camera interfaces anywhere else in the code. Isolating the camera support from the algorithms strips the algorithm code to just the essential OpenCV API’s needed.

For example, the Kinect for Azure camera support is isolated to the cameraKinect.vb class and a supporting Kinect4Azure DLL that provides the interface code to the Kinect for Azure libraries. Since there is likely to be little interest in debugging the Kinect4Azure DLL, the Release version is used even in the Debug configuration. If it is necessary to debug the camera interface, set any Build Configuration entries to the Debug version. Using Release versions naturally enables a higher framerate. As a result, the VB.Net code – which is usually run in Debug mode - is almost as fast as the Release configuration.

OpenGL Interface

There have been several attempts to provide OpenGL interfaces into managed code, but none is used here. OpenGL is simply run in a separate process. To accommodate running separately, a named-pipe moves the image data to the separate process and a memory-mapped file provides a control interface. The result is both robust and economical leaving the OpenGL C++ code independent of camera hardware specifics. The VB.Net code for the OpenGL interface is less than a page and does not require much memory or CPU usage. The OpenGL C++ code provided with OpenCVB is customized for specific applications in a format that should be familiar to OpenGL developers. There are several examples – displaying RGB and Depth, 3D histograms, 3D drawing, and IMU usage. A code snippet (See ‘Building New Experiments With Snippets’ above) provides everything needed to add a new OpenGL algorithm that will consume RGB and a point cloud.

NOTE: it is easy to forget to include any new OpenGL project in the Project Dependencies. This can be confusing because the new project will not build automatically when restarting. The OpenCVB Project Dependencies need to be updated whenever a new OpenGL application is added to the OpenCVB solution. To update dependencies, select “Project/Dependencies” from the Visual Studio menu and make sure that the “OpenCVB” project depends on any new OpenGL projects. This ensures that the new project will always be rebuilt when OpenCVB is restarted.

Python Interface

OpenCV has numerous examples of Python scripts and Python is often used for computer vision experiments. To add a new Python script for use with OpenCVB, add the Python script to the VB_Classes project so any changes to a Python script will automatically show the new or renamed Python files in the user interface. Python scripts don’t require a VB.Net wrapper – just add a new script to the VB_Classes Project – and it will appear in the user interface.

Python scripts can get a stream of images from the camera and return resulting images. There are numerous examples of how to do this: see z_AddWeighted_PS.py or Camshift_PS.py for the simplest examples. The “_PS” suffix is an OpenCVB convention that indicates it is a Python Streaming script that expects a stream of RGB and Depth images and will return images. NOTE: The Python script name MUST end with “_PS.py” to stream images to and from Python code. To see the list of all the Python Streaming scripts, select the pre-defined subset group called “<PyStream>”.

Some care is required when first using an OpenCVB “PyStream” script. The algorithm thread is writing to a pipe received by the Python script. However, if the right version of Python is not set in OpenCVB or some of the packages are missing, it will appear to hang the algorithm thread in OpenCVB. See the section below on setting up Python packages.

Python Installation

If any of the Python scripts fail, open a command line window and run the script. The error messages will indicate what is wrong. The most likely problem is a missing package. Use Visual Studio’s “Tools/Python” menu to manage your Python packages.

Python scripts are run in a separate address space when invoked by OpenCVB just like OpenGL. Visual Studio’s Python debugging environment is not available directly when running OpenCVB. When a Python script fails in OpenCVB, it will disappear, but it may be tested in a command line window to reveal the error messages.

Creating C++ “IncludeOnly” Algorithms

There are some tools included with OpenCVB which make it a lot easier to add C++ algorithms. The toolbar includes 2 icons for this purpose:

Toolbar additions: The blue “+” sign is used to add new algorithms (more below.) The yellow “T” is an interface to the translator to move the current algorithm from VB.Net to C++. The translator is OpenCVB-specific and relies on the patterns and structure of the VB.Net algorithms.

Adding a new algorithms to OpenCVB can take any of the several forms outlined below but the bottom button, labeled “Add ‘IncludeOnly’ C++ algorithm, is the latest addition to the list.

Starting New OpenCVB algorithms: The form above is accessed by clicking on the blue “+” sign in the main toolbar for OpenCVB. The bottom button is the most recent addition. It assists in creating the necessary entries to start a new C++ “IncludeOnly” algorithm. Algorithms created in this interface will be available to any C++ application just by adding an include file. An imgui application using the include-only file is available in the OpenCVB solution.

The quickest way to add a new C++ is to use the “VB_to_CPP” project included in the OpenCVB solution.

• Run OpenCVB, click the blue “+” icon, and click the “Add ‘IncludeOnly’ C++ algorithm” button in the figure above.

• Set the “Startup Project” in OpenCVB’s Visual Studio solution and switch to “VB_to_CPP”.

  • The output of “VB_to_CPP” includes the VB.Net version on one side and the C++ equivalent on the other side.

• Copy the C++ output text to the “CPP_IncludeOnly.h” file included in “CPP_Classes” project in the OpenCVB solution.

• The algorithm is translated into C++ but there are usually errors. To fix these errors, there are 2 methods:

  • Tweak the VB_to_CPP.vb code to update this and all future VB.Net translations

  • Or: tweak the C++ code in “CPP_IncludeOnly.h” to update only the current algorithm.

• Rerun the “VB_to_CPP” application until the C++ code compiles and works.

• Currently, there is no support for options for C++ IncludeOnly algorithms

  • Options are dependent on the user interface

  • Variables that are typically options will be marked with “options_...” at the start of the variable name.

  • The option values in the VB.Net algorithm can be used to fine-tune and placed in hard-coded values in the C++ code.

There are over 1000 VB.Net algorithms included in OpenCVB and there are currently less than 100 C++ translated algorithms. It is expected that future releases of OpenCVB will include more and more of the C++ editions of the algorithm. New VB.Net algorithms can be translated into C++ as part of the code review for the algorithm.

Visual Studio C++ Debugging

The Visual Studio projects can be configured to simultaneously debug both managed and unmanaged code seamlessly. The property “Enable Native Code Debugging” for the managed projects controls whether C# or VB.Net code will step into C++ code while debugging. However, leaving that property enabled all the time means that the OpenCVB will take longer to start – approximately 5 seconds vs. 3 seconds on a higher-end system. The default is to leave the “Enable Native Code Debugging” property off so OpenCVB will load faster. Of course, if there is a problem in the C++ code that is best handled with a debug session, turn on the “Enable Native Code Debugging” property in the OpenCVB VB.Net project and invoke the algorithm requiring C++ debugging.

Record and Playback

The ability to record and playback is provided with OpenCVB – see the algorithms labeled “Replay_Record” and “Replay_Play”. RGB, Depth, point cloud, and IMU data are written to the recording file. Any algorithm that normally runs with the live camera input and IMU data can be run with recorded data. Use the “Subset Combo Box” to select the option: “<All using recorded data>”. Running the regression tests with that setting will run all the algorithms with recorded data instead of a live camera feed.

StereoLabs Zed 2 Support

The StereoLabs Zed 2 camera is supported but the support is turned off by default to simplify installation. To enable this support:

• Download the StereoLabs SDK from https://www.stereolabs.com/developers/release/

• Edit the CameraDefines.hpp file to turn on the interface to the Zed 2 camera.

The Zed 2 camera support is always installed in C:\Program Files (x86)\ZED SDK (regardless of the version) so no additional changes are required to the supporting C++ project.

Mynt Eye D 1000 Support

The Mynt Eye D 1000 camera is supported but the support is turned off by default to simplify installation. To enable this support:

• Download the Mynt SDK from https://mynt-eye-d-sdk.readthedocs.io/en/latest/sdk/install_win_exe.html

• Edit the CameraDefines.hpp file to turn on the interface to the Zed 2 camera.

The Mynt D SDK creates a system environmental variable MYNTEYED_SDK_ROOT that allows the OpenCVB build to locate the Mynt D camera support no matter where it was installed.

TreeView

The Treeview shows the different layers of the algorithm and how it was built from other OpenCVB algorithms. Here is a simple algorithm tree view that shows how the KNN_PointTracker algorithm was built:

The tree above describes how the algorithm calls KNN_1_to_1 and how KNN_1_to_1 calls KNN_BasicsQT and so on. Clicking on any of the tree entries will show the output of the selected algorithm in OpenCVB’s output (if active.) This is useful to understanding the various steps needed to build the output. All output will show up in OpenCVB’s main form in the bottom 2 images. Play with this when running OpenCVB. It is a fun feature and helps increase understanding of the composition of increasingly complex algorithms.

Pixel Viewer

The Pixel Viewer allows detailed inspection of any of the 4 OpenCVB images. The dimensions of the rectangle depend on the window that the user has sized for the Pixel Viewer output – the larger the window, the larger the rectangle. Supported formats include CV_8U, CV_8UC3, CV_32F, and CV_32FC3 but more could be added if needed.

Continuous Release

Evolution would have been easier for Darwin to discover if nature had provided versioning. Software applications usually have a version identifier, but OpenCVB is an exception. The infrastructure to support all the algorithms is reliable so the only problems that should show up are in a specific algorithm and can be easily avoided. OpenCVB is a small project and only the repository version is really maintained. If there is a problem after downloading the current OpenCVB tree, post a complaint or pull request. Infrastructure problems – those that impact multiple algorithms – are always the highest priority.

How to Contribute

Adding more examples is the goal and that is the best way to contribute to this effort. There are plenty of examples to use as a model but there are also snippets that assist in the process of adding new examples (See “Building New Experiments with Snippets” above.) Any pull request that adds an algorithm will be welcome and quickly reviewed. Changing OpenCVB’s infrastructure – not the algorithms – is discouraged but always welcome. It is much more difficult to test infrastructure that can cripple all the algorithms. Adding support for a new camera is a notable exception. New cameras should have depth, point cloud support, and an IMU.

Sample Results

The following images are a preview of selected algorithms and their output.

MSER_Basics: The MSER (Maximally Stable Extremal Regions) algorithm identifies regions which look like objects. The right image shows all the objects identified – with all the duplicates and small regions. The left image shows the objects identified after consolidating the overlapping rectangles and filtering for the smaller regions.

PointCloud_Basics: The left is a side-view of the depth data that has been rotated using the IMU gravity vector. The rotation facilitates finding the horizontal lines in the left image that identify the floor and ceiling in the image. If the floor or ceiling is not visible, the yellow lines are not shown. The right image is the same side-view of the depth data before it was rotated with the IMU gravity vector.

Structured_FloorCeiling: The left highlights the floor that has been determined from the depth. The right image shows side view with the line detector output in yellow. The white in the bottom left is the plane the yellow bar depicts projected into the image coordinates. When moving the mouse on the screen, the slice of the side view shown in yellow in right image is back-projected into the color image on the left.

Benford_JPEG: Benford’s Law is an interesting empirical hypothesis. The plot above is actually 2 plots – a count of the leading digits pulled from a compressed image combined with a plot of the expected distribution of those leading digits derived from Benford’s Law. The values are so close that you cannot see the difference. The code contains some links explaining Benford’s Law and examples of when it works and when it does not.

GeneticDrawing_Basics: A genetic drawing experiment that translates any image or camera screen grab into a painting. The algorithm randomly alters DNA sequences describing brush strokes.

Histogram_Valleys: Histogram Valleys are used to create clusters in depth data. The bottom left is the histogram showing the different clusters. The bottom right is the back-projection of the different clusters into the depth image using the same colors as the histogram.

Blob_DepthClusters: Using the histogram to create clusters (see ‘Histogram Valleys’ above) allows an algorithm to segment an entire image (see right image), creating blobs that can be measured and tracked. The black segments have no depth.

IMU_FrameTime: The IMU timestamp provides clues to the relationship between the IMU capture and the image capture (the devices are on different clocks.) The image capture triggers an interrupt but how long ago did the IMU capture the pose data or Gyro/Acceleration? The plot in the right shows the actual frame durations for the IMU and host interrupts following an image capture. The IMU and host interrupt frame times are used to estimate the delay from the IMU capture to the image capture. The blue dots are the actual IMU capture frame times, the green is the host image capture frame time, and the red is the estimated delay from the IMU capture to the image capture.

MeanShift_Depth: Here the MeanShift algorithm is used to track the blob in the rectangle of the left image. The left image shows the objects tracked by the algorithm while the right shows the different histograms used to identify the object to track. The histogram is for the hue portion of the HSV format of the image.

KAZE_LeftAligned_CS: KAZE feature detection algorithm matching points on the left and right infrared images from the Intel RealSense 3D camera. The right image matches a point in the left image and shows the distance with a yellow line. Note that points in the background have only a small shift while those in the foreground have a much larger shift.

Feature_Line: The longest line in the image is detected and tracked. OpenCV’s MatchTemplate API is used to track the bottom and top of the line and the rectangles are used to determine the depth at the endpoints and the mean depth. The right image is mostly diagnostics to help debug the algorithm.

Mat_4Click: The Mat_4Click algorithm is used to display 4 output images. Clicking on any one of the 4 images will display it in the right image.

Kalman_MousePredict: A well-known use of the Kalman filter is to predict the mouse movement as shown here. The red line is the actual mouse movement, and the white line is the predicted mouse movement. Kalman is used often enough in OpenCVB that it is toggled on and off using a global parameter.

Edge_Preserving: Compare the original color and depth images with the image right below it to see the impact of an edge-preserving filter applied to both.

Hough_Lines_MT: The bottom images are the output of a multi-threaded Hough lines algorithm. The bottom right image identifies featureless RGB surfaces (shown as solid in the lower right image) as well as the lines found in each cell of the grid.

OilPaint_Cartoon: Algorithms may optionally work on only a selected region – see task.DrawRect variable and its use. Here the oil paint effect is applied only to the region selected by drawing a rectangle on the output image (lower left.) The lower right image is an intermediate stage with only edges. All algorithms may also review the pixel details at any location with the pixel viewer icon in the toolbar (the Microscope icon.)

OpenGL_Callbacks: The OpenGL window is controlled from the VB.Net user interface but is run in a separate address space. The OpenGL axes are represented in this image as well.

DNN_Basics: This dnn Caffe example in VB.Net is a reinterpretation of the C++ sample program distributed with the Intel librealsense library. The application is meant to detect and highlight different objects shown in the yellow box (right). The algorithm requires square input (shown centered in the left image.)

Vignetting_Devignetting: This is an example of a compound algorithm. The left image has been vignetted – highlighting the center of the image. The right image has been taken the left image and devignetted it. The algorithm is only a few lines because it reuses the Vignetting_Basics and the Vignetting_Removal algorithms.

Emax_PointTracker: The Expectation Maximization algorithm learns the different clusters and then predicts the boundaries of those clusters. The left image defines the different clusters (each with a different – and consistent – color.) The right image shows the old center in red and the new center in yellow.

Annealing_MultiThreaded: In this example of multi-threaded simulated annealing shows 4 annealing solutions - the 2 best solutions with 2 worst solutions. The log in the right image shows each thread and their different energy levels, confirming that each is independently searching the solution space. The example was taken directly from the OpenCV examples but was multi-threaded on an Intel Core i9 18-core (36 thread) processor here.

WarpAffine_3Points: First, two estimated triangles are created to surround two sets of random points. Then an affine transform is computed that converts one triangle to the other. The matrix of the computed affine transform is shown of the right triangle. Note that the image is twice the width of other examples. Algorithms have the option to use the bottom 2 output images as a single image. See WarpAffine_3Points for details.

WarpAffine_4Points: First, a randomly oriented rectangle is created. Then a transformation matrix is computed to change the shape and orientation of the color image to the new perspective. The matrix of the computed affine transformation is shown in the left sde of the image.

Kalman_VB_Basics: This example shows the Plot_Basics algorithm which will plot XY values. The image shows the original grayscale mean in blue, the grayscale mean after Kalman in green, and the grayscale average gray after Kalman in red. The green tracks the blue fairly well while the red is significantly different and less representative of the grayscale mean. This example makes it appear that Kalman is superior to just averaging.

Retina_Basics_CPP: Applying the bio-inspired enhancements, the high-dynamic range image is shown in the bottom left while the motion-enhanced image is shown in the bottom right. For instance, the monitor on the desk at the right side of the top left image appears black. The same monitor has much more detail in the bottom left image which shows the reflection that would likely be visible to the human retina.

Chart, radar chart Description automatically generated

Calibrate_ShowExtrinsics.py: OpenCVB supports hundreds of Python examples. In this Python example taken from the OpenCV distribution, the various positions of an etalon are visualized in a 3D matplotlib control..

Gabor_Basics_MT: In this multi-threaded version of the Gabor filter, the bottom left image is the Gabor result while the bottom right figure shows the 32 Gabor kernels used to produce the result on the bottom left. The original RGB image is in the upper left. Depth in the upper right is not used in this example.

Aruco_Test: The Aruco markers are found in an image allowing the image to be transformed using OpenCV’s WarpPerspective transform. This example is from the OpenCVSharp Sample programs.

Sound_GenWaveDisplay: OpenCVB can be used to display audio data using the NAudio package. The interface can play any available audio data or generate synthetic sound in a variety of patterns – pink/white noise, square waves, sawtooth waves, etc. The black vertical bar is tracking the sound currently playing.

OpticalFlow2_PS.py: The motion direction at each location is indicated with an arrow in the left image. In the right image is a different representation of the motion. Perhaps the most important aspect of this example is that the output appears in the OpenCVB user interface and not in a window of the Python application. This example is using the “PyStream” feature of OpenCVB where images are sent to the Python application, processed, and returned to be presented inside OpenCVB. For a simple example of how to use PyStream, see the AddWeighted_PS.py. Each PyStream algorithm has the “_PS.py” suffix.

Undistort_Basics: OpenCV’s InitUndistortRectifyMat and Remap API’s can be used to manipulate the image arbitrarily. Here one of the sliders in the interface has introduced barrel distortion in the lower left image. This is a useful exercise to get a more intuitive understanding of the parameters needed to rectify and undistort an image.

WarpModel: The 100 year-old photos of Prokudin-Gorskii were made with red, green, and blue filters. Combining the 3 images is a useful application of the OpenCV findTransformECC API that can align the images. The option to use gradients for alignment (instead of RGB values) is available.

Future Work

The plan is to continue adding more algorithms. There are numerous published algorithms on the web but there is also the task of combining different algorithms in OpenCVB. The current edition of the code contains examples of compound algorithms and more will arrive in future releases. The code almost enforces reuse because any algorithm with sliders or check boxes encourages reuse rather than duplicate a similar set of sliders and check boxes. The options for combined algorithms are automatically cascaded for easy selection.

Acknowledgements

The list of people who have made OpenCVB possible is long but starts with the OpenCV contributors – particularly, Gary Bradski, Victor Erukhimov, and Vadim Pisarevsky - and Intel’s decision to contribute the code to the open source community. Also, this code would not exist without OpenCVSharp’s managed code interface to OpenCV provided by user “shimat”. There is a further Intel contribution to this software in the form of RealSense cameras – low-cost 3D cameras for the maker community as well as robotics developers and others. RealSense developers Sterling Orsten and Leo Keselman were exceptionally helpful in educating this author. While others may disagree, there is no better platform than the one provided by Microsoft Visual Studio and VB.Net. And Microsoft’s Kinect for Azure camera is a valuable addition to the 3D camera effort. And lastly, it should be obvious that Google’s contribution to this effort was invaluable. Thanks to all the computer vision developers who posted algorithms where Google could find them. From this author’s perspective, the work of all these organizations is like catnip and feathers to a kitten.

MIT License

https://opensource.org/licenses/mit-license.php - explicit license statement

Fremont, California

Fall 2020

Addendum 1: Comparing Cameras

There are 5 supported cameras that generate a point cloud – Intel RealSense D435i, Intel RealSense L515, Microsoft Kinect4Azure, StereoLabs Zed 2, and Mynt Eye D1000. The cameras are setup to measure the distance to the same wall in the same environment. Here is a look at how the cameras were setup:

The cast of characters – (left to right) Intel D435i, Intel L515, StereoLabs Zed 2, Mynt Eye D1000, and Microsoft Kinect 4 Azure. Using the “PointCloud_GVector_TopView”, all the above cameras were used to measure the distance to the wall approximately 2.5 meters from the cameras. The results are below. Not shown above is the latest Intel RealSense camera – the D455. The output is shown below and looks good.

Microsoft Kinect 4 Azure results – these were the best results.

Intel Realsense L515 Lidar camera

Intel RealSense D435i camera. (Distance to the wall looks incorrect.)

StereoLabs Zed 2 camera. FOV looks a little skewed to the right. The problem could be specific to this camera but there was no available second camera to test (the camera is a little expensive.)

Mynt Eye D 1000 camera.

Intel RealSense D455 camera.

Addendum 2: Change Log

Recent Changes - September 2020

• Dropped support for Intel T265 camera (no point cloud) and the Intel RealSense L515 (no IMU). All supported cameras have a point cloud and IMU.

• Tree view – some of the algorithms are a combination of several other algorithms. A tree view was built to display the hierarchy.

• There are now over 750 algorithms implemented.

Recent Changes – December 2020

• Over 800 algorithms – almost all less than a page of code.

• Depth updates are guided by motion – produces more stable 3D images. See Depth_SmoothMin algorithm.

• Recently used algorithms are listed in the menus.

• More snippets to help adding options to existing algorithms.

• Algorithm options are now collected in a single form – easier usage on laptops or smaller screens.

• Intel Realsense cameras are supported in native 640x480 modes (as well as 1280x720.)

Recent Changes – January 2021

• Over 870 algorithms – almost all less than a page of code.

• The new “Best Of” module contains the best example of common techniques. Need an example of contours, look in the BestOf.vb first.

• OpenCV’s new Oak-D camera has arrived. Some python scripts were added for users that have installed it.

  • https://docs.luxonis.com/en/latest/pages/api/ - to get the official support.

• Motion detection is easier to use with an “AllRect” cv.rect that encompass all RGB changes.

• Image segmentation is more stable and consistent from frame to frame. See ImageSeg.vb.

• OptionsCommon.vb defines options common to all algorithms.

• StructuredDepth shows promise as a path to exploiting structured light technology.

• PythonDebug project is now integrated into the OpenCVB.sln. Python debugging is easier.

Recent Changes – February 2021

• Over 900 algorithms – almost all less than a page of code

• New pixel viewer to inspect image pixels in 8UC1, 8UC3, 32F, and 32FC3 formats

• Versioning policy set - The Repository IS The Release - TRISTR

• Improved threading support for switching between camera interfaces

• Oak-D camera support is working – still missing IMU and point cloud support

• VTK support was improved – still optional (it is a lot to install)

• Upgraded to the latest RealSense2, OpenCVSharp, and Kinect4Azure software

• Motion Filtered Data series of algorithms – an attempt at reducing data analysis at input

Recent Changes – March 2021

• Almost 940 algorithms – almost all less than a page of code

• Stream-lined install: no environmental variable, library builds are automated.

• Latest version of the OpenCV library - 4.5.2

• Improved Python support – now using “requirements.txt”

• An experimental Python interface to the LibRealSense2 cameras has been added

• VTK support is being dropped – it is too big and cumbersome. Recommended: Python Pyglet

• Oak-D camera Python interface is present but turned off pending IMU support from the vendor

• “PyStream” support is now a 2-way pipeline. Output can appear in the OpenCVB interface

• Tensorflow database downloads are automated with algorithm “Download_Database”

• Emgu examples removed. LineDetector library removed – it was redundant

• Version 1.0.0 defined and released

Recent Changes – May 2021

• 980 algorithms – almost all less than a page of code

• Global variables introduced – settings that apply to all algorithms, line width, max depth, font size.

  • Global variables are remembered across runs and can be reset to known working values

• Fewer lines of code. Code size dropped about 4000 lines with more algorithms. Average algorithm: 31 lines.

• Algorithms are now ranked by usage (“Reuse Rank”) and “Value Rank”, a graded estimate of algorithm value.

  • Rankings are entries in the Group ComboBox.

• New Survey function to build images of all algorithm output to allow visual searches for desired algorithm.

• Global setting for palette control

• Improved regression testing – all algorithms are tested with each attached camera at all supported resolutions.

• Navigation aids now available – back to previous algorithm, forward to next, and full history.

• Image microscope works even when stream is paused, allowing more detailed image analysis.

• Improved tree view to study how algorithm was constructed from other algorithms.

Recent Changes – July 2021

• Over 1000 algorithms – almost all less than a page of code. Average algorithm is 31 lines of code

• TreeView now shows algorithm cost in addition to algorithm components

• Improved intermediate views – click anywhere in TreeView to see intermediate outputs

• Depth Object algorithm identifies areas of interest 4 different ways with mask and enclosing rectangle

• All algorithms can extend their output to all 4 images (only 2 were available before)

• Upgraded to the latest versions of OpenCV, librealsense, and Kinect4Azure libraries

• Framerate for all cameras upgraded to 60 fps

New Feature Highlight – TreeView

The TreeView now shows the cost of each component in the algorithm, including global algorithms, in the right side of the TreeView. In the list of component costs above, the “Non-Algorithm” time is the largest individual item. Non-Algorithm refers to all the other costs in the OpenCVB application which includes the user interface and the cost of obtaining the camera images and IMU data.

The active algorithm at the time this TreeView was captured was “TimeView_Basics” – also the top entry in the tree view at the left of the image above. TimeView_Basics is the active algorithm, but it obtains the histogram of both the Side and Top Views (see Histogram_TopView2D and Histogram_SideView2D costs) which are projections of the point cloud (see Depth_PointCloud_IMU cost.) The point cloud was rotated using the gravity vector in the IMU_GVector algorithm at a cost of only 1%.

Some of the algorithms above are executed for all algorithms and will be present in every cost analysis. The IMU_IsCameraLevel and IMU_IsCameraStable are low-cost algorithms that make global variables available that can determine if the camera is level or if the camera is moving. OptionsCommon_Depth operates on the depth using the maximum specified range (a global setting in the user interface.) It is run on every frame regardless of the algorithm because depth data is commonly used and an important component in OpenCVB.

At the bottom of the new TreeView form is a checkbox that allows the time to accumulate or be refreshed with only the latest time interval (approximately 1 second.) If the algorithm contains a variety of different approaches, leaving this item unchecked will allow quick review of the cost of each algorithm variation. Leaving the box checked will accumulate all the time used since starting the algorithm.

With this new TreeView, the cost analysis is available for every algorithm in OpenCVB – automatically.

All previous features of the TreeView are still fully supported in the new edition. The tree view controls what output is shown in the user interface. The default output is always the main algorithm – in this case “TimeView_Basics”. Clicking on “Histogram_SideView2D” will show the intermediate output from the Histogram_SideView2D component. When building a new algorithm, clicking through the TreeView can determine which step in the process did not provide the expected output.

Recent Changes – September 2021

• Almost 1100 algorithms – almost all less than a page of code. Average algorithm is 31 lines of code

• Improvements to the TreeView indicate how many cycles are available (see Highlight below.)

• The reduced point cloud predictably divides an image for analysis.

• “Reuse Rank” in algorithm groups shows all algorithms reused at least twice.

• Quarter Resolution option finds bottlenecks without code change.

• First example of using low resolution internally while displaying full resolution

• RGB Depth can be displayed with numerous different palettes. You can create your own.

New Feature Highlight – Is my Algorithm Processor-bound?

Example 1: Room to grow (at left)

Example 2: Saturated processor (at right)

The key to understanding whether an algorithm is processor-bound is provided in the images above. Example 1 shows the algorithm spent over 88% of its time in the “inputSyncWait” algorithm. This is the algorithm task function that waits for additional input from the camera task. Note that the frame rate is 66 FPS. In Example 2, the “inputSyncWait” algorithm is way down the list at 1.0% indicating that whenever the algorithm task finished a set of buffers, a new set of buffers is almost immediately available. Note that the frame rate is only 20 FPS in Example 2. The second algorithm is processor-bound while the first is not.

Recent Changes –November 2021

• Almost 1100 algorithms – almost all less than a page of code. Average algorithm is 31 lines of code

• The reduced point cloud predictably divides an image for analysis.

• “Reuse Rank” in algorithm groups shows all algorithms reused at least twice.

Feature Highlight – OpenCVB Algorithm Rank

New to OpenCVB: start with the Algorithm Ranks

With over a thousand algorithms in OpenCVB, it can be overwhelming for a new user to explore. To help, there are 2 kinds of rankings inside OpenCVB. The “Reuse Rank” shows how often algorithms reuse another algorithms. This is a useful measure of how general or useful the algorithm is. The “Value Rank”, on the other hand, is manually inserted in each algorithm. The snippet code automatically assigns a value of 1 to a new algorithm since this is the lowest ranking. There is no upper limit on the Value Rank.

What is the algorithm most often reused in OpenCVB? The “Thread_Grid” which is used to divide up images for use with multi-threading. To see all the algorithms using this algorithm, select “Thread_Grid” in the rightmost combo box. The leftmost combo box will show all the algorithms that use “Thread_Grid”. The second highest “Reuse Rank” has 2 entries – “Kalman_Basics” and “Reduction_Basics”. Both are often used throughout OpenCVB. Setting the rightmost combo box to “Kalman_Basics” will update the leftmost combo box with the list of all algorithms using “Kalman_Basics”. Similarly, setting the rightmost combo box to “Reduction_Basics” will update the leftmost combo box with the list of all the algorithms using “Reduction_Basics”.

The Value Rank is manually updated so there are some lag between an algorithm’s arrival and an update to its value rank.

Recent Changes –November 2021

• Almost 1100 algorithms – almost all less than a page of code. Average algorithm is 32 lines of code

• This version includes point cloud heat maps (see Highlight below)

• All the Structured.vb algorithms were updated to use heat maps

• TimeView algorithms were removed now that heat maps are available.

• Updated to use the latest RealSense interface.

Feature Highlight – Point Cloud Heat Maps

The bottom left image is a heat map for the side view of the point cloud data while the bottom right image is the same but for the top down view.

The heat map is a well-known method to display populations – blue is cool or low population while red is hot and high population. The plots are actually just histograms of the point cloud data projected into a plane at the side (bottom left) and top (bottom right) of the point cloud. The field of view of the camera is outlined in the markings and the distances are noted as well. The projection can be rotated to align with gravity. The short red lines emanating from the camera can show the field of view (FOV) after rotation. The snapshot was taken using the low-resolution Intel RealSense D435i.

Recent Changes –January 2022

• Oak-D and Oak-D Lite support is now included.

• All OpenCVB’s 1100+ algorithms are now accessible using the Oak-D and Oak-D Lite cameras

• Oak-D installation automated with “Update_OakD.bat”

• Oak-D point cloud is created on the host from the captured depth data

• Oak-D cameras are supported through a separate Python process using a pipe to move images to OpenCVB

  • Breakpoints do not interfere with camera image capture as it would if depthai-core provided the camera images

  • RGB, depth, disparity, and rectified left and right images are provided on every iteration.

  • Oak-D Lite has no IMU but IMU data is provided for the original Oak-D camera with every frame

  • Calibration data for the RGB camera is available as well (used for the point cloud computation.)

• OpenCVB’s “RGB depth” image (upper right) now represents the point cloud data – useful data (not just a pretty picture)

• The Python interface for OpenCVB is now built with requirements for Python 3.9. See installation instructions in “Python Installation” below.

OpenGL View of Oak-D Lite Point Cloud

An OpenGL screen capture shows the output of the point cloud data from an Oak-D Lite camera.

The images above were captured at the same time as the OpenGL image above. The upper left image is the RGB captured from the Oak-D Lite camera and the upper right is the point cloud (computed on the host using the calibration metrics provided by the camera.) The bottom left image is a representation of the depth data used to create the point cloud.

Recent Changes –February 2022

• Switched to Visual Studio 2022! The Community Edition is free and an easy install which can run with other versions of VS installed.

  • Post with any problems transitioning to VS 2022. They will be high priority.

• Further testing and improvements to the Oak-D interface

• Most changes were focused on the RedCloud algorithms to build consistent cells in an image.

• Point cloud filtering is available for each frame as an option

• Added a heartbeat available to all algorithms for once-a-second activity.

• Added a global motion test to update a motion threshold flag available to all algorithms. Redo an image if flag is set.

Recent Changes – March 2022

• Reviewed and reworked the RedCloud algorithms for speed and simplicity.

• Convex Hull algorithm applied to the RedCloud output (see below)

• New ML_Basics example

• Simplified KNN interface and added working examples of 3- and 4-dimensional data.

• Reviewed and reworked all the FloodFill algorithms. Code structure is now like RedCloud algorithms.

• More TrueType text usage throughout.

The image in the bottom left is the output of the RedCloud_Basics algorithm where each cell is consistently identified by applying reduction in X and Y to the point cloud image. The bottom right image shows the RedCloud output after processing with OpenCV’s ConvexHull API. The upper left and right images appear with every algorithm – RGB on the left and the point cloud on the right.

Recent Changes – April 2022

• KNN examples now have higher dimensions – 2, 3, 4 and N dimensions.

• KNN improvement provides a 1:1 matching of points in a series of images.

• Options handling is a key feature of OpenCVB. All algorithms now have a single flag that indicates any options were updated.

• OpenCV’s MatchTemplate can be used for tracking. See Match_TrackFeatures and image below.

• SVM algorithms were reviewed and simplified with better presentation.

• To see previous editions of the “Recent Changes”, see Addendum 3 below

• Install problems get high priority. Please notify of any problems with ‘UpdateAll.bat”

In this first version of the Match_TrackFeatures algorithm, OpenCV”s MatchTemplate (correlation coefficient calculator) is used to track the output of OpenCV’s GoodFeatures. The points are matched with the previous frame using KNN 1:1 matching. In the lower right image, the blue dots were matched to the previous frame while the yellow dots were not matched. In the lower left frame, the correlation coefficient of the region around the feature is shown using the previous and current frame.

Recent Changes – May 2022

• The first tutorial on OpenCVB is now available in the OpenCVB tree. See “Tutorial – Introduction”

• EMax algorithms reviewed – now more general and useful with consistent colors. A new example is provided.

• Related algorithms in different series can now be merged and presented simultaneously – see “Related.vb” examples.

• Global options are now reset to default values before each algorithm is started.

• Updated KNN_One_To_One – simpler to use

• Lines in an image can now be identified and tracked

• LaneFinder example identifies road lane markers (used for the new tutorial.)

• Thread_Grid output now available during all algorithms.

• To see previous editions of the “Recent Changes”, see Addendum 3 below

• Install problems get high priority. Please notify of any problems with ‘UpdateAll.bat”

• OpenCV’s “GoodFeatures” can now be traced as the camera moves – see below.

In this example from “Features_GoodFeatureTrace”, the upper left image shows the good features to track in the current image. The lower right image shows a trace of those same points as the camera is rotated. The motion of the camera is more pronounced in the lower left image.

Recent Changes – June 2022

• Over 1200 working algorithms. Average algorithm length is 31 lines of code.

• A new tutorial was added describing how to find the longest vertical line:

  • https://github.com/bobdavies2000/OpenCVB/tree/master/Tutorials/(3)%20Longest%20Vertical

  • Several new algorithms were add including one using OpenGL to display the 3D results.

• A new tutorial was added describing how to find vertical and horizontal lines in 3D

  • https://github.com/bobdavies2000/OpenCVB/tree/master/Tutorials/(2)%20Lines

  • Accompanying algorithms were also added to Feature.vb – Feature_Lines_Tutorial1, Feature_Lines_Tutorial2, and Feature_Lines

• There is now a global option to use high Brightness/Contrast image input. See Entropy_Highest and toggle the global “Use RGB Filter” checkbox.

• Similarly, there are several other global options to sharpen detail or adjust white balance or filter the RGB data.

  • Any algorithm can be tested with the altered RGB input.

  • Additional RGB filters may be added with a single line of code

  • The default is to not use any RGB Filter

• Double pendulum algorithm added using GitHub example code.

• Highlight color is automatically switched to handle variable backgrounds.

• Robust line-tracking is available in the Feature_Line algorithm

• Robust point-tracking is available in Feature_ MatchTemplate (see next image)

Feature_MatchTemplate: In this example, the highlighted rectangles in the left image are tracked. The correlation coefficient for each rectangle is in the right image. When a correlation coefficient drops below a threshold value (see “Match_Options”), the tracked point is dropped. If more than a percentage of the tracked points are lost, tracked points are recomputed using OpenCV’s GoodFeatures.

Related_MouseClick: In this example, 4 different algorithms are featured. They are “Related” algorithms in the sense that all 4 use the mouse to perform various tricks – the upper left image uses the mouse to highlight an entry in the histogram for back projection, the upper right uses the mouse to slice through the projected side view, the lower left provides the back projection for the mouse selection from the histogram (look at the floor), and finally the lower right uses the mouse for both the x and y coordinates to use in a 2D histogram. The “Related” series of algorithms might be a good place to start looking when the output is remembered but not the name of the algorithm. The Related algorithms are new with the June release of OpenCVB.

Recent Changes – July 2022

• Install tested with the latest Visual Studio 2022 (17.2.6) and OpenCV 4.6 release

  • Keep Visual Studio 2019 around – only way to keep the .Net Framework 4.0 (required for librealsense)

  • OpenCV needed to be cmake’d and built manually – problem in OpenCVUtils.cmake (?)

• Over 1200 working algorithms in a single app with average length 31 lines of code.

• 4 tutorials describing how to build new algorithms. See the ‘Tutorials’ directory.

• “QT” algorithms were introduced to help reduce the clutter in the options presentation. Some options don’t always need to be present.

  • Full option versions of the algorithm are available with the same name – without the “QT” on the end.

  • Want to see all the QT options? See Global Setting “Show Quiesced (QT) Options”.

• Expanded IMU alternatives are available. A tutorial describes how to access and compare the choices.

• New icon in the main toolbar –blue with a white plus sign. See the toolbar in the image below. What does it do?

  • Clicking “plus” will inject the infrastructure for a new algorithm – better than a snippet.

  • Open the dialog and then select the type of new algorithm: VB.Net, C++, OpenGL, C#, or Python

  • Restart OpenCVB and the new algorithm is ready to run!

• OpenGL examples were reworked to make it easier to create new OpenGL algorithms and reuse more code.

• RGB lines are classified as horizontal or vertical using the point cloud – see image below.

OpenGL_3DLines: In the lower left, the horizontal lines are shown in yellow and the vertical lines in blue. In the lower right, the OpenGL output shows the vertical lines in 3D. The OpenGL point cloud is reoriented to gravity so the lines can be verified as vertical or horizontal.

Recent Changes – August 2022

• All the OpenGL algorithms were reviewed and updated with many new features and a simpler interface.

  • More code reuse was main objective

  • New OpenGL algorithms can be added automatically with the new toolbar button.

  • The OpenGL app location is remembered across runs; the OpenGL window will open where it was last used.

  • New OpenGL algorithms depict the 3D scene with color-coded triangles and quads.

  • Beginner OpenGL algorithms for a colored cube and pyramid were added.

• The interfaces for algorithm sliders, checkboxes, and radio buttons are now simpler and easier to use.

  • Scrollbars now provide access to overflow options when there are too many to fit in the options form.

  • Updated snippets reflect the changes to the algorithm options

• RedCloud Hulls improve on cells by creating OpenCV hulls for each cell.

• RedCloud cells now have more accurate min and max depth values.

  • Depth min is never zero

  • Depth max is trimmed using the standard deviation limits

• OpenCVB was tested under Windows 11 – no changes required

• A complete history of “Recent Changes” is included at the bottom of this document

OpenGL_QuadMinMax: The RedCloud image cells (lower left) are presented in OpenGL (lower right) as OpenGL quads – rectangles with min and max depth. The colors in the RedCloud cells are the same as those in the OpenGL presentation. The highlighted cell shown in white in the lower left is also shown as white in the OpenGL presentation in the lower right.

Recent Changes – September 2022

• Over 1200 algorithms are included with an average of 31 lines of code per algorithm

• BackProject_Full builds a histogram and backprojects it into a segmented color image

  • A median filter removes most of the textured output

  • At low resolution, the BackProject_Full has only a minimal impact on performance.

• RedCloud_Hulls output can be input to OpenGL and provide a 3D model of objects in the scene

  • The scene is rendered in OpenGL as 3D solids, not just a point cloud

• All options were updated to simplify the search for sliders, checkboxes, and radio buttons.

  • No more indexed references to any of the option controls

  • The options for sliders can now access any number of trackbars using a scroll bar.

  • Options are no longer counted in the lines of code or algorithm count

• Floor and ceiling are automatically identified using the point cloud and rendered in OpenGL

  • See “OpenGL_FlatFloor” and “OpenGL_FlatCeiling”

• OpenGL Models include both quads and tiles

• Additional improvements to the snippets files for adding Options

  • The new “Add Algorithm” button in the main toolbar was also improved

• A complete history of “Recent Changes” is included at the bottom of this document

RedCloud_ColorAndCloud: The output of the “BackProject_Full” can be input into the RedCloud image segmentation and the results are in the lower left image. Previously, the reduced point cloud was the source of all the RedCloud input – it is shown on the lower right image. With the latest version of OpenCVB, both the color data and depth data can be used to segment the image.

Recent Changes –November 2022

• Over 1300 algorithms are included with an average of 31 lines of code per algorithm

• A new series of algorithms using the “feature polygon” is available for use in camera stabilization and tracking.

• Another new series of algorithms allows C++ algorithms to be constructed and reused the same way as VB.Net algorithms.

  • All the C++ algorithms are accessed from VB.Net through the CPP_Basics algorithm.

  • All the C++ algorithm output is displayed in the VB.Net interface as is normally done for all algorithms.

  • All the CPP_Basics algorithms were modeled on an equivalent VB.Net algorithm.

  • All the CPP_Basics algorithms are available to C++ as an “include only” file – just drop it in.

  • All the C++ algorithms can be stacked into more complex algorithms just like the VB.Net algorithms

• The CPP_Basics algorithms are intended to export any OpenCVB algorithm to other environments.

  • A new sample project shows how the “include only” code can be mainstreamed into an imgui C++ application.

  • There is a new button in the interface to add algorithms conforming to the CPP_Basics style guide.

• There were some improvements to the install process – it is no longer necessary to have MSBuild.exe in the path.

  • OpenCVB’s install process assumes Visual Studio 2022 Community Edition is installed in the default location.

  • For alternate Visual Studio editions, a change is needed to the “Update_<package>.bat” files.

• There was no update to this ReadMe.md in October.

• Install problems? Pull requests for install problems will get the highest priority.

  • NOTE: the current CMake RC 3.25 will fail to install OpenCV. Use the latest release (3.24.3).

• A complete history of changes is included at the bottom of this document

FPoly_Basics: The FPoly (Feature Polygon) series of algorithms use the “good” features of the current image to create a vector describing the orientation of the camera. The white double bar line was captured in an earlier frame while the yellow double bar line is the current orientation of the same vector. A rotate and shift allows a rough comparison between frames separated by time. The values in the figure in the bottom left indicate how many generations (or frames) that the Delaunay region has been present in the image. The older the polygon, the more stability the feature polygon will exhibit. In the lower left image, the black region (highlighted with a yellow edge) shows the oldest of the regions.

Recent Changes – December 2022

• Over 1370 algorithms are included with an average of 31 lines of code per algorithm

• C++ Translator: an OpenCVB-specific tool translates the VB.Net algorithms to C++

  • The patterns used in the VB.Net code translate most of the algorithm to C++

  • The translator is specific to the OpenCV API’s and OpenCVB’s structure

  • The resulting C++ algorithm is similar in structure to existing VB.Net algorithms

    • C++ “IncludeOnly” algorithms can be reused by other C++ algorithms and even VB.Net algorithms.

  • OpenCVB’s “Algorithm Starter” tool generates the C++ template (See “Creating C++ ‘IncludeOnly’ Algorithms” section)

  • There are now some 50 new C++ algorithms available in an “IncludeOnly” file

    • An imgui example shows how to include all 50 algorithms in your C++ application with one include file

• FeatureLess regions are mapped and tracked with the RedCloud image segmentation tool (example below)

• FloodFill examples now use RedCloud_Basics for image segmentation and tracking (previously they used a similar algorithm.)

• ColorMatch algorithms now use RedCloud_Basics as well for image segmentation and tracking.

• A monthly history of changes is included at the bottom of this document

• OpenCVB’s support for the Oak-D Lite and Oak-D Pro was brought up to date with the latest C++ interface (depthai-core)

  • The Python interface is no longer needed as the C++ interface is much more direct.

  • The Oak-D cameras are not installed by default – only Kinect for Azure and RealSense are required.

• Oak-D testing showed that install problems were present and now resolved.

  • Any reported problems with installation get the highest priority.

• An excellent overview of the current 3D camera technology and 3D cameras:

  • https://www.aivero.com/overview-of-depth-cameras/

CPP_RedCloud_FeatureLess: The scene in the upper left is segmented into different featureless regions using only RGB. The image in the bottom right is the output of the edge-drawing C++ algorithm and is the input to a distance transform. A mask created by the distance transform is used to create the identified regions in the lower left image with RedCloud_Basics.