Welcome to the home of Modular Tracking Framework (MTF) !

• Introduction
  • Documentation
  • Implementation
  • Platforms
  • License and citing
• Prerequisites
  • General
    • Required
    • Optional
  • Windows
    • CMake with Visual Studio (Recommended)
      • Required
      • Optional
    • GNU Make with MinGW
• Installation
  • Recommended
  • Advanced
    • Commands
      • Examples
      • Python
      • MATLAB
    • Compile time switches
    • Clean up
• Installation Issues
  • Cmake
    • Windows
  • Compile time
  • Runtime
  • Macintosh
  • OpenCV 3.x
  • make build system
• Setting Parameters
• Running the demo application
• Building ROS package
• Building a new application
  • cmake
    • Windows
  • make
• For Developers
  • Adding a new AM or SSM
    • cmake
    • make
  • Examples
    • Implementing minimalistic AM that can be used with Nearest Neighbour SM
    • Implementing a minimalistic AM that can be used with Particle Filter SM

Introduction

MTF is a highly efficient and extensible library for registration based tracking, also called direct visual tracking. It utilizes a modular decomposition of trackers in this domain wherein each tracker comprises the following 3 modules:

1. Search Method (SM): ESM, IC, IA, FC, FA, LM, NN, PF, RANSAC or LMS
2. Appearance Model (AM): SSD, SAD, ZNCC, SCV, LSCV, NCC, MI, CCRE, KLD, SSIM, SPSS, RIU, NGF, PCA or DFM
   • Illumination Model (ILM): GB, PGB and RBF
3. State Space Model (SSM): Spline (50+ DOF), TPS (50+ DOF), Homography (8 DOF), Affine (6 DOF), ASRT (Anisotropic Scaling, Rotation and Translation - 5 DOF ), Similitude (4 DOF), AST (Anisotropic Scaling and Translation - 4 DOF), Isometry (3 DOF), IST (Isotropic Scaling and Translation - 3 DOF ) or Translation (2 DOF)

Documentation

Please refer these papers: [cviu] [iros17] for details on the system design, these: [crv16][wacv17][crv17] for some performance results and this thesis for a comprehensive description. There is also a dedicated website where Doxygen documentation will soon be available along with detailed tutorials and examples. It also provides several datasets formatted to work with MTF.

Implementation

The library is implemented entirely in C++ though interfaces for Python and MATLAB are also provided to aid its use in research applications. A simple interface for ROS is likewise provided for seamless integration with robotics projects. In addition to the registration tracking modules, MTF comes bundled with several state of the art learning and detection based trackers whose C++ implementations are publicly available - DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT. It can thus be used as a general purpose tracking test bed too.

Platforms

MTF supports both Unix and Windows platforms. Though the Unix build system has been tested comprehensively only under Linux, specifically Ubuntu 14.04/16.04, it should work on Macintosh systems too (see Installation Issues section below for resolving possible issues). The Windows build system has been tested on Windows 8.1 and 10 with Visual Studio 2015, 2017 and 2019 though it should work fine with any non-ancient versions of the OS and IDE.

License and citing

MTF is provided under BSD license and so is free for research and commercial applications. We do request, however, that this paper [bibtex] [arxiv version] be cited by any publications resulting from projects that use MTF so more people can get to know about and benefit from it. Finally, if any issues are encountered while installing or running the library, please create an entry in the issues section and we will do our best to resolve it as soon as possible.

Prerequisites

General

Required

• compiler supporting C++11 (GCC 4.7 or newer)
• Eigen
  • needs to be added to the C/C++ include paths which can be done, for instance (linux), by running the following commands, assuming that Eigen is installed in /usr/include/eigen3:
    echo "export C_INCLUDE_PATH=${C_INCLUDE_PATH}:/usr/include/eigen3" >> ~/.bashrc
echo "export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/include/eigen3" >> ~/.bashrc
  • version 3.2.10 or newer is needed
• OpenCV
  • comprehensive testing has only been done with versions 2.4.13, 3.3.0 and 3.4.1;
  • 2.4.x is recommended for widest compatibility as rare issues with OpenCV 3.x have been known to prevent successful compilation with some configurations (see Installation Issues section below)
  • 3.3.x or 3.4.x is recommended for widest functionality
  • 4.x.x is not supported
  • the nonfree / contrib module should be installed too if the corresponding feature detectors and descriptors (SIFT and SURF) are to be available in the feature tracker
• Boost
  • testing has been done with 1.5.x - 1.7.x though older versions should work too

Optional

• FLANN
  • needed for the faster variant of NN search method
  • Compilation from source of version 1.8.4 available on the website is recommended as the version in the Ubuntu (and git) repo might give linker errors (Issue #3).
  • NN can be disabled at compile time using nn=0 switch if FLANN is not available (see compile time switches).
• Intel TBB / OpenMP
  • needed for parallelization
  • preliminary testing has not shown significant performance improvements
• ViSP
  • needed if its template tracker module or input pipeline are enabled during compilation (see compile time switches).
  • Note that version 3.0.0+ is required. The Ubuntu apt package is 2.8 and is therefore incompatible.
  • also needed for interactive operations in the multi threaded versions of the Matlab and Python interfaces as OpenCV window system has an unresolved bug/incompatibility with boost threading library which causes it to hangup or crash the program.
• Caffe
  • needed for some optional modules including FMaps, Regnet and GOTURN if these are enabled during compilation
• Xvision
  • obsolete and only needed if it is enabled during compilation (see compile time switches).
  • Not recommended as it is very difficult to install and configure on modern systems
• Additional installation instructions for Ubuntu 18.04 with Matlab 2018b are available here. These may contain solutions for some issues on newer platforms that are not documented here.

Windows

CMake with Visual Studio (Recommended)

Required

• Install a recent version of Visual Studio
  • the installation has been tested comprehensively with Visual Studio 2015 but any recent version (2013 or newer) should work fine;
  • it seems that the OpenCVConfig.cmake file that comes with the latest version of OpenCV simply does not recognize Visual C++ 2017 so cmake might not be able to find it and instead exit with the erroneous message: Found OpenCV Windows Pack but it has not binaries compatible with your configuration; a similar issue is exhibited by ViSP so it is best to use 2015 or older versions of Visual Studio; a possible way to resolve this issue is in the Installation Issues section if a newer version must be used.
  • the freely available Express/Community edition can be used too
• Set EIGEN3_ROOT, EIGEN3_DIR and EIGEN3_ROOT_DIR environment variables to the folder containing the Eigen header files
  • this folder should contain the sub folders Eigen and unsupported and a file called signature_of_eigen3_matrix_library
  • if cmake still fails to find Eigen, set EIGEN_INCLUDE_DIR_WIN variable in CMakeLists.txt (line 13) to the location of this folder
• Build OpenCV from source using Visual Studio or download the pre built binaries built with the correct version of visual studio if available.
  • as mentioned before, version 2.4.x is recommended; if 3.x is used instead and compile time errors occur, try disabling third party trackers by passing the switch -DWITH_THIRD_PARTY=0 to cmake
  • set environment variable OpenCV_DIR to the folder where the binaries have been built or downloaded; this should contain the OpenCVConfig.cmake file along with the lib and bin folders
  • add the location of the bin folder to the Path environment variable
  • if cmake still fails to find OpenCV, set OPENCV_INCLUDE_DIR_WIN, OPENCV_LIB_DIR_WIN and OpenCV_SUFFIX variables in CMakeLists.txt (line 14-16) to the respective values
• Build Boost from source using Visual Studio (toolset=msvc) or download prebuilt binaries built with the correct version of visual studio if available.
  • using the steps under Simplified Build From Source should work too
  • set BOOST_INCLUDEDIR and BOOST_LIBRARYDIR environment variables to the locations of the header and library (*.lib) files respectively.
  • add the location of the bin folder containing the *.dll files to the Path environment variable
  • if cmake still fails to find Boost, set BOOST_INCLUDE_DIR_WIN, BOOST_LIB_DIR_WIN and Boost_SUFFIX variables in CMakeLists.txt (line 17-19) to the respective values
  • prebuilt binaries from a similar though not exacly matching version of visual studio should be usable too after modifying the contents of all .cmake files and names of all .lib and .dll files to change the version number as needed.

Optional

• If FLANN is needed, build it from source using Visual Studio and set FLANN_ROOT environment variable to the folder containing the compiled library and header files respectively
  • this folder should have sub folders called lib, bin and include that contain the *.lib, *.dll and the header files
  • add the bin sub folder to the Path environment variable
• If ViSP is needed, build ViSP from source using Visual Studio and set VISP_DIR environment variable to the folder containing the library and header files
  • this folder should have a file called VISPConfig.cmake along with sub folders called x86/vc<version>/lib, x86/vc<version>/bin and include that contain the *.lib, *.dll and the header files respectively.
  • add the path of the bin folder to the Path environment variable
• if the Python interface is to be built, install Python 2.7.x and Numpy.
  • CMake should normally be able to find the needed Python/Numpy libraries and headers without any other setup
  • if any problems are encountered, please open an issue to let us know about it
• MATLAB interface
  • the MATLAB finding module of cmake has a limitation that it can only find a MATLAB installation of the same build type as the target build, i.e. a 32 bit target build (default) can only find a 32 bit installation of MATLAB and same for 64 bit
  • the recommended approach is to install 32 bit version of MATLAB but if that is not possible, a 64 bit CMake target build can be specified using, for instance the instructions given here
    • 64 bit builds have not been tested so cannot be guaranteed to work;
    • all dependencies - OpenCV, Boost, FLANN and ViSP - must also be built using 64 bit builds for the linking to work
  • if multiple MATLAB versions are installed or if the installed version is not found, set variable Matlab_ROOT_DIR or Matlab_ROOT_DIR_DEFAULT in Examples.cmake to the MATLAB installation folder to help CMake find the correct version
  • the module is installed to <Matlab_ROOT_DIR>/toolbox/local by default; this location may need to be added to the MATLAB path for it to be found;

GNU Make with MinGW

• Note: This method is not recommended due to the following issues:
  • it is known to run into "out of memory" exceptions while compiling some of the executables
  • importing pyMTF from a Python script results in an error: ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed. due to an obscure incompatibility issue between the python import system and third party modules compiled with MinGW.
• Install MinGW, Make for Windows and CoreUtils for Windows
• Build OpenCV from source using MinGW and set OPENCV_INCLUDE_DIRS and OPENCV_LIB_DIRS in the makefile to the locations of header and library files respectively.
• Build Boost from source using MinGW (toolset=gcc) and set BOOST_LIB_DIRS and BOOST_INCLUDE_DIRS in the makefile to the locations of the header and library (*.dll.a) files respectively.
• Set EIGEN_INCLUDE_DIRS in the makefile to the location containing Eigen header files.
• Set appropriate values for BOOST_LIBS_SUFFIX, and OPENCV_LIBS_SUFFIX based on the installed versions of the respective libraries.
• if the Python interface is to be built, install Python 2.7.x and Numpy and set appropriate locations for PYTHON_INCLUDE_DIR, PYTHON_LIBS_DIR, NUMPY_INCLUDE_DIR and MTF_PY_INSTALL_DIR in Examples/Examples.mak.
• Add the bin folders of the MinGW and GnuWin32 installations (e.g. C:\MinGW\bin, C:\Program Files (x86)\GnuWin32\bin) along with folders containing the *.dll files of OpenCV and Boost installations (e.g. C:\OpenCV\build\x86\mingw\bin) to the PATH environment variable and reboot the system for the changes to take effect.

Installation

Download the source code as zip file or clone using:
git clone https://github.com/abhineet123/MTF.git

Recommended

MTF comes with both a make and a cmake build system but the latter is recommended for most users of the library and can be used as follows:

• use the standard method (i.e. mkdir build && cd build && cmake ..) to create the makefile for Unix or Visual Studio solution for Windows
• Unix: use make to compile (make -j<n> for n-threaded compilation) and sudo make install to install
• Windows: open MTF.sln in Visual Studio and run Build->Build Solution
  • if needed, change the solution configuration to Release from Debug to build the faster version
  • right click on Solution Explorer->INSTALL and select Build to install the library, header and executable files to the installation directory (C:/MTF by default); add <installation_folder>/bin to Path environment variable to be able to run the executables from any location;

The make system might be used by developers/contributors as it can potentially offer a finer level of control. Note, however, that it is not tested as well as the cmake system so is not guaranteed to work in all configurations or be up to date with the latest changes.

Advanced

Following are commands and switches for both make and cmake systems to customize the installation by disabling parts of the library or to only compile specific components. It also contains general information about the Python and Matlab wrappers and example applications. Not all of the make commands may work with the cmake system.

Commands

• make or make mtf : compiles the shared library (libmtf.so) to the build directory (Build/Release)
• make install : compiles the shared library if needed and copies it to /usr/local/lib; also copies the headers to /usr/local/include; (use make install_lib or make install_header for only one of the two); if third party trackers are enabled, their respective library files will be installed too;
  • this needs administrative (sudo) privilege; if not available, the variable MTF_INSTALL_DIR in the makefile can be modified to install elsewhere. This can be done either by editing the file itself or providing it with the make/cmake command as:
    make install MTF_INSTALL_DIR=<installation_dir>
cmake .. -D MTF_INSTALL_DIR=<installation_dir>
    • the lib and include sub folders within this folder should be present in LD_LIBRARY_PATH and C_INCLUDE_PATH/CPLUS_INCLUDE_PATH environment variables respectively so any application using MTF can find the library and headers.

Examples

• make exe/make install_exe : compile/install the main example file Examples/cpp/runMTF.cc to create an executable called runMTF that uses this library to track objects.
  • library should be installed before running this, otherwise linking will not succeed
  • installation folder is /usr/local/bin by default; this needs administrative privilege too - change MTF_INSTALL_DIR as above if this is not available
• make mtfi : all of the above
• make uav/make install_uav : compile/install an application called trackUAVTrajectory that tracks the trajectory of a UAV in a satellite image of the area over which it flew while capturing images from above
• make mos/make install_mos : compile/install an application called createMosaic that constructs a live mosaic from a video of the region to be stitched
• make qr/make install_qr : compile/install an application called trackMarkers that automatically detects one or more markers in the input stream and starts tracking them
  • this uses the Feature tracker so will only compile if that is enabled (see below)
  • this reads marker images from Data/Markers folder by default; this can be changed by adjusting qr_root_dir in Config/examples.cfg where the names of marker files and the number of markers can also be specified along with some other parameters;
• make all/make install_all : compile/install all example applications that come with MTF along with the Python and Matlab interfaces
• make app app=<APPLICATION_NAME>: build a custom application that uses MTF with its source code located in <APPLICATION_NAME>.cc; the compiled executable goes in the build directory;
  • location of the source file (<APPLICATION_NAME>.cc) can be specified through MTF_APP_SRC_DIR (defaults to Examples/cpp)
  • make mtfa will install it too - installation location can be specified through MTF_APP_INSTALL_DIR (defaults to the current folder)

Python

make py/make install_py (make py2/make install_py2) : compile/install the Python interface to MTF (or its multi threaded version) - this creates a Python module called pyMTF (pyMTF2) that serves as a front end for using MTF from Python.

• usage of this module is demonstrated in Examples/python/runMTF.py (runMTF2.py) and function specification is in the ReadMe in the Examples folder
• supports both Python 2.7 and 3.x but only a particular version is built at a time - this can be specified using the PY_VER/py_ver cmake/make variable (set to 2.7 by default) (e.g. -D PY_VER=3.5 for Python 3.5)
• if multiple Python versions are installed on the system, cmake might not be able to find the desired one even if PY_VER is provided; in this case Python header and library locations must be provided though the variables PYTHON_INCLUDE_DIR and PYTHON_LIBRARY (e.g. -D PYTHON_INCLUDE_DIR=/usr/include/python3.5 -D PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so for Python 3.5)
• neither cmake nor make systems currently support inferring the correct install location for pyMTF from Python library/header paths so this must be specified through MTF_PY_INSTALL_DIR (in Examples.mak/cmake) if using a custom Python version (defaults to /usr/local/lib/python2.7/dist-packages/)
• refer to cmake/cmake.md for an example cmake command to compile pyMTF with Python 3.5;
• pyMTF2 needs ViSP to be installed and enabled during compilation for its interactive operations
• there is also an additional wrapper for pyMTF2 input pipeline called mtf.VideoCapture that provides an interface similar to cv2.VideoCapture but can be used, through ViSP pipeline, to access USB and firewire cameras with custom resolution and other settings that are unavailable in cv2. It can be installed by running python setup.py install or python3 setup.py install from Examples/python folder. Its usage is demonstrated in the ReadMe in Examples folder.

MATLAB

make mex (make mex2) : compile the MATLAB interface to MTF (or its multi threaded version) - this creates a MATLAB module called mexMTF (mexMTF2) that serves as a front end for using MTF from MATLAB.

• set MATLAB_DIR variable in Examples.mak to the root of the MATLAB installation folder
• usage of this module is demonstrated in Examples/matlab/runMTF.m (runMTF2.m) and function specification is in the ReadMe in the Examples folder
• installation location can be specified through MTF_MEX_INSTALL_DIR in Examples.mak/cmake (defaults to <MATLAB_DIR>/toolbox/local)
• if CMake is being used under Unix, the restriction of matching build target and MATLAB installation type mentioned in the Windows installation section applies here too; however, it is possible to compile mexMTF in Unix even if MATLAB is not detected by CMake by running the command written to a file called mtf_mex_cmd.txt in the build folder;
  • the command can be run at the MATLAB prompt or even at the terminal if the location of the mex executable is present in the PATH environment variable;
  • a few changes need to be made to it first as detailed in the CMake message;
• mexMTF2 needs ViSP to be installed and enabled during compilation for its interactive operations

Compile time switches

These apply to all of the above commands and the equivalent cmake options, where such exist, are given within parenthesis:

• only_nt=1/nt=1(WITH_TEMPLATED=OFF) will enable only the Non Templated (NT) implementations of SMs and disable their templated versions that are extremely time consuming to compile though being faster at runtime (disabled by default)
  • can be very useful for rapid debugging of AMs and SSMs where headers need to be modified;
  • the NT implementation of NN only works with GNN since FLANN library needs its object to be templated on the AM;
• nn=0(WITH_FLANN=OFF) will disable the templated implementation of NN search method (enabled by default).
  • should be specified if FLANN is not available
  • FLANN has some compatibility issues under Windows so this is disabled by default;
• spi=1(WITH_SPI=ON) will enable support for selective pixel integration in modules that support it (disabled by default)
  • currently only SSD and NCC AMs support this along with all the SSMs
  • this might decrease the performance slightly when not using SPI because some optimizations of Eigen cannot be used with SPI
• grid=0(WITH_GRID_TRACKERS=OFF) will disable the Grid trackers and RKLT (enabled by default).
• feat=0(WITH_FEAT=OFF) will disable the Feature tracker (enabled by default).
  • this uses optional functionality in the nonfree / contrib module of OpenCV so this should be installed too if these are to be available.
• lt=0(WITH_THIRD_PARTY=OFF) will disable the third party open source learning based trackers - DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT - that are also bundled with this library (in ThirdParty subfolder) (enabled by default except MIL, DFT and GOTURN).
  • several third party trackers that have a CMake build system do not compile under Windows yet and are thus not available;
• gtrn=1(WITH_GOTURN=ON) will enable GOTURN deep learning based tracker (disabled by default)
  • requires Caffe to be installed and configured
    • if not installed in ~/caffe/build/install, specify CAFFE_INSTALL_DIR either at compile time or by editing ThirdParty/GOTURN/GOTURN.mak
  • optional data for use with it is available here - this should be extracted inside the MTF root folder before running GOTURN
• dft=1(WITH_DFT=ON) will enable the Descriptor Fields Tracker (disabled by default)
  • this is known to have issues with OpenCV 3.x so should not be enabled if that is being used
• mil=1(WITH_MIL=ON) will enable the Multiple Instance Learning tracker (disabled by default)
  • this too is known to have issues with OpenCV 3.x so should not be enabled if that is being used
• vp=1(WITH_VISP=ON) will enable ViSP template trackers and input pipeline (disabled by default).
  • if a runtime linking error is encountered, add ViSP library install path to LD_LIBRARY_PATH variable by running, for instance, echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/x86_64-linux-gnu/" >> ~/.bashrc
• xv=1(WITH_XVISION=ON) will enable Xvision trackers and input pipeline (disabled by default and not recommended).
• o=0 will compile the library in debug mode (libmtf_debug.so) and disable optimization - the corresponding executables will be suffixed with 'd' ( for example runMTFd)
• ver=<version_postfix> will postfix all compiled library and executable files with the provided postfix so that multiple versions may coexist on the same system without conflict; e.g. ver=2 will create libmtf_v2.so and runMTF_v2 and same for all libraries created during MTF compilation (including those for third party trackers);
• header_only=1 can be used with make exe or make mtfe to build a stand alone version of runMTF called runMTFh that does not need to link with libmtf.so
  • this disables third party trackers automatically as several of those have their own libraries built from within MTF
  • it will still need to link against external libraries that it uses (like OpenCV, FLANN and boost)
  • it may take some time (20-30 minutes) to compile as the entire library has to be consolidated into a single executable
  • this version should be faster in theory as the compiler has access to all code at once and can thus optimize better but tests so far have not found any significant difference in speed from the standard version

Clean up

• make clean : removes all the .o files and the .so file created during compilation from the Build folder.
• make mtfc : also removes the executable.

Installation Issues

Cmake

• ViSP cmake system has a bug in Ubuntu 16.04 so if a cmake error of type Found ViSP for Windows but it has no binaries compatible with your configuration. You should manually point CMake variable VISP_DIR to your build of ViSP library. occurs, add -DVISP_DIR=<path to ViSP build folder> to the cmake command and rerun after removing the existing cmake files

Windows

Using Visual Studio 2017 or newer to build OpenCV 3.x.x might lead to a cmake error like Found OpenCV Windows Pack but it has not binaries compatible with your configuration. This can be resolved using the following steps which are for OpenCV 3.4.1 64 bit build with Visual Studio 2019 but adapting for other versions should be straightforward.

1. Modify OpenCVConfig.cmake to add

    elseif(MSVC_VERSION MATCHES "^192[0-9]$")
        set(OpenCV_RUNTIME vc16)
   

right after

    elseif(MSVC_VERSION EQUAL 1900)
      set(OpenCV_RUNTIME vc14)
    elseif(MSVC_VERSION MATCHES "^191[0-9]$")
      set(OpenCV_RUNTIME vc15)

2. Move bin, lib and include folders to x64/vc16/ after creating the same.

Compile time

• if an error like cannot find -l<name of library> (for instance cannot find -lhdf5 or cannot find -lmtf) occurs at compile time, location of the corresponding library should be added to LIBRARY_PATH and LD_LIBRARY_PATH environment variables in the .bashrc file.

  • For instance HDF5 installs in /usr/local/hdf5/lib by default so following commands can be run to add its path there:
    echo "export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/hdf5/lib" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hdf5/lib" >> ~/.bashrc
    Similarly, MTF installs to /usr/local/lib by default so following can be run for it:
    echo "export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
  • The bashrc file should be reloaded by either restarting the terminal or running . ~/.bashrc and compilation should be tried again.

• similar compile time errors pertaining to missing header files may be encountered in which case the location of the include folder of the corresponding library should be added to C_INCLUDE_PATH and CPLUS_INCLUDE_PATH variables. For instance, following commands can be run for HDF5:
  echo "export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/hdf5/include" >> ~/.bashrc
echo "export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/local/hdf5/include" >> ~/.bashrc

• if a compile time error similar to this occurs: /usr/include/eigen3/Eigen/src/Core/util/BlasUtil.h:233:98: error: no matching function for call to ‘Eigen::internal::blas_traits, please update Eigen to the latest version;

• if a compile time error similar to this occurs:
  /usr/bin/ld: cannot find -lQt5::Core
/usr/bin/ld: cannot find -lQt5::Gui
/usr/bin/ld: cannot find -lQt5::Widgets
/usr/bin/ld: cannot find -lQt5::Test
/usr/bin/ld: cannot find -lQt5::Concurrent
/usr/bin/ld: cannot find -lQt5::OpenGL
  there is probably something wrong with the OpenCV installation. Reinstalling that should fix it (refer to issue #13).

• A compile time error similar to this:
  error: no matching function for call to vpDisplay::init(vpImage<vpRGBa>&, int, int, std::string&)
  probably indicates that an outdated version of ViSP is installed. This can be resolved by either updating it to 3.0.1 or newer or using -DWITH_VISP=0/vp=0 cmake/make option to disable ViSP.

• A compile time error similar to this:
  undefined reference to LZ4_compress_HC_continue
  occurs when the cmake module for one of the dependencies is not configured properly and does not return one or more libraries it needs to be linked against (liblz4 in this case).

  • This can be resolved by adding -D MTF_LIBS=<name_of_missing_library> (in this case: -D MTF_LIBS=lz4) to the cmake command.
  • The corresponding library might also need to be installed, for instance, by using sudo apt-get install liblz4-dev

Runtime

• if a runtime error like cannot find libmtf or something similar is encountered while running runMTF, /usr/local/lib should be added to LIBRARY_PATH and LD_LIBRARY_PATH environment variables as above (assuming the default MTF installation directory has not been changed - otherwise the new path should be used instead).

• if a run time error similar to this:
  undefined symbol: _ZN3cmt7Matcher10initializeERKSt6vectorIN2cv6Point_IfEESaIS4_EENS2_3MatERKS1_IiSaIiEES9_S4_
  suddenly occurs or running runMTF causes segmentation fault with no messages right after installing ROS, it is probably because MTF is linking with an incorrect version of OpenCV installed by ROS.

  • The only known solution guaranteed to work is to uninstall either the original or the ROS version of OpenCV and then recompile MTF.
  • Possible ways to have both versions installed and have MTF link with the correct one are suggested here. These have not been tested with MTF so might not work.

Macintosh

• if make build system is used, some third party modules might not compile successfully if cmake uses clang instead of gcc as the default compiler; in such cases compiling with lt=0 to disable all third party modules is the best option.
• if cmake build system is used, then following commands must be run so that cmake uses gcc during compilation:
  export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
• if FLANN based NN is enabled with cmake, environment variables may need to be set for hdf5 to be found using, for instance:
  export CMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}:/Applications/HDF_Group/HDF5/1.10.0-patch1/

OpenCV 3.x

• MTF has been tested and found to work on Ubuntu 14.04 with OpenCV 3.1 installed using this method (option 2) and OpenCV 3.3 installed using this method but general compatibility with all system configurations is not guaranteed.
• third party trackers in particular are likely to have issues with OpenCV 3.x as many use legacy versions so these should be disabled using lt=0/-DWITH_THIRD_PARTY=OFF if compilation or linking errors pertaining to these are found.
• if a linking error of type /usr/bin/ld: cannot find -lippicv occurs, remove -ippicv from opencv pkg-config configuration file which is usually located at /usr/local/lib/pkgconfig/opencv.pc or follow the procedures suggested here

make build system

• If MTF had earlier been compiled with only_nt=0 (or nn=1) and is recompiled with only_nt=1 (or nn=0) such that the executable does not get recompiled, a runtime error of the type: symbol lookup error: runMTF: undefined symbol: may occur on running the executable; to resolve this, remove runMTF (or whatever executable is causing the error) from the build folder (Build/Release or Build/Debug) by running, for instance rm Build/Release/runMTF and recompile; this happens because using only_nt=1 or nn=0 turns off part of the code in the library but the executable will continue to expect these to be present till it is recompiled;
• if compilation is taking too long and optional components are not needed, following cmake/make commands can be used to build a minimal version of MTF with full core functionality:
  cmake .. -DWITH_THIRD_PARTY=OFF -DWITH_TEMPLATED=OFF
make mtfall lt=0 nt=1
  • if FLANN based NN is not needed (graph based NN will still be available), -DWITH_FLANN=OFF/nn=0 can be specified too
  • if feature tracker is not needed, -DWITH_FEAT=OFF/feat=0 can be specified too

Setting Parameters

MTF parameters can be specified either in the cfg files present in the Config sub folder or from the command line. Please refer the ReadMe in the Config sub folder or this configuration page for detailed instructions.

Some preconfigured cfg files are provided here to get the system up and running quickly. Config files involving dataset sequences assume that all datasets are located in the Datasets sub folder under the active directory. The zip file should be extracted in the Config folder to replace the existing cfg files there (if any):

• RKLT with a 10x10 grid of trackers using Efficient Second order Minimization SM, Normalized Cross Correlation AM and 2 DOF (translation) SSM followed by an 8 DOF (homography) tracker with the same SM and AM - this is the current state of the art in 8 DOF tracking to the best of our knowledge
  • live sequence from USB or Firewire camera
  • nl_cereal_s3 sequence from TMT dataset
• Nearest Neighbour with 1000 samples followed by Inverse Compositional Levenberg Marquardt SM with Mutual Information AM and 8 DOF (homography) SSM
  • live sequence from USB or Firewire camera
• Particle Filter with 500 particles followed by Forward Compositional Levenberg Marquardt SM with Structural Similarity AM and 8 DOF (homography) SSM
  • towel sequence from LinTrack dataset

Running the demo application

Use either make run or runMTF to start the tracking application using the settings specified as above.

Building ROS package

A simple ROS package called mtf_bridge that demonstrates the integration of MTF with ROS applications is included in the ROS sub directory. Details about building and using this package are in the ReadMe in this folder.

Building a new application

cmake

The build process generates a mtfConfig.cmake file in the build folder with the main defines. There are two ways to use this file:

• The standard find_package command can be used but this requires MTF_DIR variable to be set to the MTF build folder while running the cmake command. This can be done, for instance, as:
  cmake .. -DMTF_DIR=<path to MTF build folder>

• The mtfConfig.cmake file can be copied into the project tree and included in the CMakeLists.txt file. Then, the defined variables can be used to obtain the header files, linked libraries and compile time definitions. More details can be found in the comments section of issue #11.

An example CMakeLists.txt file for a standalone project that uses either of the above methods is included in cmake/CMakeLists.txt.example.

If neither of the above methods work, a manually created FindMTF.cmake file is also included in the cmake/Modules folder. It has not been widely tested but should work with most standard configurations.

Windows

If MTF needs to be used within an existing VS project that was built without using cmake, the required preprocessor definitions and linked libraries can be added manually.

• Right click on one of the executables in MTF solution (e.g. runMTF), go to Properties -> C/C++ -> Preprocessor -> Preprocessor Definitions and copy all flags there to the same location in your own project.

• Repeat this with Properties -> Linker -> Input -> Additional Dependencies to add all required libraries for linking.

make

Use the make app app=<APPLICATION_NAME> command as detailed in the make commands section.

For Developers

Adding a new AM or SSM

cmake

1. Modify the .cmake files in the respective sub directories:

   • Modify the AM/AM.cmake file to add the name of the AM to set(APPEARANCE_MODELS …)
   • Modify the SSM/SSM.cmake file to add the name of the SSM to set(STATE_SPACE_MODELS …)

2. Modify "mtf.h" to add a new AM/SSM option in the overloaded function getTrackerObj (starting lines 498 and 533 respectively) that can be used with your config file to pick the AM/SSM, and create an object with this selected model.

3. Modify "Macros/register.h" to add the new AM/SSM class name under _REGISTER_TRACKERS_AM/_REGISTER_HTRACKERS_AM or _REGISTER_TRACKERS/_REGISTER_TRACKERS_SSM/_REGISTER_HTRACKERS/_REGISTER_HTRACKERS_SSM respectively

make

1. Modify the .mak file in the respective sub directory to:
   • add the name of the AM/SSM to the variable APPEARANCE_MODELS or STATE_SPACE_MODELS respectively
   • add rule to compile the .o of the new AM/SSM that is dependent on its source and header files - simplest method would be to copy an existing command and change the names of the model.
2. Modify makefile to add any extra dependencies that the new files need to include under variable FLAGS64 and extra libraries to link against under LIB_MTF_LIBS

All steps are identical for adding a new Search Method (SM) too except the last one which is not needed. Instead this header needs to be included and the appropriate macro needs to be called from its source file to register the SM with all existing AMs and SSMs. Refer to the last 2 lines of the .cc file of any existing SM to see how to do this.

Examples

Implementing minimalistic AM that can be used with Nearest Neighbour SM

You need to create a new derived class from AppearanceModel. Implement the following functions:

1. initializePixVals/updatePixVals : takes the current location of the object as a 2xN matrix where N = no. of sampled points and each column contains the x,y coordinates of one point. By default, it is simply supposed to extract the pixel values at these locations from the current image but there is no restriction by design and you are free to compute anything from these pixel values.
2. initializeDistFeat/updateDistFeat : these compute a distance feature transform from the current patch.
3. getDistFeat: returns a pointer to an array containing the distance feature vector computed by the above function. There is also an overloaded variant of updateDistFeat that takes a pointer as input and directly writes this feature vector to the pre-allocated array pointed to by this pointer.
4. distance functor (operator()): computes a scalar that measures the dissimilarity or distance between two feature vectors (obtained using the previous two functions).

Implementing a minimalistic AM that can be used with Particle Filter SM

You need to create a new derived class from AppearanceModel. Implement the following functions:

1. initializePixVals/updatePixVals : Same as with NN. No need to overwrite if you just need to extract the pixel values
2. getLikelihood: get the likelihood/weights of the particle
3. initializeSimilarity: initialize some class variables like the initial features.
4. updateSimilarity: update the similarity between current patch and the template