This is the fifth blogpost about the implementation of a tracker for the colored sphere of the PSMove controller. This time it is less about the multiple controller tracking, real world distance estimation, performance issues, and the build-toolchain.
1. Multi Controller Tracking
A lot has happened in the meantime. Most exciting regarding the tracker are its ability of multi controller tracking, increased robustness and the calulation for the distance to the camera.Have a look at a short video featuring tracking of two controllers.
There are some things that are noticeable in that example. First of all and of course the obvious ... two controllers are tracked at the same time at reasonable speed of 350-900 FPS depending of its distance to the camera. The farer away the controller is from the camera the higher the FPS and vice versa. This is caused by the different levels of regions of interest i use to search for the colored blob in the image. But i already told you that in one of the previous posts [3rd OpenCV Example].
2. Real world distance estimation
The Example now displays also a estimate of the distance between the controller and the camera in [mm]. This might get very handy vor applications that like to make use of the 3D position of the controller. You might ask why to use the distance in [mm] rather than the spheres radius. Using the realworld unit has two advantages. One is, that [mm] is easier to understand as it relates directly to the real world in which the interaction takes place. And the second ist, that the distance in [mm] has 1:1 relation with the user on the Z-Axis, while the readius has a logarithmic relation. To put it simply ... it makes app development easier :P.3. Increased worst case performance
One big problem of the old examples was, that if the controller was not visible in the camera image at all, the framerate dropped drastically (around 60FPS on a 2.5Ghz dual core) which was a no go on slower systems e.g. laptops that are in power-saving mode. To increase the FPS, if the controller is not tracked, the tracker simply only scans a quater of the image, and if it does not find the controller, it immediately returns NOT_FOUND. In the next iteration a different quater of the image is evaluated an so on. This is best recogizeable in the video at second 00:17 when i hide the controllers behind my back.
4. Increased robustness to occlusions
Another nice feature, is its increased robustness to occlusions. Have a look of the screenshots taken from the video above.5. The new buildprocess and PSMoveAPI integration
Thanks to Thomas Perl the optical tracking of the controller has now been integrated into the branch "tracker" of the official psmoveapi repository hosted on github. It'll soon be moved the master branch but for now just stick with the "tracker" branch. It also contains a detailed description how to build the whole system and the different examples (also the one from the video) that are part of the psmoveapi. As the build-instructions for windows are quite long, they are included at the end of this post :).6. Remaining problems of the optical tracking
Unfortunately the optical tracking has still some problems.- Only the Magenta, Cyan and Blue can be tracked robustly. Others colors suffer a lot of motion blur which resuls in a low tracking performance. Especially regarding the z-position of the controller.
- Strong daylight coming trough windows might be big problem for tracking performance, too. Avoid direct sunlight by closing curtains if the controllers cannot be tracked.
- There are still too much false detections (sphere estimated to small / wrong position) when the controller is partly occluded.
- Artifical light, especial the one emitted of fluorescent light, causes unwanted jitter in the estimation of the 3D position.
enjoy!
cherio benjamin
System requirements:
- psmove-api + tracker [branch tracker] -> https://github.com/thp/psmoveapi.git
- CL-Driver for the PS-Eye
- CL-SDK for the PS-Eye
- Windows operating system
- Linux operating system (tested on Ubuntu 12.04)
- OpenCV 2.4 headers --> http://opencv.willowgarage.com/wiki/
- OpenCV 2.4 library build --> OpenCV 2.4 win binary
---- Build Instructions For Windows -----------
Get and install
- MinGW : http://sourceforge.net/projects/mingw/files/latest/download?source=files
- CMake : http://www.cmake.org/cmake/resources/software.html
- OpenCV : http://sourceforge.net/projects/opencvlibrary/files/opencv-win/
- GIT : e.g. http://code.google.com/p/msysgit/
- PSEyeDriver : http://codelaboratories.com/get/cl-eye-driver/
[optional]
- CLEyeSDK : http://codelaboratories.com/get/cl-eye-sdk/
1. build and configure OpenCV with cmake
:: you may skip to build OpenCV on your own, however i had no luck
:: the binary distribution did not work on my system
cd <where you extracted opencv>
mkdir build
cd build
cmake .. -G "MinGW Makefiles"
mingw32-make
:: now go for a coffe-break
2. Get you clone of the psmoveapi
git clone https://github.com/thp/psmoveapi.git
3. Check out the "tracker" branch
cd psmoveapi
git checkout tracker
4. Init and update the submodules
git submodule init
git submodule update
4. Copy blue-tooth headers and library to your MinGW installation
:: e.g. MinGW installed at C:\MinGW\
:: e.g. your cloned repository is at D:\dev\psmoveapi
copy D:\dev\psmoveapi\external\mingw-w64-headers\*.h C:\MinGW\include
copy D:\dev\psmoveapi\external\mingw-w64-headers\*.a C:\MinGW\lib
5. make OpenCV known to your system and the cmake toolchain
set OpenCV_DIR=<the path where you extracted opencv>
set PATH=%PATH%;%OpenCV_DIR%\build\bin
7. prepare a new build with cmake for the psmoveapi
::
mkdir build
cd build
:: only with OpenCV Camera access
cmake .. -G "MinGW Makefiles"
:: additionally with Code Laboratories PS Eye SDK
cmake .. -G "MinGW Makefiles" -DPSMOVE_USE_CL_EYE_SDK=ON
8. finally build
mingw32-make
9. start one of the desired test applications
------------------------------------------------
- MinGW : http://sourceforge.net/projects/mingw/files/latest/download?source=files
- CMake : http://www.cmake.org/cmake/resources/software.html
- OpenCV : http://sourceforge.net/projects/opencvlibrary/files/opencv-win/
- GIT : e.g. http://code.google.com/p/msysgit/
- PSEyeDriver : http://codelaboratories.com/get/cl-eye-driver/
[optional]
- CLEyeSDK : http://codelaboratories.com/get/cl-eye-sdk/
1. build and configure OpenCV with cmake
:: you may skip to build OpenCV on your own, however i had no luck
:: the binary distribution did not work on my system
cd <where you extracted opencv>
mkdir build
cd build
cmake .. -G "MinGW Makefiles"
mingw32-make
:: now go for a coffe-break
2. Get you clone of the psmoveapi
git clone https://github.com/thp/psmoveapi.git
3. Check out the "tracker" branch
cd psmoveapi
git checkout tracker
4. Init and update the submodules
git submodule init
git submodule update
4. Copy blue-tooth headers and library to your MinGW installation
:: e.g. MinGW installed at C:\MinGW\
:: e.g. your cloned repository is at D:\dev\psmoveapi
copy D:\dev\psmoveapi\external\mingw-w64-headers\*.h C:\MinGW\include
copy D:\dev\psmoveapi\external\mingw-w64-headers\*.a C:\MinGW\lib
5. make OpenCV known to your system and the cmake toolchain
set OpenCV_DIR=<the path where you extracted opencv>
set PATH=%PATH%;%OpenCV_DIR%\build\bin
7. prepare a new build with cmake for the psmoveapi
::
mkdir build
cd build
:: only with OpenCV Camera access
cmake .. -G "MinGW Makefiles"
:: additionally with Code Laboratories PS Eye SDK
cmake .. -G "MinGW Makefiles" -DPSMOVE_USE_CL_EYE_SDK=ON
8. finally build
mingw32-make
9. start one of the desired test applications