Fast Object Tracking – Robot Computer Vision

Tags: , ,

I wanted my robot to be able to track object and follow them. The first thing I wanted to do is give the robot the ability to follow an object with its head camera. The head camera is mounted on a pan-tilt servo system, and hence is capable of moving left and right, up and down (as seen in the picture below).

My second object tracking goal was to make the robot chase after an object, much like a dog would chase a ball thrown by his owner. This kind of tracking is quite harder – it would use the head camera tracking from the previous step, and combine it with rest of the robot sensors to follow the object.

How?

To achieve that I’m going to use several basic image processing \ computer vision algorithms. I’m going to use the OpenCV library. OpenCV, as it’s name suggests, is an open-source computer-vision library originally

developed by Intel. It is cross-platform (I have used it both on a PC and on the ARM based Beagleboard). OpenCV is fairly easy to use if you have basic knowledge in image-processing.

The first object I wanted to track was a plain colored orange ball.

Color based tracking

Filter only the orange color from the image. To do that, I converted the image to the HSV color-space, and then used the cvInRange filter twice to filter the orange colors.

In the first picture below you can see the video steam converted to the HSV color space, and in the second picture you can see the result of the red color filtering (all colors except red were filtered out).

(python code)

</p>
<p>#declare necessary objects</p>
<p>hsv_frame = cvCreateImage(size, IPL_DEPTH_8U, 3)<br />
thresholded = cvCreateImage(size, IPL_DEPTH_8U, 1)<br />
thresholded2 = cvCreateImage(size, IPL_DEPTH_8U, 1)<br />
hsv_min = cvScalar(0, 50, 170, 0)<br />
hsv_max = cvScalar(10, 180, 256, 0)<br />
hsv_min2 = cvScalar(170, 50, 170, 0)<br />
hsv_max2 = cvScalar(256, 180, 256, 0)</p>
<p># convert to HSV for color matching<br />
# as hue wraps around, we need to match it in 2 parts and OR together<br />
cvCvtColor(frame, hsv_frame, CV_BGR2HSV)<br />
cvInRangeS(hsv_frame, hsv_min, hsv_max, thresholded)<br />
cvInRangeS(hsv_frame, hsv_min2, hsv_max2, thresholded2)<br />
cvOr(thresholded, thresholded2, thresholded)</p>
<p>


Shape based tracking

Then, I used Hough transform to detect the shape of a circle. Before applying the hough transform I smoothed the image because it seems to improve the results.

</p>
<p># pre-smoothing improves Hough detector<br />
cvSmooth(thresholded, thresholded, CV_GAUSSIAN, 9, 9)<br />
circles = cvHoughCircles(thresholded, storage, CV_HOUGH_GRADIENT, 2, thresholded.height/4, 100, 40, 20, 200)</p>
<p>

The result

The result is pretty impressive. The code functions very well and detects the ball under most circumstances (i.e. the ball is far from the camera, close to the camera, slow movement, fast movement, etc).

You can see the result in this video:

Full python sources including servo movement

<br />
/*****************************************************************************************<br />
*  Name    : Fast object tracking using the OpenCV library                               *<br />
*  Author  : Lior Chen &lt;chen.lior@gmail.com&gt;                                             *<br />
*  Notice  : Copyright (c) Jun 2010, Lior Chen, All Rights Reserved                      *<br />
*          :                                                                             *<br />
*  Site    : http://www.lirtex.com                                                       *<br />
*  WebPage : http://www.lirtex.com/robotics/fast-object-tracking-robot-computer-vision   *<br />
*          :                                                                             *<br />
*  Version : 1.0                                                                         *<br />
*  Notes   : By default this code will open the first connected camera.                  *<br />
*          : In order to change to another camera, change                                *<br />
*          : CvCapture* capture = cvCaptureFromCAM( 0 ); to 1,2,3, etc.                  *<br />
*          : Also, the code is currently configured to tracking RED objects.             *<br />
*          : This can be changed by changing the hsv_min and hsv_max vectors             *<br />
*          :                                                                             *<br />
*  License : This program is free software: you can redistribute it and/or modify        *<br />
*          : it under the terms of the GNU General Public License as published by        *<br />
*          : the Free Software Foundation, either version 3 of the License, or           *<br />
*          : (at your option) any later version.                                         *<br />
*          :                                                                             *<br />
*          : This program is distributed in the hope that it will be useful,             *<br />
*          : but WITHOUT ANY WARRANTY; without even the implied warranty of              *<br />
*          : MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               *<br />
*          : GNU General Public License for more details.                                *<br />
*          :                                                                             *<br />
*          : You should have received a copy of the GNU General Public License           *<br />
*          : along with this program.  If not, see &lt;http://www.gnu.org/licenses/&gt;        *<br />
******************************************************************************************/<br />
#!/usr/bin/python<br />
# -*- coding: utf-8 -*-</p>
<p>from opencv.cv import *<br />
from opencv.highgui import *<br />
from threading import Thread<br />
#import serial</p>
<p>class RobotVision:<br />
	cvSize size<br />
	cvImage hsv_frame, thresholded, thresholded2<br />
	cvScalar hsv_min, hsv_max, hsv_min2, hsv_max2<br />
	cvCapture capture;</p>
<p>	def InitBallTracking():<br />
		globals size,  hsv_frame, thresholded, thresholded2, hsv_min, hsv_max, hsv_min2, hsv_max2, capture<br />
		print &quot;Initializing ball Tracking&quot;<br />
		size = cvSize(640, 480)<br />
		hsv_frame = cvCreateImage(size, IPL_DEPTH_8U, 3)<br />
		thresholded = cvCreateImage(size, IPL_DEPTH_8U, 1)<br />
		thresholded2 = cvCreateImage(size, IPL_DEPTH_8U, 1)</p>
<p>		hsv_min = cvScalar(0, 50, 170, 0)<br />
		hsv_max = cvScalar(10, 180, 256, 0)<br />
		hsv_min2 = cvScalar(170, 50, 170, 0)<br />
		hsv_max2 = cvScalar(256, 180, 256, 0)</p>
<p>		storage = cvCreateMemStorage(0)</p>
<p>		# start capturing form webcam<br />
		capture = cvCreateCameraCapture(-1)</p>
<p>		if not capture:<br />
			print &quot;Could not open webcam&quot;<br />
			sys.exit(1)</p>
<p>		#CV windows<br />
		cvNamedWindow( &quot;Camera&quot;, CV_WINDOW_AUTOSIZE );</p>
<p>	def TrackBall(i):<br />
		t = Thread(target=TrackBallThread, args=(i,))<br />
		t.start()</p>
<p>	def TrackBallThread(num_of_balls):<br />
		globals size,  hsv_frame, thresholded, thresholded2, hsv_min, hsv_max, hsv_min2, hsv_max2, capture<br />
		while 1:<br />
			# get a frame from the webcam<br />
			frame = cvQueryFrame(capture)</p>
<p>			if frame is not None:</p>
<p>				# convert to HSV for color matching<br />
				# as hue wraps around, we need to match it in 2 parts and OR together<br />
				cvCvtColor(frame, hsv_frame, CV_BGR2HSV)<br />
				cvInRangeS(hsv_frame, hsv_min, hsv_max, thresholded)<br />
				cvInRangeS(hsv_frame, hsv_min2, hsv_max2, thresholded2)<br />
				cvOr(thresholded, thresholded2, thresholded)</p>
<p>				# pre-smoothing improves Hough detector<br />
				cvSmooth(thresholded, thresholded, CV_GAUSSIAN, 9, 9)<br />
				circles = cvHoughCircles(thresholded, storage, CV_HOUGH_GRADIENT, 2, thresholded.height/4, 100, 40, 20, 200)</p>
<p>				# find largest circle<br />
				maxRadius = 0<br />
				x = 0<br />
				y = 0<br />
				found = False<br />
				for i in range(circles.total):<br />
					circle = circles[i]<br />
					if circle[2] &gt; maxRadius:<br />
						found = True<br />
						maxRadius = circle[2]<br />
						x = circle[0]<br />
						y = circle[1]</p>
<p>				cvShowImage( &quot;Camera&quot;, frame );</p>
<p>				if found:<br />
					print &quot;ball detected at position:&quot;,x, &quot;,&quot;, y, &quot; with radius:&quot;, maxRadius</p>
<p>					if x &gt; 420:<br />
						# need to pan right<br />
						servoPos += 5<br />
						servoPos = min(140, servoPos)<br />
						servo(2, servoPos)<br />
					elif x &lt; 220:<br />
						servoPos -= 5<br />
						servoPos = max(40, servoPos)<br />
						servo(2, servoPos)<br />
					print &quot;servo position:&quot;, servoPos<br />
				else:<br />
					print &quot;no ball&quot;<br />

Sample Sources

This c++ code takes a video stream from an attached video camera, looks for an orange ball inside the stream, and prints the coordinates of the ball.
Three “debug” windows are shown to clarify the process: 1) the video capture. 2) the stream after the conversion to HSV, 3) the stream after conversion to HSV, color-filtering, and Hough transform.

<br />
/*****************************************************************************************<br />
*  Name    : Fast object tracking using the OpenCV library                               *<br />
*  Author  : Lior Chen &lt;chen.lior@gmail.com&gt;                                             *<br />
*  Notice  : Copyright (c) Jun 2010, Lior Chen, All Rights Reserved                      *<br />
*          :                                                                             *<br />
*  Site    : http://www.lirtex.com                                                       *<br />
*  WebPage : http://www.lirtex.com/robotics/fast-object-tracking-robot-computer-vision   *<br />
*          :                                                                             *<br />
*  Version : 1.0                                                                         *<br />
*  Notes   : By default this code will open the first connected camera.                  *<br />
*          : In order to change to another camera, change                                *<br />
*          : CvCapture* capture = cvCaptureFromCAM( 0 ); to 1,2,3, etc.                  *<br />
*          : Also, the code is currently configured to tracking RED objects.             *<br />
*          : This can be changed by changing the hsv_min and hsv_max vectors             *<br />
*          :                                                                             *<br />
*  License : This program is free software: you can redistribute it and/or modify        *<br />
*          : it under the terms of the GNU General Public License as published by        *<br />
*          : the Free Software Foundation, either version 3 of the License, or           *<br />
*          : (at your option) any later version.                                         *<br />
*          :                                                                             *<br />
*          : This program is distributed in the hope that it will be useful,             *<br />
*          : but WITHOUT ANY WARRANTY; without even the implied warranty of              *<br />
*          : MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               *<br />
*          : GNU General Public License for more details.                                *<br />
*          :                                                                             *<br />
*          : You should have received a copy of the GNU General Public License           *<br />
*          : along with this program.  If not, see &lt;http://www.gnu.org/licenses/&gt;        *<br />
******************************************************************************************/</p>
<p>#include &lt;opencv/cvaux.h&gt;<br />
#include &lt;opencv/highgui.h&gt;<br />
#include &lt;opencv/cxcore.h&gt;<br />
#include &lt;stdio.h&gt;</p>
<p>#include &lt;stdio.h&gt;<br />
#include &lt;stdlib.h&gt;<br />
#include &lt;string.h&gt;<br />
#include &lt;assert.h&gt;<br />
#include &lt;math.h&gt;<br />
#include &lt;float.h&gt;<br />
#include &lt;limits.h&gt;<br />
#include &lt;time.h&gt;<br />
#include &lt;ctype.h&gt;</p>
<p>int main(int argc, char* argv[])<br />
{</p>
<p>    // Default capture size - 640x480<br />
    CvSize size = cvSize(640,480);</p>
<p>    // Open capture device. 0 is /dev/video0, 1 is /dev/video1, etc.<br />
    CvCapture* capture = cvCaptureFromCAM( 0 );<br />
    if( !capture )<br />
    {<br />
            fprintf( stderr, &quot;ERROR: capture is NULL \n&quot; );<br />
            getchar();<br />
            return -1;<br />
    }</p>
<p>    // Create a window in which the captured images will be presented<br />
    cvNamedWindow( &quot;Camera&quot;, CV_WINDOW_AUTOSIZE );<br />
    cvNamedWindow( &quot;HSV&quot;, CV_WINDOW_AUTOSIZE );<br />
    cvNamedWindow( &quot;EdgeDetection&quot;, CV_WINDOW_AUTOSIZE );</p>
<p>    // Detect a red ball<br />
    CvScalar hsv_min = cvScalar(150, 84, 130, 0);<br />
    CvScalar hsv_max = cvScalar(358, 256, 255, 0);</p>
<p>    IplImage *  hsv_frame    = cvCreateImage(size, IPL_DEPTH_8U, 3);<br />
    IplImage*  thresholded   = cvCreateImage(size, IPL_DEPTH_8U, 1);</p>
<p>    while( 1 )<br />
    {<br />
        // Get one frame<br />
        IplImage* frame = cvQueryFrame( capture );<br />
        if( !frame )<br />
        {<br />
                fprintf( stderr, &quot;ERROR: frame is null...\n&quot; );<br />
                getchar();<br />
                break;<br />
        }</p>
<p>        // Covert color space to HSV as it is much easier to filter colors in the HSV color-space.<br />
        cvCvtColor(frame, hsv_frame, CV_BGR2HSV);<br />
        // Filter out colors which are out of range.<br />
        cvInRangeS(hsv_frame, hsv_min, hsv_max, thresholded);</p>
<p>        // Memory for hough circles<br />
        CvMemStorage* storage = cvCreateMemStorage(0);<br />
        // hough detector works better with some smoothing of the image<br />
        cvSmooth( thresholded, thresholded, CV_GAUSSIAN, 9, 9 );<br />
        CvSeq* circles = cvHoughCircles(thresholded, storage, CV_HOUGH_GRADIENT, 2,<br />
                                        thresholded-&gt;height/4, 100, 50, 10, 400);</p>
<p>        for (int i = 0; i &lt; circles-&gt;total; i++)<br />
        {<br />
            float* p = (float*)cvGetSeqElem( circles, i );<br />
            printf(&quot;Ball! x=%f y=%f r=%f\n\r&quot;,p[0],p[1],p[2] );<br />
            cvCircle( frame, cvPoint(cvRound(p[0]),cvRound(p[1])),<br />
                                    3, CV_RGB(0,255,0), -1, 8, 0 );<br />
            cvCircle( frame, cvPoint(cvRound(p[0]),cvRound(p[1])),<br />
                                    cvRound(p[2]), CV_RGB(255,0,0), 3, 8, 0 );<br />
        }</p>
<p>        cvShowImage( &quot;Camera&quot;, frame ); // Original stream with detected ball overlay<br />
        cvShowImage( &quot;HSV&quot;, hsv_frame); // Original stream in the HSV color space<br />
        cvShowImage( &quot;After Color Filtering&quot;, thresholded ); // The stream after color filtering</p>
<p>        cvReleaseMemStorage(&amp;storage);</p>
<p>        // Do not release the frame!</p>
<p>        //If ESC key pressed, Key=0x10001B under OpenCV 0.9.7(linux version),<br />
        //remove higher bits using AND operator<br />
        if( (cvWaitKey(10) &amp; 255) == 27 ) break;<br />
    }</p>
<p>     // Release the capture device housekeeping<br />
     cvReleaseCapture( &amp;capture );<br />
     cvDestroyWindow( &quot;mywindow&quot; );<br />
     return 0;<br />
   }<br />

Subscribe now via RSS and never miss a post!

51 Responses to "Fast Object Tracking – Robot Computer Vision"

  • triveni says:
  • alexting says:
  • Dinesh Gosavi says:
  • Jaka says:
  • Balakrishna says:
  • Alireza says:
  • falkonin says:
  • arijit says:
  • ban says:
  • debanjan says:
  • Suchet says:
  • endah says:
  • Nick says:
  • Peter says:
  • Peter says:
  • ti7aaa says:
  • Inprimus says:
  • Kai says:
  • Kai says:
  • Kai says:
  • ROYY says:
  • Nikhil says:
  • Rafi... says:
  • Shawn says:
  • Ross says:
  • alin says:
  • bnc says:
  • swatz says:
  • Doli says:
  • Rizwan says:
  • Rizwan says:
  • Mox says:
  • Morpheus says:
  • andi says:
  • behzad says:
  • SAgirl:) says:
  • amit says:
  • RoboGraphica says:
  • hSaif says:
  • joshu says:
    • Lior Chen says:
  • NIKHIL says:
  • voelker says:
    • Lior Chen says:
  • bebatech says:
    • tcmichals says:
    • Lior Chen says:
  • Andre (ddois) says: