I'm having trouble successfully tracking a user-selected feature point using the KLT tracking algorithm and header files. I'm programming in C++ VS Express 2010, and am trying to do this without OpenCV or other libraries. Here are the steps I'm taking to track each point at a time:
Make a tracking context
Make a feature list
Adjust feature list count (to one), set x and y position
Fill two image data containers
Call KLTTrackFeatures() with appropriate arguments
Take new x and y values from feature list, and insert back into custom data structure
That's pretty much it. I've fiddled with some attributes like borders etc to no avail. The results I get can vary from coords marked as -1, to positions where the point shouldn't be. I've found it difficult searching for this one, because most searches result in OpenCV related topics which don't seem to deal directly with klt itself. Anyone have any thoughts or suggestions on how to go about solving this?
Kind regards,
OJnr.
Related
I'm currently working on a multiple object tracking problem. I think using Tracking-by-Detection is a good choice. However, I do not know how to combine tracking and detection result so that detection can help improve tracking results.
I'm using Faster-RCNN, tensorflow object detection API as a simple starting point for detection.
For tracking, I use KCF algorithm from opencv.
Detection is unstable because every frame is independent to the model, while tracking is much more stable.
Although tracking is more stable, when the object moves, tracker can not follow the object, which is not accurate.
So I'm thinking of combining these two methods to improve my result as both stable and accurate.
I have a background of computer vision but I'm new to this field (Multiple Object Tracking). Could anyone please give me some advice on how I should deal with this kind of problem ?
Thanks alot! :)
I have tried to use detection to track objects recently. The unstable problem can be resovled by classic filting techology such as Kalman filtering(In that field, the point from signal processing is also "unstable" due to noise.). You can set a small region around the tracked object and try to find same one in that region in next frame. A "matched" relationship is established from that, and then you try to match the object in next frame from next next one... A trace can be built from the process. Any smoothing method can be employed to suppress predicted box noise. A example can be shown in:
The transparent points are detected trace points and the soild one are smoothed points.
The corresponding trace shown in background:
Some tricks are also useful, if detection fail on some random position, you can set a "skip gate", to try find one matching point in later frame(In my experiment, 60 is not bad for 24fps video). You will prefer recall more than accuracy since you can build a pretty long sequence and drop short noise sequence come from false alarm detection.
Reference code:https://github.com/yiyuezhuo/detection-tracking
I think you should try using CSRT tracker from opencv which is much more stable than KCF. For the detection, you could use it after a fixed set of frames to reinitialize the tracker using the detections. This way you can fuse a tracker with the detector.
I am working on Opencv application that need to count any object which motion can be detected by the camera. The camera is still and I did the object tracking with opencv and cvblob by referring many tutorials.
I found some similar question:
Object counting
And i found this was similar
http://labs.globant.com/uncategorized/peopletracker-people-and-object-tracking/
I am new to OpenCV and I've gone through the opencv documentation but I couldn't find anything which is related to count moving objects in video.
Can any one please give me a idea how to do this specially the counting part. As I read in article above, they count people who crosses the virtual line.Is there a special algorithm to detect the object crossing the line?
Your question might be to broad when you are asking about general technique that count moving objects in video sequences. I would give some hints that might help you:
As usual in computer vision, there does not exist one specific way to solve your problem. Try do do some research about people detection, background extraction and motion detection to have a wider point of view
State more clearly user requirements of your system, namely how many people can occur in the image frame? The things get complicated when you would like to track more than one person. Furthermore, can other moving objects appear on an image (e.g. animals)? If no and only one person are supposed to be track, the answer to your problem is pretty easy, see an explanation below. If yes, you will have to do more research.
Usually you cannot find in OpenCV API direct solution to computer vision problem, namely there is not such method that solve directly problem of people counting. But for sure there exists some paper, reference (usually some scientific stuff) which can be adopted to solve your problem. So there is no method that "count people crossing vertical line". You have to solve problem my merging some algorithms together.
In the link you have provided one can see that they use some algorithm for background extraction which determined what is a non-moving background and moving foreground (in our case, a walking person). We are not sure if they use something more (or sophisticated), but information about background extraction is sufficient to start with problem solving.
And here is my contribution to the solution. Assuming only one person walks in front of the stable placed camera and no other objects motion can be observed, do as following:
Save frame when no person is moving in front of the camera, which will be used later as a reference for background
In a loop, apply some background detector to extract parts in the image representing motion (MOG or even you can just calculate difference between background and current frame, followed by binary threshold and blob counting, see my answer here)
From the assumption, only one blob should be detected (if not, use some metrics the chooses "the best one". for example choose the one with maximum area). That blob is the person we would like to track. Knowing its position on an image, compare to the position of the "vertical line". Objects moving from left to right are exiting and from right to left entering.
Remember that this solution will only work in case of the assumption we stated.
I recently saw the virtual mirror concept on you tube, I tried it out and researched about it. It seems that the creators have used augmented reality so that people can see the output on their screens. On researching I found out that we identify a pattern on which a 3D image is superimposed.
Question 1:How are they able to superimpose the jewellery and track the face of the person without identifying any pattern?
I also tried to check various libraries that I can use to make a program similar to the one they show. Seems to me that a lot of people are using Android phones and iPhones and making apps that use augmented reality.
Question 2:Is there any way that I can use c++ and try to make a program that uses augmented reality?
Oh, and the most important thing, the link to the application is provided below:
http://www.boutiqueaccessories.com.au/virtual-mirror/w1/i1001664/
Do try it out. Its a good experience. :D
I'm not able to actually try the live demo, but the linked video suggests that they either use some simplified pattern recognition (get the person's outline), or they simply track you based on the initial image (with your position/texture being determined by the outline being shown.
Following the video, it's easy to see that there's no real/advanced AR behind this. The images are simply overlayed or hidden (e.g. in case it's missing track of one ear due to you looking to the side) and they're not transformed (no perspective or resizing happening). They definitely seem to track the head (or features like ears, neck, etc.). depending on your background and surroundings that's actually a rather trivial task.
Question 2: Sure! There are lots of premade toolsets out there, but you could as well use some general image processing library such as OpenCV to do the math. Augmented reality usually uses some kind of pattern (e.g. a card or page with a known pattern) to determine the correct position and transformation for the contents to be added to the image. There are also approaches using the device's orientation and perspective changes in camera images to determine depth/position (I really like this demo).
I am currently attempting to use OpenCV 2.3 and c++ to detect a hand (wearing a green glove), and distinguish between different hand gestures.
At this very moment, my next step is to acquire specific features of the hand (convex defects).
So far I've used these functions in my process:
mixChannels(); //to subtract non-green channels from green.
threshold(); //to convert to binary
erode(); dialate(); //to get rid of any excess noise
findContours(); //to find contours of course
these have worked splendidly and I have been able to output findContours() through the use of drawContours().
Next step, and this is where I'm at, is using convexHull(), which also works in OpenCV 2.3. I have however yet to find out how the vector results of convexHull() actually look (what features they contains).
But this is where the tricky part comes.
I found that the older version of OpenCV (using c which uses IplImage), has a neat little function called cvConvexityDefects() which can give a set of deficiencies on the convex hull. These are what I need, but there seems to be no such function for the OpenCV 2.3 and I don't see how I can use the old syntax to get these results.
Here's a link to Open CV documentation on cvConvextDefects.
What I'm asking for, is either a similar OpenCV 2.3 function, or a selfwritten piece of code or algorithm for finding these defects. Or a method to use the old 2.1 syntax for a vector result or something like that.
(I know that I can use other features, rectangular bounding boxes, and fitted circles. But I'm sure that convex defects yield the most distinguishable features.)
Solution - I ended up using a c++ wrapper from this post
The only thing not working for this wrapper seems to be a leek of the defects vector, which should be easily solvable.
Next step is getting some usable data from these defects. (At first glance. the data seems to be single points on the convexHull or the Contour or the count of these. I had at first expected a set of two points, or a single point and a length, which it does not seem to be. If I'm hitting 'brick wall' with this, I'll make another post)
The new C++ interface does not (yet) support all the functions in C. And the opposite is true, also (not everything in cpp is in c). The reasons are various, but the good news is that you can easily use whatever function you want. By example, here you have to convert contours to sequences (CvSeq) and send them to your function.
Moreover, the FindContours method is a wrapper over cvFindContours. You can call it
cvFindContours((IplImage*)matImage, ...);
and then use directly the result.
Another way would be to create a nice, clean C++ wrapper over cvConvexDefects() and submit it to OpenCV. In the findContours source you will find some help for that (the opposite transform)
I would like to attempt to convert my convexHull vector from the c++ syntax to the sequence which you mentioned, but I don't know where to start. Could you perhaps shed some light on this?
Check out this question here, I beleive it goes into it.
Convexity defects C++ OpenCv
What kind of debugging is available for image processing/computer vision/computer graphics applications in C++? What do you use to track errors/partial results of your method?
What I have found so far is just one tool for online and one for offline debugging:
bmd: attaches to a running process and enables you to view a block of memory as an image
imdebug: enables printf-style of debugging
Both are quite outdated and not really what I would expect.
What would seem useful for offline debugging would be some style of image logging, lets say a set of commands which enable you to write images together with text (probably in the form of HTML, maybe hierarchical), easy to switch off at both compile and run time, and the least obtrusive it can get.
The output could look like this (output from our simple tool):
http://tsh.plankton.tk/htmldebug/d8egf100-RF-SVM-RBF_AC-LINEAR_DB.html
Are you aware of some code that goes in this direction?
I would be grateful for any hints.
Coming from a ray tracing perspective, maybe some of those visual methods are also useful to you (it is one of my plans to write a short paper about such techniques):
Surface Normal Visualization. Helps to find surface discontinuities. (no image handy, the look is very much reminiscent of normal maps)
color <- rgb (normal.x+0.5, normal.y+0.5, normal.z+0.5)
Distance Visualization. Helps to find surface discontinuities and errors in finding a nearest point. (image taken from an abandoned ray tracer of mine)
color <- (intersection.z-min)/range, ...
Bounding Volume Traversal Visualization. Helps visualizing a bounding volume hierarchy or other hierarchical structures, and helps to see the traversal hotspots, like a code profiler (e.g. Kd-trees). (tbp of http://ompf.org/forum coined the term Kd-vision).
color <- number_of_traversal_steps/f
Bounding Box Visualization (image from picogen or so, some years ago). Helps to verify the partitioning.
color <- const
Stereo. Maybe useful in your case as for the real stereographic appearance. I must admit I never used this for debugging, but when I think about it, it could prove really useful when implementing new types of 3d-primitives and -trees (image from gladius, which was an attempt to unify realtime and non-realtime ray tracing)
You just render two images with slightly shifted position, focusing on some point
Hit-or-not visualization. May help to find epsilon errors. (image taken from metatrace)
if (hit) color = const_a;
else color = const_b
Some hybrid of several techniques.
Linear interpolation: lerp(debug_a, debug_b)
Interlacing: if(y%2==0) debug_a else debug_b
Any combination of ideas, for example the color-tone from Bounding Box Visualization, but with actual scene-intersection and lighting applied
You may find some more glitches and debugging imagery on http://phresnel.org , http://phresnel.deviantart.com , http://picogen.deviantart.com , and maybe http://greenhybrid.deviantart.com (an old account).
Generally, I prefer to dump bytearray of currently processed image as raw data triplets and run Imagemagick to create png from it with number e.g img01.png. In this way i can trace the algorithms very easy. Imagemagick is run from the function in the program using system call. This make possible do debug without using any external libs for image formats.
Another option, if you are using Qt is to work with QImage and use img.save("img01.png") from time to time like a printf is used for debugging.
it's a bit primitive compared to what you are looking for, but i have done what you suggested in your OP using standard logging and by writing image files. typically, the logging and signal export processes and staging exist in unit tests.
signals are given identifiers (often input filename), which may be augmented (often process name or stage).
for development of processors, it's quite handy.
adding html for messages would be simple. in that context, you could produce viewable html output easily - you would not need to generate any html, just use html template files and then insert the messages.
i would just do it myself (as i've done multiple times already for multiple signal types) if you get no good referrals.
In Qt Creator you can watch image modification while stepping through the code in the normal C++ debugger, see e.g. http://labs.qt.nokia.com/2010/04/22/peek-and-poke-vol-3/