Image may be NSFW.
Clik here to view.
Of all the emails I receive thru this website, the question I am asked the most is “how do I teach OpenCV to detect my own objects?” Although the questions are usually worded very differently, they’re all pretty much asking the same thing. “How do I create my own OpenCV Haar Classifiers?” Even though I’ll be explaining this process in much greater detail in my upcoming book, Computer Vision with OpenCV and C#, I’ve decided to go ahead and share a much slimmer explanation here. I’ll do my best to explain things the best I can in the shortest amount of time as possible. But, as always, if you have any questions after reading this article, please leave them in the comments below and I’ll answer you as soon as possible. With that said, let’s begin.
Creating your own Haar classifiers is pretty much a 3 step process. The process begins by gathering material that will be fed to OpenCV which it will be trained to detect. Training material consists of 3 groups of sub-material. The first sub-material you will need is your positive set. Your positive set is basically a collection of images that contain the object you want OpenCV to detect. In this article, we will be teaching OpenCV to detect Blackberry cellphones within images and video. So, our positive set will contain a collection of images that all have a Blackberry in them. The second sub-material you will need will be your negative set. The negative set is basically a collection of images that do not contain the object you want detected. So, for the purposes of this article, we will need a collection of images that do not include a Blackberry. The third and final sub-material you will need is another set of images that do contain the object you want to detect. This set of images will be used for testing our classifier later on.
For training a Haar classifier, it is recommended to use as many training samples as possible. However, I’ll warn you now that the training process takes a long time and the more training samples you use, the longer it will take. In order to create the Haar classifier for this article, it took 5 days of non-stop processing on one of my extra laptops (2.8GHz, 4GB RAM, Windows Vista). My training set consisted of 2000 positives, 2000 negatives, and 500 tests. Let me explain how I created those image sets.
To create your positives, you will need as many pictures of your object as possible, Blackberry cellphones in my case. Now, you can grab a camera and start snapping pictures of your object from several different angles and in different lighting environments. But, this can take forever, especially when you’re considering thousands of images. Instead, I found that it was easier to record a couple of videos of my object from different angles and using different lighting. Then, all I had to do was extract each image from the videos one frame at a time and crop them accordingly.
To do that, I made use of a tool called “Positive Builder” which is a simple Windows application written in C# that can be downloaded from https://code.google.com/p/opencv-haar-cascade-positive-image-builder/. Even though it’s called “Positive Builder”, it’s also capable of creating our negatives which we’ll do shortly. As mentioned, Positive Builder is written in C#, Visual C# 2005 to be exact. I’m not sure if there is a newer version available somewhere else. But, the version I got from the address above seemed to be incomplete. So, I have updated the project to work with Visual C# 2010 and have made a lot of adjustments to it. I have added a lot of the missing pieces and have finished up some parts that were lacking. With my new version, you should have everything you need to make full use of this tool. You can download my updated version of the app from http://www.prodigyproductionsllc.com/downloads/opencv_haar_cascade_postitive_builder.zip. (Update: The 64bit version can be downloaded at the end of this article.) On a side note, Positive Builder uses OpenCvSharp for the heavy lifting which I especially like this since OpenCvSharp is the OpenCV wrapper I prefer to work with in C#.
Once you have the app loaded in Visual C#, you will need to open App.config from the Solution Explorer. This file contains the path and filename of the video you want to get your frames from. It also contains the folder path that you want your images output to as well as a string to prefix your image names with and a number to begin your file name incrementing from. This number is helpful if you’ve already processed several images, closed and / or restarted the app, and want to pick up from where you left off. After you have setup your path variables, go ahead and launch the application.
When you do, you should see the first frame of your video in the form. If so, you will need to specify where your object is located within the frame. To do that, click once in the upper-left corner just above where your object is located and click a second time in the lower-right corner just outside of where your object is located. This will draw a green box that should outline only your object. If so, press the “P” key 1 time to save that image into your positives folder and write its dimensions into positives.txt of your output folder. After the frame has been saved to your positives folder, the app will automagically jump to the next frame in the video if there is one. Once you reach the last frame of your video, the app will no longer skip to the next frame since there are no more. If there are more frames, the app will automagically jump to the next frame where you will need to repeat the process until you have cropped all of your positive images.
Image may be NSFW.
Clik here to view.
The Positive Builder tool also includes other functions which are defined on the form. For instance, if you come across a frame that does not include your object or the object is obstructed too much by something else, you can press the “Q” key to query the next frame in the video. If you draw your box and aren’t happy with it, you can press the “R” key to reset your lines and start over with the current frame. If you come across a frame that does not include your object, you can press the “N” key to flag it as a negative image. Remember, negative images are images that do not contain the object you want to detect.
To quickly create negative images, you can load up a video that you know does not contain your object and keep pressing the “N” key until all frames have been processed as negatives. You can also press the “2″ key to process every frame in a video as negatives. Then, if any of your frames did in fact contain your object, you can browse to your negatives folder and manually delete the images that did contain your object. You can also do the same for positives by pressing the “1″ key to process every frame as a positive frame that does include your object. However, I wouldn’t recommend doing this because your object won’t be cropped and will return the width and height of the video as the cropped dimensions. So, be cautious when using the “All Positives” option.
When cropping and processing your positive images, each image will be stored in your positives folder. And, all positive images get recorded in a file called “positives.txt”. This file contains the full path and filename of each positive image as well as a few other parameters. The first parameter after the image name is an indicator that tells OpenCV how many instances of your object can be found within that image. The rest of the parameters are the coordinates of your object within the image (left, top, width, height). The “negatives.txt” file only contains the full path and file names of your negative image set.
Now that we have all of our negative and positive images, it’s time to create our .vec file. To do this, we will make use of our “createsamples.exe” file that comes with OpenCV. You can either copy the exe file into the same folder that contains your positives.txt and negatives.txt files or you can reference the executables by fully qualifying the path to where createsamples.exe lives (probably inside C:\Program files\OpenCV\bin\). Or, you can always use my version of Positive Builder by pressing the “V” button which will dump your .vec file into the same folder as your positives.txt and negatives.txt files (output folder).
Once you have your .vec file, it’s just a matter of Haar training. All you have to do to build your cascade at this point is to press the “T” key and the “haarTraining.exe” application will take care of the rest. This is the final step, yet it is also the longest. This is the step that took 5 days to run for me. I’ve read several articles about other OpenCV users getting the same results. So, I know it’s the application and not just a fluke. Anyways, when the Haar training has completed, you will have yourself a brand new haarcascade.xml file in your output folder. You can now test out your shiny new cascade by loading it up in any of the OpenCV head, face, or eye detection and tracking applications I’ve already shared with you on this website.
P.S. I’ve already begun updating the Positive Builder application. My next version will include a built-in video recorder which you can use for stripping frames out of for your positive and negative images. The next version will also include a built-in testing tool that you can use to auto-load your new cascade into for testing. I’ll also be doing a lot of cleanup to make the application easier to use and will be providing a way to pick your output folder, images, and other parameters without having to change the app.config file. Until then, this should hopefully help you get the job done. Let me know what you think about this article in the comments below. Let me know if you found it helpful or not. Also, feel free to leave your suggestions and questions in the comments below as well. Until next time, HAPPY CODING!
Update: Here is the updated 64bit version of the Positive Builder. It is also using the latest versions of OpenCV and OpenCvSharp (2.3.1). http://www.prodigyproductionsllc.com/downloads/opencv_positive_builder_64bit.zip
Originally posted at http://www.prodigyproductionsllc.com/articles/programming/how-to-train-opencv-haar-classifiers/