Recognizing object of interest in object tracking using Lucas-Kanade method
Tracking objects can be simple to complex depending on what type of background you have, whether the background is static or moving, whether the object is clearly distinguishable or can share similar properties with background there by blending in the background, and whether we have unique object at hand or we are tracking multiple objects which may cross the trajectories. 

The good news is, the solution to all of these exists. Depending on the nature of task at hand, you may choose simple to complex solution. 

What exactly we need? 

As you implement any tracking algorithm, you have an estimate of where the new position of the object is likely be. Now you need to verify that indeed the object is at the same position or different. The difference is the error that you feedback to your model. For this you need to identify/segment the object in the vicinity where your projected value. In other words you need to detect that object with in a given window. 

If the object's appearance or key behaviour doesn't change quite drastically you can capture feature vectors from original reference and compare the same around the vicinity where your target is expected to be. This is search of feature vector within a window based on some distance criteria. 

So the question boils down to - what should be the feature vector and what should be the distance matrix?

There are many possible answers. Writing down some of them (from simple to complex) with remark on when they might work.


Color based features: This include identifying mean or dominant color, color histogram over a local window etc. If the object color is unique and doesn't blend with background then you can identify object with this manner. See [Ref 1] and [Ref 2] below 
Texture based features: There is a whole range of features here. The famous Gabor filter banks, then there is co-occurrence matrix, you can take gradient of images or find edges and have edge histograms over local regions, or you can identify Local binary patterns near the regions.
See [Ref 3]. Also, many approaches combines Color and Texture information for more effectiveness. See [Ref 4] and [Ref 5]
Shape based features: Shape based technique works when the object you are tracking has a particular shapes that distinguishes from other objects and from background. This includes many techniques. See [Ref 6], [Ref 7] and [Ref 8] below. 


More complex features:

If the above features doesn't work out, specifically if the object is complex and the background as well -then we can use these features:


SIFT - The scale-invariant feature transform allows to compute key points which characterises the object and hence you can compare the same object from past frame against a new frame.  See [Ref 9] 
SURF, FAST, ORB:  - If SIFT turns out to be blocked because of patent issue, and due to it's speed, then you can use it's alternatives which are SURF, FAST, ORB. See [Ref 10] for a good comparison.
HoG and Haar: Finally you can use HoG - which is a good generic object detection framework. [Ref 11] and Haar based classifiers when the problems are really complex. [Ref 12] 


I am not prescribing any specific solution, and none of the above could be your direct solution - yet this will help you thinking through the problem at hand. You can pick whichever feature works for you irrespective of tracking methods that are used in those respective papers.   

Go through the paper Good Features to Track  thoroughly and it will give you much better perspective on the same. 

What about distance criteria?

For each different feature typically different distance measure should make sense. But, distance matrix is usually dependent on your feature vector itself. 



[Ref 1]: Object Tracking Based on Color Information
Employing Particle Filter Algorithm Budi Sugandi, Hyoungseop Kim et. al. 

[Ref 2]: Applying a New Spatial Color Histogram in Mean-Shift
Based Tracking Algorithm Dong Xu, Yimin Wang, and Jinwen An

[Ref 3]: Texture Features for Browsing and Retrieval of Image Data B.S. Manjunathi and W.Y. Ma 

[Ref 4]: ROBUST OBJECT TRACKING USING JOINT COLOR-TEXTURE HISTOGRAM JIFENG NING, LEI ZHANG and DAVID ZHANG, CHENGKE WU

[Ref 5]: Multi-Object Tracking Using Color, Texture and Motion Valtteri Takala and Matti Pietikainen

[Ref 6] :  A Survey on Feature Extraction Techniques for Shape based Object Recognition Mitisha Patel, Purvi Tandel 

[Ref 7]: RAPID AND ROBUST HUMAN DETECTION AND TRACKING BASED ON OMEGA-SHAPE
FEATURES Min Li, Zhaoxiang Zhang, Kaiqi Huang and Tieniu Tan

[Ref 8]: An Evaluation of Local Shape-Based Features for Pedestrian Detection Edgar Seemann, Bastian Leibe, Krystian Mikolajczyk, Bernt Schiele

[Ref 9]: Object Tracking using SIFT Features and Mean Shift Huiyu Zhou, Yuan Yuan, and Chunmei Shi 

[Ref 10]: Image Matching Using SIFT, SURF, BRIEF and ORB: Performance Comparison for Distorted Images Ebrahim Karami, Siva Prasad, and Mohamed Shehata

[Ref 11]: Part-based Multiple-Person Tracking with Partial Occlusion Handling Guang Shu, Afshin Dehghan et. al. 

[Ref 12]: Real-time hand tracking using a set of cooperative classifiers based on Haar-like features Andre L. C. Barczak and Farhad Dadgostar