Visual tracking, which has been widely used in many vision fields, has been one of the most active research topics in computer vision in recent years. However, there are still challenges in visual tracking, such as illumination change, object occlusion, and appearance deformation. To overcome these difficulties, a reliable point assignment (RPA) algorithm based on wavelet transform is proposed. The reliable points are obtained by searching the location that holds local maximal wavelet coefficients. Since the local maximal wavelet coefficients indicate high variation in the image, the reliable points are robust against image noise, illumination change, and appearance deformation. Moreover, a Kalman filter is applied to the detection step to speed up the detection processing and reduce false detection. Finally, the proposed RPA is integrated into the tracking-learning-detection (TLD) framework with the Kalman filter, which not only improves the tracking precision, but also reduces the false detections. Experimental results showed that the new framework outperforms TLD and kernelized correlation filters with respect to precision, f-measure, and average overlap in percent.