Keypoint matching using HoG and SIFT
In general, you can use brute force or a smart feature matcher implemented in openCV. 
Another approach is seeing the task as image registration based on extracted features. 

The question may be what is the relation of HoG and SIFT if one image has only HoG and other SIFT or both images have detected both features HoG and SIFT.