Want to do shot boundary detection via SVM, what are some good features?
Basically, Shot Boundary detection technique's output is affected by camera motion, object motion and illumination conditions. The feature you choose should be invariant to these issues. The problem in your case may be due to poor training dataset/ imbalanced dataset. threshold can be fixed experimentally and it is tedious.