Evaluating new features
Actually in these cases I usually plot the data based on features two by two. This approach is so much similar to watching covariance matrix. In this way you should see the new added features whether they have correlation, whether they change linearly with each of previous features. Suppose that you have already one feature for a classification task. Then you may want to add another feature. You have to plot data with each feature as one axis and investigate whether they have linear correlation or not. If they don't have any correlation or their correlation is near to zero, you should add the feature because that may help you, they provide a kind of knowledge that the previous features didn't provide. If a new feature has correlation with the previous feature, it means that adding that does not help you have a new knowledge or perception of that concept. Although there are debates here I prefer not to add correlated features because of computation complexity. That will be so time consuming. 

For illustrating more, suppose you have a data-set A = {X1, X2, y} in which X1 and X2 are the feature and y is the label and all are binary values. Also suppose that the covariance matrix between these is as follows.




We may infer that we can ignore X1 in the process of classification without loss of accuracy, if e is equal to zero which means label y is independent of X1, since they are binary valued. We can ignore X1 in classification, regardless of other values. 
We may infer that we can ignore either one of features in the process of classification without loss of accuracy, if |d| >> 0 which means there is high correlation between X1 and X2 and hence information redundancy. So we may ignore one of them without problem.
We may infer that we can not ignore neither X1 nor X2 in the process of classification, if d = 0 and none of e or f are zero. Then the feature are independent and both are influential on label. So we may not ignore any of them.