decision rules for each feature (binary classification)
You are describing every binary classifier.  However, you are missing a key point. If your classes are separable by the value of just ONE feature, you can do what you're saying and find e.g. F1 > 1.23 as a threshold.  If classification involves a combination of features, you will need to describe some combination of thresholds for each feature, or (equivalently) some relationship between the features that tells you about the class label.  It's the job of every binary classifier to do exactly this - they just do it in different ways.  See for example this post.  Your desire to have a combination of fixed sets of thresholds will only work if you can have a set of thresholds that will encompass/describe/classify every combination of feature values.

If you want a set of easy-to-read thresholds like you mention, you should read about decision tree classifiers.  They'll do something like what you want - but will also ensure that you provide a class label for every possible combination fo features values. The nice about about decision trees is that they'll let you leave out your current feature selection step - they just do it for you by (1) picking the feature that best discriminates the two classes overall, (2) picking the threshold value of that feature that gives the most information about the class label (usually), and (3) repeating (1-2) several times.