Disclaimer: I created this question on Cross-Validated where it is well on the way to earning me the Tumbleweed badge, but then I found this site and thought it might be more at home here.I'm using Azure ML, and I've produced a basic PoC model of student success using a Two-class Boosted Decision Tree. One of the features I've included is whether the student has received a particular kind of support. What I'd like to do is test whether providing that support to a particular student will result in an improved success rate.In order to test this, I thought I would run through the details of students who haven't received that support to get the baseline score from the model, then run through the same information but flipping the feature indicating that they've received support, and record the new score.Assuming that the support feature is reasonably independent (i.e. it's causing the label, not caused by it - one that I tried to test initially was a contact type that only happened when the students had been absent, so it wasn't appropriate to apply it to other students), is this a valid tactic for evaluating whether we should provide that support to other students in the future? Are there any pit-falls I can run in to when performing these kinds of tests?
Can I test actions on a decision tree (or other classification model)?
predictive modeling;decision trees;azure ml