1. A system, comprising: one or more computer processors; and a memory containing a program, which when executed by the one or more computer processors, performs an operation to provide selective supporting evidence processing in a question answering (QA) system, the operation comprising: initiating processing of a first question using an execution pipeline of the QA system, wherein the execution pipeline comprises (i) a first stage configured to determine candidate answers to the first question without consideration of any items of supporting evidence and (ii) a second stage, downstream from the first stage, configured to process items of supporting evidence relating to the candidate answers in order to generate updated confidence scores; upon determining a first candidate answer for a first question during the first stage of the execution pipeline and prior to executing the second stage of the execution pipeline: generating, by the QA system, using a first machine learning (ML) model, a first confidence score value for the first candidate answer, wherein the first confidence score value reflects a degree to which the first candidate answer is a correct response to the first question, wherein the first ML model does not consider supporting evidence features for the first candidate answer; generating, by the QA system, using a second ML model, a measure of expected change to the first confidence score value based at least in part on supporting evidence features for the first candidate answer; and upon determining that the measure of expected change does not exceed a first threshold, returning, by the QA system, at least the first candidate answer in response to the first question, without processing the first question using the second stage of the execution pipeline.