1. A method, in a data processing system comprising a processor and a memory, for dynamically selecting a subset of candidate answer sources for use by a question and answer (QA) system implemented by the data processing system, the method comprising: receiving, by the QA system implemented by the data processing system, an input question for which an answer is sought; generating, by the QA system, one or more queries based on the input question; selecting, by the QA system, a subset of candidate answer sources, from a group of candidate answer sources, based on a ranked listing of candidate answer sources, wherein the ranked listing of candidate answer sources is generated based on characteristics of previous candidate answers generated by the QA system for previously input questions; applying, by the QA system, the one or more queries to the subset of candidate answer sources; and generating, by the QA system, at least one candidate answer for the input question based on results of applying the one or more queries to the subset of candidate answer sources.