21. A non-transitory physical storage medium storing program code, which when executed by a processor, causes the processor to execute a process for replaying performance of a predictive engine, the program code when executed by the processor causes the processor to perform steps to: deploy an engine variant of the predictive engine based on an engine parameter set; receive a plurality of queries to the engine variant from an end-user device, wherein each of the plurality of queries contains a timestamp; apply a first query segment filter to the plurality of queries to generate a first replay group of queries; for each query in the first replay group, generate utilizing the engine variant a corresponding predicted result, receive a corresponding actual result, and compute a corresponding prediction score based on the corresponding predicted result and the corresponding actual result; group queries in the first replay group into one or more query segments of the first replay group according to the timestamps; and generate, as a replay of the performance of the engine variant, an accumulated prediction score over each of the one or more query segments of the first replay group by applying an accumulation function.
21. A non-transitory physical storage medium storing program code, which when executed by a processor, causes the processor to execute a process for replaying performance of a predictive engine, the program code when executed by the processor causes the processor to perform steps to: deploy an engine variant of the predictive engine based on an engine parameter set; receive a plurality of queries to the engine variant from an end-user device, wherein each of the plurality of queries contains a timestamp; apply a first query segment filter to the plurality of queries to generate a first replay group of queries; for each query in the first replay group, generate utilizing the engine variant a corresponding predicted result, receive a corresponding actual result, and compute a corresponding prediction score based on the corresponding predicted result and the corresponding actual result; group queries in the first replay group into one or more query segments of the first replay group according to the timestamps; and generate, as a replay of the performance of the engine variant, an accumulated prediction score over each of the one or more query segments of the first replay group by applying an accumulation function. 26. The non-transitory physical storage medium of claim 21 , wherein the first query segment filter is determined manually by an operator.