13. In a computing environment, a system, comprising, at least one processor, a memory communicatively coupled to the at least one processor and including components comprising, an evaluation mechanism configured to run on the at least one processor and compare a first search engine implementation with the second search engine implementation using a task-based assessment metric, wherein the evaluation mechanism is further configured to extract first sets of tasks from a dataset comprising samples of user activity logs for the first search engine implementation and second sets of tasks from another dataset comprising samples of user activity logs for the second search engine implementation, to compute a first task success rate for each of the first sets of tasks and a second task success rate for each of the second sets of tasks, to apply the task-based assessment metric to the first sets of tasks and the second sets of tasks to produce a first distribution and a second distribution, respectively, and to compare the first distribution and the second distribution to determine indicia of a performance difference between the first search engine implementation and the second search engine implementation in terms of user behavior.
13. In a computing environment, a system, comprising, at least one processor, a memory communicatively coupled to the at least one processor and including components comprising, an evaluation mechanism configured to run on the at least one processor and compare a first search engine implementation with the second search engine implementation using a task-based assessment metric, wherein the evaluation mechanism is further configured to extract first sets of tasks from a dataset comprising samples of user activity logs for the first search engine implementation and second sets of tasks from another dataset comprising samples of user activity logs for the second search engine implementation, to compute a first task success rate for each of the first sets of tasks and a second task success rate for each of the second sets of tasks, to apply the task-based assessment metric to the first sets of tasks and the second sets of tasks to produce a first distribution and a second distribution, respectively, and to compare the first distribution and the second distribution to determine indicia of a performance difference between the first search engine implementation and the second search engine implementation in terms of user behavior. 17. The system of claim 13 further comprising an evaluation classifier configured to identify one or more first queries associated with each task of the first sets of tasks and one or more second queries associated with each task of the second set of tasks.