273691a7__Relevance_Issues___Lucidworks__Name
[Description] Given a set of queries from beta testing or an existing system, select the top X queries in terms of volume and Y randomly selected queries not in the top X. Have your QA or business team execute the queries against the system and rate the top five or ten results as relevant, somewhat relevant and not relevant. Alternatively, you could ask them to make binary decisions (relevant or not) or add in a fourth option: embarrassing. X is usually between 25-50 and Y is often between 10 and 20, but you should select based on available time and resources. [Pros] Real queries, real documents, real results. Laser-like focus on those queries that are most important to your system, without neglecting the others. Also, allows unhappy users to directly, and quantifiably show how the system doesn’t meet their expectations. [Cons] Time consuming depending on your choice of X and Y. Judgments are still subjective and may not extrapolate to others. [Name] 
Empirical Testing
A d - h o c
F o c u s G r o u p s
T R E C a n d o t h e r P u b l i c E v a l u a t i o n s
O n l i n e R a t i n g s
L o g A n a l y s i s o n a B e t a P r o d u c t i o n S i t e
A / B T e s t i n g
E m p i r i c a l T e s t i n g
Debugging Search Application Relevance Issues - Lucidworks
Name