How can I measure the usability of a catalogue? This question might seems vague but heres the context:

When we are focusing on HCI we would most likely be interested on knowing first how the user usually deals with a certain object. We then try to see how our system could take away one of the tasks he would do himself and try to do it itself. 


The object of my interest here is a simple paper catalogue. How would you measure its usability (paper one). 
Then, how would you map it to a system interface? How would you measure the usability now on the system?
How would you compare the two usabilities measures?


This question narrows down this approach which is suggested on Stones book - User Interface and Evaluation. 

What the catalogue is about is not the point, that why I left it without a description: To avoid suggestions trying to measure what the catalogue is about. My focus here is on the particular mapping of this kind of object on the real world as a simple paper and when it is mapped to a system interface. Assume the catalogue to consist of rows and tables, where each matching row and table gives you a suggestion and you must first reason about each row and each column to see if it suits you (Perhaps you would suggest another template for the catalogue?).
In the past I worked on usability of interfaces in the information retrieval area, so I give you some practical ideas on how to measure the usability.

You didn't give details of what a "catalogue" is, so I will assume that a user has a problem (or need) and he is searching your catalogue (in paper or electronic format) for one or more solutions.

Then you can build a set of problems $P = \{p_1,...,p_n\}$, and for each problem a set $S^i$ of (possibly weighted) correct solutions: $S^i = \{ s^{i}_1,...,s^{i}_m\}$.

Then you can setup a test with real users.

Every user must solve all the problems in $P$ (in shuffled order) using your catalogue, in one of the two format (paper or electronic).

For each problem, you measure the time spent solving it. At the end you ask the users to give some judgements of satisfaction (using weighted values) on the two catalogues (covering different aspects of satisfaction, for example: "Do you think the catalogue helped you?", "Do you think it is easy to learn how to use it",  ....)

Then you can measure (and compare) the usability of the two catalogues in term of:


standard precision and recall of the results found for each problem $p_i$. Precision: number of correct solutions among the total number of answers; recall: number of correct solutions found among the total number of correct solutions in $S^i$
time spent for solving each problem $p_i$ (efficiency of the system);
user satisfaction


For what concerns the mapping between a paper catalogue to a system interface, you can "record" what users do during their searches on the paper catalogue, then try to "simulate" the actions of those users that performed better using an expert system (see expert system design).