PROMETHEUS: PROcedural METhodology for developing HEuristics of USability
Cristhy Jimenez Hector Allende-Cid Ismael Figueroa






: PROcedural METhodology for
  developing HEuristics of USability
    Cristhy Jiménez, Héctor Allende Cid,
  Ismael Figueroa
  The original version of this work appears in Spanish in the
    IEEE Latin American Transactions
    journal <cit.>. We present a version in
    English for further dissemination of our work. C. Jiménez,
    H. Allende Cid, I. Figueroa, Pontificia Universidad Católica de
    Valparaíso, Chile. <cristhy.jimenez.g@mail.pucv.cl>,
    <hector.allende@pucv.cl>, <ismael.figueroa@pucv.cl>
    December 30, 2023
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================




  Usability is a key discipline related to the development of modern
  software systems. Its goal is to assess the user-friendliness and
  effectiveness of a software product from the user point of
  view. Therefore, proper methodologies and techniques to perform this
  assessment are definitely relevant. Heuristic evaluation is probably
  the most commonly used method for usability assessment. Initially
  developed by Nielsen and Molich in the '90s, traditional heuristic
  evaluations rely on Nielsen's well-known 10 usability
  heuristics. However, recent evidence suggests that such heuristics
  are not sufficiently complete for dealing with new domains such as
  interactive television, virtual worlds, and many others. In addition
  to the lack of suitability of traditional heuristics, in the past
  years the lack of a robust methodology or process to effectively
  develop and validate these new domain-specific heuristics has been
  documented. In this paper we summarize current evidence regarding
  the lack of suitability of traditional heuristics, as well as the
  need to develop new domain-specific heuristics. After identifying
  and acknowledging existing gaps in heuristics for state-of-the-art
  technology, as documented by other researchers, we present
  , a PROcedural METhodology for developing HEuristics of
  USability. refines the methodology of Rusu   
  (2011), and is composed of 8 stages. clearly defines the
  artifacts that are required and produced by each stage, and also
  presents a set of quality indicators in order to assess the need for
  further refinement in the development of new heuristics. As an
  initial validation of , we apply a questionnaire to
  several researchers that have used the methodology of Rusu   ,
  and we have also performed a small retrospective study, computing
  the quality indicators of several previous studies. Our results
  suggest that is a very promising methodology, and that
  the metrics and indicators are indeed pertinent with respect to the
  conclusions of previous studies.



  usability, heuristic evaluation, usability heuristics, procedural
  methodology, human-computer interaction, empirical studies.




§ INTRODUCTION



Constant technological development is reflected in the
development of new applications, many of them based on new
  information technologies or emerging technologies, such as
mobile devices, touch interfaces, distributed applications, virtual
learning environments, to name just a few. These applications
characteristically belong to a specific domain, and are
accessed using different physical and logical devices, as well
as specific contexts of use. Since it is increasingly common
that non-specialist users need to access and use such applications on
a frequent basis, it is necessary and important to facilitate such
users’ experience to improve their satisfaction levels and task
performance.

Usability is a discipline which, in the field of Software
Engineering, allows us to estimate the degree to which a
  software product can be used by specific users to achieve specific
  goals effectively, efficiently and satisfactorily in a specific
  context of use <cit.>.

Therefore, one of its main activities is determining usability
  problems in specific applications, in order to improve the quality
of this attribute in future iterations. Given the high costs of
usability tests <cit.>, where the behavior of
real users is observed and evaluated, two lower-cost techniques are
generally used to identify usability problems: the inquiry and
the usability inspection.

Usability inquiries characteristically use a qualitative approach
based on user opinions, and include methods such as questionnaires,
field studies, and different kinds of interview.

On the other hand, usability inspections consist of detailed
examinations carried out by groups of evaluators, who report on
specific and general problems regarding a particular application.

One of the most widely used inspection methods is heuristic
  evaluations, proposed by Nielsen and
Molich <cit.>. In this method, a small group of 3-5
evaluators inspect a particular software system's usability, guided by
a list of usability criteria known as usability
  heuristics <cit.>. Heuristic
evaluation is a low-cost easily applicable method, which can be used
at various stages of an application’s
development <cit.>. It is also an effective method,
as it can detect around 75% of usability problems with groups of 3 to
5 evaluators <cit.>. However, while heuristic
evaluations are traditionally carried out using Nielsen’s 10 usability
heuristics as criteria <cit.>, the need to develop
domain-specific heuristics, or put simply, domain heuristics,
is ever more pressing.

In a recent study, Hermawati and Lawson <cit.>
performed a bibliographic review of 70 articles related to the
development of domain heuristics, identifying two large problems:
significant deficiencies in efforts to validate new heuristics, and
lack of rigor, robustness and standardization in the effectiveness
analysis of domain heuristics. As a solution to these problems, in
this article we present , a PROcedural METhodology for
developing HEuristics of USability, which refines the existing
methodology of Rusu  <cit.>–—which we call
, based on its authors’ initials—for greater formality, rigor
and precision in the process of constructing new domain
heuristics. has 8 stages, in each of which the activities
to be performed, the expected and produced artifacts, and the quality
indicators for heuristics — which in turn guide the 8 stages’
continuous refinement process — are described in detail. In this
article, we begin by presenting an overview of how domain heuristics
have been usually generated (<Ref>) and the methodology (<Ref>), to then present the main
contribution, which is (<Ref>). Finally,
we conclude with the initial validation of (<Ref>).



§ HOW DOMAIN HEURISTICS ARE DEVELOPED



There is vast literature on heuristic evaluations and the development
of usability heuristics. In fact, as is studied by Hermawati and
Lawson <cit.>, there are at least 70 relevant
studies regarding the development of domain heuristics, so a detailed
review is beyond the scope of this article. Below we present a summary
of the main conclusions of <cit.>, which serve as
the foundation for the artifacts and stages proposed in .



 §.§ Collecting and transforming information



The first point to emphasize is that most of the studies analyzed in
<cit.> include two large stages: collecting
information on heuristics, and transforming that information into
domain heuristics. Hermawati and Lawson describe 4 strategies for
collecting information:



  * Adopt existing theories.

  * Analyze context of use.

  * Analyze existing case study reports.

  * Analyze and/or create a common set of usability problems.


Then, the transformation process uses one of the following 3
strategies:




  * Make a list of all the information, eliminating redundancies and
  overlaps, and using the end result as the new domain heuristics.


  * Perform the same normalization process, and sort the resulting
  information into categories, which are then transformed into
  heuristics.


  * Extend and/or modify Nielsen's heuristics.
  




 §.§ Validating the heuristics



An important result from <cit.> is that they detected deficient
validation of domain heuristics. In short, 34% of the analyzed
studies reported no validation. Of the remaining 66%, the most
commonly used methods to validate domain heuristics are:



  * Experts applying the domain heuristics (24 studies).

  * Comparing the results with those from other heuristics (20
  studies).

  * Comparing the results, based on tests with real users (5
  studies).


However, there are few studies that attempt to determine the
effectiveness of the new heuristics. As reported
in <cit.>, only 19 of the 70 cases studied
present a comparison in terms of effectiveness  the other cases do
not present validation, do not use a control group of heuristics, nor
do they perform any comparison or quantitative analysis. This
notwithstanding, in the cases where the new domain heuristics versus a
set of control heuristics are studied, the following categories of
validation techniques were identified:



  * Quantify all usability problems, per heuristic, and compare with
  an evaluation using control heuristics.


  * Quantify and compare the frequency, severity and distribution of
  those problems via each specific heuristic and via each set of
  heuristics.


  * Identify and quantify problems detected in both sets of
  heuristics, and unique problems detected separately by the
  heuristics sets.



However, there is great variability in the extent and rigor of the
quantitative analyses. On the other hand, only 3 studies used the
predefined metrics proposed by
Hartson  <cit.>. To conclude this summary,
the authors of <cit.> identified 3 areas to
improve the rigor and robustness of validating domain heuristics:




  * Adopt robust and rigorous validation metrics, to determine the
  effectiveness of new domain heuristics.

  * Create new domain heuristics based on existant heuristics in the
  domain.

  * Improve the definition and categorization of expert, so
  as to control the variability introduced by evaluators.





§ METHODOLOGY



In this section, and with the objective of contextualizing the
contribution of , we describe the methodology. This
is a methodology consisting of 6 stages, which we quote
textually <cit.>:





  * Exploratory: to collect bibliography related with the
  main topics of the research: specific applications, their
  characteristics, general and/or related (if there are some)
  usability heuristics.


  * Descriptive: to highlight the most important
  characteristics of the previously collected information, in order to
  formalize the main concepts associated with the research.


  * Correlational: to identify the characteristics that the
  usability heuristics for specific applications should have, based on
  traditional heuristics and case studies analysis.


  * Explicative: to formally specify the set of the
  proposed heuristics, using a standard template.


  * Validation (experimental): to check new heuristics
  against traditional heuristics by experiments, through heuristics
  evaluations performed on selected case studies, complemented by user
  tests.


  * Refinement: based on the feedback from the validation
  stage.
  


Evaluation criterion identifies and defines three
types of problem that can be detected in the experimental stage,
consisting of groups of evaluators who use the new heuristics, and
groups of evaluators using the control heuristics. Note that we use a
new notation to refer to these types of problem.



  * Common problems, : Problems identified
  by both groups of evaluators.

  * Domain problems, : Problems
  identified only by the group of evaluators who used the new
  heuristics.

  * Control problems, : Problems
  identified only by the group of evaluators who used Nielsen’s
  heuristics (or others if appropriate) as control.


With this sorting, establishes that the new heuristics work well
when and/or include the
highest percentage of problems. In practice, the severity of specific
and general problems is also considered. Finally, note that is a
simple methodology, which has been applied successfully and offers a
framework that encourages experimental validation, thus avoiding some
of the problems described in <Ref>. It is precisely
because of this that we take as starting point for the
development of .



§ 



is a , which has emerged as a refinement
of . We say it is a procedural methodology as it
accurately describes the steps to follow—and the artifacts to
construct—during the elaboration process of new domain
heuristics. originates from a previous investigation by
Jiménez   <cit.> to determine the usability of
. Therefore, they conducted a questionnaire with the researchers
who developed heuristics using , in the domains of grid
computing <cit.>, interactive
television <cit.>, virtual
worlds <cit.> and touchscreen
devices <cit.>. On the quantitative side, the aggregated
results are inconclusive, because the ease of use was evaluated as
“neutral”. However, there were specific difficulties in the
explanatory, experimental and refinement stages. This was confirmed by
the comments made by those surveyed, which indicate lack of clarity in
these stages, and especially in refinement.

Next, we present , starting with its contextualization
regarding , and then describing the critical path for the
methodology. A full-fledged description of is presented
in <Ref>.





 §.§ Refinements to 



<Ref> presents a diagram with the stages and the stages. At a first glance, the main
difference is that breaks up the exploratory and
correlational stages, each into two new stages. In addition, the
descriptive and explanatory stages have been renamed, and the
validation and refinement stages have been refined. The specific
contributions of to the stages are:



  * Exploratory: specifies the stages
  search for specific information and search for
    heuristics of usability. The first considers 4
  characteristic dimensions to describe the specific domain:
  context of use, logical devices, physical devices, and user
  profiles. The second stage provides guides for a systematic
  literature review that produces a set of heuristics applicable to
  the domain.


  * Descriptive: specifies an encoding table of
  specificity indices, based on the domain’s characteristic
  dimensions.


  * Correlational: proposes two stages:
  normalization of heuristics and prioritization of
    heuristics. In the first stage, cases of duplication or overlap
  found in the heuristics are resolved. The second case creates a
  specificity ranking for each heuristic, also considering the
  characteristic dimensions, among other factors.


  * Explanatory: the explanatory stage is kept from ,
  although recent evidence suggests it is possible to improve this
  stage <cit.>,  considering novice evaluators.


  * Validation: we have kept the idea from to perform
  experimental validation. However, requires at least one
  heuristic evaluation using the domain heuristics and a group of
  control heuristics. In addition, specifies quality
  indicators for the heuristics, based on this experiment.


  * Refinement: in , the quality indicators
  obtained during the validation stage are used to suggest in which
  stages changes must be made or which specific problems need to be
  solved.







 §.§ Overview of the critical path



<Ref> illustrates the 8 stages of to develop domain heuristics. The figure shows the required artifacts
as input, and the produced artifacts as output for each stage. It also
shows potential intermediate outputs, when suitable heuristics are
found, thus making the development of new heuristics
unnecessary. However, the methodology’s critical path consists
of all the steps taken by a researcher for the effective development
of new usability heuristics for a specific domain. That is to say,
despite considering situations of intermediate outputs, is
focused on situations where it is necessary to create new domain
heuristics.

For starters, in the first iterative application of the methodology,
all stages should be applied sequentially. Then, based on feedback
from the refinement stage, greater adaptability and flexibility are
possible in the stages performed during the progressive improvement of
the new heuristics. Obviously, the number of iterations and the stages
that have to be repeated will depend on the researcher's needs, the
results of the validation, and other factors specific to each
project. However, the indicators generated in the validation stage
allow us to make an informed decision whether or not to continue with
the refinement. The critical path is summed up in four major
phases:



  §.§.§ Systematic search


The process starts by selecting the domain for which a set of specific
usability heuristics is required. Stages 1 to 5 consist in a
systematic search process, specificity encoding, and prioritizing one
or more sets of existing heuristics, based on the characteristic
domain dimensions, using a standard template.



  §.§.§ Defining the domain heuristics


At the end of stage 5, the researcher will have a set of heuristics
applicable to the domain, where there should be no duplication or
overlapping problems. Crucially, at this stage the researcher
  can create new heuristics based on existing data. Then, stage 6
describes the heuristics using a standard template, like in . The
detailed description aims to facilitate the evaluators’ task of
performing the experimental validation. The result of stage 5 and/or 6
is the new domain heuristics. 



  §.§.§ Experimental validation


Subsequently, stage 7 is to experimentally validate the domain
heuristics. Unlike , and in line with the suggestions of
Hermawati and Lawson <cit.>, requires at least one heuristic evaluation in a domain-specific
case study.

This evaluation must consider the domain heuristics, and a set
of control heuristics, which by default can be Nielsen's 10
heuristics, so as to generate the indicators that are detailed
below. It would also be ideal to have several groups of evaluators for
each set of heuristics, to minimize the differences between
groups. Another difference from is that the evaluators that use
the control heuristic must assign a specificity score to each
problem found concerning the domain. This score uses the same scale
used by the researcher in stage 3.

The validation phase culminates with the calculation of the following
indicators:




  * Rate of unique problems,
  =  /:
  This represents which group of heuristics found more unique
  problems. If > 1, more problems were found in the
  domain heuristics.


  * Rate of problem dispersion: To measure the distribution
  of problems in the groups of heuristics, we consider the values
  and , which represent
  the standard deviation of problem distribution for the control
  heuristics and domain heuristics respectively. Given the above, we
  define the rate of dispersion
  =  /. If > 1, problems are
  better distributed in the domain heuristics than in the control.


  * Rate of severity: Similar to the rate of dispersion, we
  define the rate of severity
  =  /,
  which represents the relationship between the average severity
  of the problems encountered with the domain
  heuristics, versus the average severity of the
  problems encountered with the control heuristics. If
  > 1 it means that on average, the domain heuristics
  found more severe problems than the control heuristics.


  * Rate of specificity: Finally, we define the rate of
  specificity
  =  /, which relates specificity averages
  and , of the
  problems encountered with the domain and control heuristics
  respectively. If > 1, then the problems found
  by the domain heuristics are, on average, more specific that those
  found by the control heuristics.



It is important to remark that the indicators are uniform  when its
value is greater than 1, it is considered positive, and we consider
that there are potential efficiency problems when its value is less
than 1. This is useful, for example, to construct visualizations
like <Ref>, which compares the indicator values from
various existing studies.



  §.§.§ Refinement



As shown in <Ref>, the refinement stage takes
the heuristic quality indicators , ,
and into consideration, so as to
propose possible refinements to the domain heuristics. Such
refinements involve returning to work in one of the specific stages of
, and then continuing with the critical path as needed. The
need for refinement was born, at least, from the following problematic
scenarios, or any combination of them.




  * < 1, that is, the domain heuristics found fewer
  unique problems. This may indicate that the heuristics are not well
  adjusted to the domain, that the application examined has mainly
  general problems, or else that the detailed explanations for the
  domain heuristics were not understood well by the evaluators.


  * < 1, that is, problems are worse distributed
  in the domain heuristics than in the control. This can be a symptom
  of overlapping heuristics, too many problems, or even overly
  specific heuristics, lack of problems, or that they are difficult to
  implement for evaluators.


  * < 1, that is, the problems found by the
  control heuristics are considered more specific to the domain than
  those found by the domain heuristics. This problem can indicate
  problems with the prioritization of domain heuristics, or problems
  in the experiment design, for example selecting groups of
  evaluators.


  * < 1, that is, the problems found by the control
  heuristics are more severe than those found by the domain
  heuristics. In general, this may indicate problems in the experiment
  design, in particular with the selection of the groups of
  evaluators.



The end decision determining when the domain heuristics are considered
validated depends ultimately on the researchers, the domain,
and the specific applications being studied. However, provides a precise methodological process with objective quality
indicators, which promote continuous feedback and provide quantitative
support to make such a decision.



§ INITIAL VALIDATION



For the initial validation of we have used a qualitative
approach via a short questionnaire, taken by researchers who used
in their work, and a quantitative approach by calculating
quality indicators (described in <Ref>) from the
existing results mentioned in <Ref>, as far as the
data allowed.



 §.§ Questionnaire for investigators that used 



We gave a questionnaire (see <Ref>) to 5
investigators that used in cases studies related to domains of:
U-Learning, applications for tablets, transactional web
applications, and smartphone applications. At the time the
questionnaire was taken, four of the interviewees were senior students
in Computer Engineering at the Pontificia Universidad Católica de
Valparaíso, and one, Inostroza, the author of <cit.>,
was a doctoral student in the final stages of his thesis. Two
interviewees worked on tablet domains, but are considered as experts
individually, for the purposes of this validation. The objective of
the questionnaire was to have expert evaluation of the following
criteria:




  * Objective of the study

  * Applicable stages

  * Stages that appear easiest to apply

  * Stages that appear hardest to apply

  * Recommendations for adding new stages

  * Recommendations for eliminating stages

  * General comprehension of 

  * Specific comprehension of each stage

  * Pertinence of quality indicators

  * Ease of calculation of quality indicators

  * Other recommendations



We gave each participant a document with a preliminary description of
, so they could evaluate the methodology’s applicability to
their case studies. We also carried out individual sessions to clarify
any doubts. After these sessions, the participants had three months to
evaluate the applicability of .

Results <Ref> summarizes the main
results from the questionnaires taken. The following information is
noteworthy:

=26pt

 






  * 3 of the 5 participants had the objective of constructing new
  heuristics, while the other 2 were looking to refine existing
  heuristics. This has an effect on the potential number of applicable
  stages, as the investigators from the first group considered more
  stages of applicable to their projects. In the case of
  U-Learning, 6 of 8 stages were considered applicable. In contrast,
  for transactional web and smartphone applications, only stage 6 was
  considered applicable.
 

  * In general, participants considered that the applicable stages
  would also be easy to apply. The only explicit case of a stage found
  hard to apply is stage 4 (heuristic normalization), in the
  U-Learning domain.


  * All participants indicated they did not consider it necessary
  to add or eliminate stages within the methodology, that the general
  application of the methodology was clear and that the methodology
  was useful. Regarding comprehension of each particular stage, there
  were only issues with understanding stage 6 in the case of
  transac-tional web applications.


  * Unfortunately, the quality indicators were only consid-ered
  applicable, pertinent and easy to calculate in the case of
  U-Learning. In general, the other investigators were not interested
  in calculating them.


  * Among the recommendations given, the following stand out:

  
  
  * Include an example of how to fill the template in for the
    “detailed heuristic description” stage.

  
  * Better explain the calculations included in the methodology.

  
  * Include a real application example.
  





 §.§ Validation of quality indicators



Preliminaries Before showing the obtained results, let us
recall the classification of problems proposed in (<Ref>):



  * Common problems, : Common problems
  identified by both groups of evaluators.

  * Domain problems, : Problems only
  identified by the group of evaluators that used the new heuristics.

  * Control problems, : Problems only
  identified by the group of evaluators that used Nielsen’s heuristics
  (or oth-ers if applicable) as control.


Now, let be the total set of problems found when
applying both the control heuristics and the domain heuristics; we
have that:


    = ∪ – (domain heuristics problems)
       = ∪ – (control heuristics problems)
       = ∪∪ = ∪ – (total problems)


Similarly, in addition we consider the average severity
of the problems in and the average
severity of the problems in
. Considering all of the aforementioned, we define
the indicators
=  / and
=  / that
approximate the uniqueness and severity rates, respectively, in cases
where the common problems are not separated between
the two groups of heuristics.

Considering the previous definitions, the second point of the initial
validation consists in analyzing the data from the 19
studies identified by <cit.> that attempt to
determine the effectiveness of the new heuristics
(<Ref>), and considering two additional
recently-published
studies <cit.>. We calculate the
heuristic quality indicators for each of them as possible according to
the published information, and we underscore the conclusions regarding
whether or not the developed heuristics are more effective than the
control heuristics. <Ref> summarizes the methodology,
the criteria used by the authors, the indicators that could be
calculated, and whether or not the authors conclude that the domain
heuristics developed are more effective than the control heuristics
used. Here we synthesize the main results obtained. 

Regarding the number of problems

Of the 21 studies analyzed, we were able to calculate an indicator for
17 of them. In these 17 cases, the most commonly used indicator
corresponds to the uniqueness rate , used in 9 cases, or
the approximate uniqueness rate , used in 4
cases. Two other cases only considered the number of problems based on
the domain heuristics; in other words, only was
considered. In the remaining two cases, no indicator based on the
number of detected problems could be calculated.

Regarding the dispersion of problems

In a very distant second place, calculating the dispersion rate
was possible in four cases, while two cases only had
information regarding the problem dispersion for the domain
heuristics; in other words, σ().

Regarding the severity and specificity of the problems

Similarly, it was possible to calculate the severity rate
in four cases. In 2 cases it was possible to calculate
an approximation , and in another 2 cases only the
problem severity of the domain heuristics,   , was
considered.  Moreover, it was not possible to calculate the problem
specificity rate .

Domain heuristics vs control heuristics

In 20 of the 21 cases, the authors conclude that the domain heuristics
developed (or refined) are more efficient than then control
heuristics. The case wherein this does not occur is in the domain of
security management applications <cit.>; in this case
only = 1.03 could be calculated, which is difficult to
interpret without additional information.

Relation between indicators and positive conclusions

<Ref> shows the indicators calculated
in <Ref>, breaking each sub-case down as an
independent input in the graph. Each vertical line represents a case
study involving more than one indicator. A fundamental observation is
that the condition > 1, or its approximation
> 1, seems to be the best predictor to validate
the domain heuristics’ efficiency. This is not so in only two cases,
for = 0.93 and = 0.73. Under the
scheme, these cases are difficult to interpret and would
require a refinement, as well as calculating the other specified
indicators. Furthermore, other qualitative factors may contribute to
positive evaluation in these case studies. Regarding the distribution
and severity rates, we observed values close to 1, and which
complement the uniqueness rates. In just 3 cases, only severity was
considered, due to a lack of other quantitative data.



To conclude this section, we observe that the retrospective analysis
of the selected cases in which domain heuristics have been developed
supports the pertinence of the indicators proposed in
. Particularly, it is clear that the most important
indicator is the uniqueness rate, and that the other indicators are
more complementary and can help discriminate in less conclusive
cases. We also observed existent studies that base their positive
conclusions on a rather weak quantitative argument, which suggests
that the refinement recommendations in are on the right
track.



§ CONCLUSIONS AND FUTURE WORK



is a proposed to increase the robustness
of investigations that aim to develop new domain-specific usability
heuristics. Based on Hermawati y Lawson’s
study <cit.>, and using the methodology as
a base <cit.>, we propose a process consisting in
7 principal stages, and a continuous refinement cycle. The main
contributions of ’ are: (i) it specifies each artifact
consumed or generated in detail for each stage, (ii) it defines
quantitative quality indicators, and (iii) it defines the refinement
based on feedback provided by the indicators. Based on a questionnaire
completed by expert investigators, and a retrospective analysis on
relevant, state-of-the-art case studies, we have performed an initial
validation on which is, in our opinion, very positive and
suggests new and immediate lines of future work for the additional and
complete validation of .

In essence, it is necessary to apply using real case
studies that aim to develop new domain heuristics, and follow the
critical path, so as to obtain specific feedback in each of the
proposed stages. To this end, we propose applying to two
new case studies:



  * Virtual Learning Environments, with the case study in
  question to be found at http://elearning.espoch.edu.ec.

  * E-Government Pages. Also based on public ser-vices available in
  Ecuador.


Both of these cases are expected to have at least two groups of
evaluators in the experimental validation stage. In addition,
semi-structured interviews and questionnaires will be conducted in
order to obtain the opinion of everyone involved in these case
studies.


in-for-ma-ción smart-pho-nes geo-grá-fi-ca pre-sen-ta-ción



@|p2.2cm|p0.5cm|p3.5cm|p3cm|p3.8cm|p2cm|@

Domain Heuristics vs Control Heuristics
  - Efficiency Analysis Summary  


    1|cDomain     
    1c# Hs    
    1cCriteria    
    1cMethodology    
    1cIndicators    
    1c|better than ? 

    

    1|cDomain    
    1c# Hs    
    1cCriteria     
    1cMethodology    
    1cIndicators    
    1c|better than ? 

    

    6|r|Continues in the next page... 
 

    

    Grid Computing <cit.>
        12
        Quantity of unique, specific and general problems. Severity of
      specific vs general problems
            Case 1:
      = 2.0,  = 0.97,  = 1.16
      Case 2: = 1.83,  =
      1.11 
    
        


    Virtual worlds <cit.>
        16
        Quantity of unique, specific and general problems. Severity of
      specific vs general problems         Case 1:
    = 1.93,  = 0.71, 
    =0.91
    Case 2:
    = 2.75,  = 0.78, 
    =1.07     


  Inter-cultural web sites <cit.>
        13
        Quantity of unique, specific and general problems. Severity of specific
      vs general problems        Case 1:
      = 1.97,  = 1.11 Case 2: = 1.71,  =
      1.25(based on percentages)     


    Digital Television <cit.>     14    
    Quantity of unique, specific and general problems. Severity of specific
      vs general problems        
    = 5.0,  = 1.28     


    Notification systems <cit.>     8     Quantity of
                                                       problems found
                                                       on 3 interfaces
                                                       used as case
                                                       study. Level of
                                                       compliance of
                                                       the heuristics
                                                       developed
        Based on usability problems related to notification systems
       
    Case 1: = 0.93
    Case 2: = 1.64
    Case 3: = 1.23
        


    Asistential Robotics <cit.>     9    
    Quantity of total and unique problems     Adaptation of existing
                                            heuristics     = 4.33     


    Security management applications <cit.>    
    7     Quantity and severity of problems, effectiveness,
        meticulousness, validity and reliability of the heuristics    
                                                                     It
                                                                     uses
                                                                     grounded
      theory analysis to generate the heuristics    
    = 1.03     


    Educational web sites <cit.>     n/a    
    Quantity and severity of problems, effectiveness, meticulousness,
    validity and reliability of the heuristics, time used and associated cost
        Own 4 steps methodology that analyzes the characteristics of the
      domain and evaluates cases through experts and users     = 9.17     


    Mobile-based applications <cit.>     11    
    Quantity and severity of problems     Extends Nielsen's heuristics     = 1.9     


  Mobile computing <cit.>     8    
   Quantity and severity of problems, quantity of problems per
   heuristic, time used by evaluators     Own 3 steps methodology based
   on the analysis of 3 experts    
    Case 1: = 26,
     = 22= 1.18
    Case 2: = 38,
     = 28= 1.36
    Combined cases: = 0.57
        


    Learning based in clinic cases <cit.>
                                    22     Quantity and severity of
                                       problems in specific vs general
                                       heuristics     It uses
    grounded theory analysis to generate the heuristics    
    
    Case 1: = 27,  =
    37= 0.73,  < Case 2: = 74,  =
    56= 1.32,  >
    
        


    Online games <cit.>     10    
    Quantity and severity of problems, quantity and severity of
                                                  problems per
                                                  heuristic
        Based on the usability analysis on games    
    = 67, =49,  = 1.37
    = 2.30,  = 2.39,  = 0.96
    σ() = 3.16
        


    Mobile map applications <cit.>     10    
    Quantity and severity of problems     Adaptation of Nielsen's
                                        heuristics using a
                                        theoretical-conceptual focus    
    = 19,  = 15,
     = 1.27
        


    Smartphones <cit.>     12     Quantity of
                                                     unique, specific
                                                     and general
                                                     problems. Severity
                                                     of specific vs
                                                     general problems         = 28.33, =2.36σ() = 2.24    


    Groupware <cit.>NoControl     24    
    Problems found in case study and their severity     Based on design patterns    
    = 39, σ() = 1.85 = 2.15
         Without , positive conclusions 


    Mobile touch devices <cit.>     12
        Quantity of unique, specific and general problems. Severity of
      specific vs general problems         Case 1:
    = 1.37 Case 2:
    = 0.83     


    ERP Applications <cit.>     5     Mean, median and
                                                   mode of problem
                                                   severity per
                                                   heuristic     Based
                                                               on 5
                                                               evaluation
                                                               criteria
                                                               for ERP
                                        = 1.76;
    = 0.88,  = 2     


  Ambient display <cit.>     12     Percentage and
                                                   severity of
                                                   problems identified
                                                   by both groups of heuristics
                                                   Porcentaje y
       Adaptation of Nielsen's heuristics taking into account
     characteristics of the domain and the opinion of experts and
     designers
       The presented results do not permit us to compute the indicators     


    Computer games <cit.>[ No
  comparison against control heuristics]     18
       Kinds of problems, quantity and severity     Based on the
                                                literature and the
                                                review of experts in
                                                gameability and game
                                                designers      The
                                                             presented
                                                             results
                                                             do not
                                                             permit us
                                                             to
                                                             compute
                                                             the
                                                             indicators
                                      
    


    Information display in big screens <cit.>     8     Quantity of problems, quantity
                                       of real problems,
                                       meticulousness, validty,
                                       effectiveness and reliability
                                       of the heuristics
       It uses critical parameters using scenario-based designs    
                                                                The
                                                                presented
                                                                results
                                                                do not
                                                                permit
                                                                us to
                                                                compute
                                                                the indicators     


    Web Sites <cit.>     13     Quantity of problems,
                                             time used by evaluators
        Adaptation of Nielsen's heuristics
        The presented results do not permit us to compute the indicators     


    







IEEEtran



10

url@samestyle








jimenezAl:ieeelatam2017
C. Jimenez, H. A. Cid, and I. Figueroa, “Prometheus: Procedural methodology
  for developing heuristics of usability,” IEEE Latin America
  Transactions, vol. 15, no. 3, pp. 541–549, March 2017.

iso19989241
W. ISO, “9241-11. ergonomic requirements for office work with visual display
  terminals (vdts),” The international organization for
  standardization, vol. 45, 1998.

UsabilityMethods
A. Holzinger, “Usability engineering methods for software developers,”
  Communications of the ACM, vol. 48, no. 1, pp. 71–74, 2005.

NielsenMolich1990
J. Nielsen and R. Molich, “Heuristic evaluation of user interfaces,” in
  Proceedings of the SIGCHI Conference on Human Factors in Computing
  Systems, ser. CHI '90.1em plus 0.5em minus 0.4emNew York, NY,
  USA: ACM, 1990, pp. 249–256. [Online]. Available:
  <http://doi.acm.org/10.1145/97243.97281>


luna2015using
H. Luna, R. Mendoza, M. Vargas, J. Munoz, F. J. Alvarez, and L. C. Rodriguez,
  “Using design patterns as usability heuristics for mobile groupware
  systems,” IEEE Latin America Transactions, vol. 13, no. 12, pp.
  4004–4010, 2015.

Nielsen1993usability
N. Jakob, “Usability engineering,” Fremont, California: Morgan, 1993.

JackobNielsen1995
Jackob Nielsen, “10 Heuristics for User Interface Design: Article by Jakob
  Nielsen,” 1995. [Online]. Available:
  <https://www.nngroup.com/articles/ten-usability-heuristics/>


hermawatiLawson:2016
S. Hermawati and G. Lawson, “Establishing usability heuristics for heuristics
  evaluation in a specific domain: Is there a consensus?” Applied
  Ergonomics, vol. 56, pp. 34 – 51, 2016. [Online]. Available:
  <http://www.sciencedirect.com/science/article/pii/S0003687015301162>


rusu2011methodology
C. Rusu, S. Roncagliolo, V. Rusu, and C. Collazos, “A methodology to establish
  usability heuristics,” in Proc. 4th International Conferences on
  Advances in Computer-Human Interactions (ACHI 2011), IARIA, 2011, pp.
  59–62.

hartsonAl:ijhci2003
H. R. Hartson, T. S. Andre, and R. C. Williges, “Criteria for evaluating
  usability evaluation methods,” International Journal of Human-Computer
  Interaction, vol. 15, no. 1, pp. 145–181, 2003.

Jimenez2012SCCC
C. Jimenez, C. Rusu, S. Roncagliolo, R. Inostroza, and V. Rusu, “Evaluating a
  methodology to establish usability heuristics,” in Chilean Computer
  Science Society (SCCC), 2012 31st International Conference of the, Nov 2012,
  pp. 51–59.

rusu2011usabilityGrid
C. Rusu, S. Roncagliolo, G. Tapia, D. Hayvar, V. Rusu, and D. Gorgan,
  “Usability heuristics for grid computing applications,” in
  Proceedings of The 4th International Conferences on Advances in
  Computer-Human Interactions ACHI, 2011.

solano2011usability
A. Solano, C. Rusu, C. Collazos, S. Roncagliolo, J. L. Arciniegas, and V. Rusu,
  “Usability heuristics for interactive digital television,” in The
  Third International Conference on Advances in Future Internet, 2011, pp.
  60–63.

Munoz2012d
R. Munoz and V. Chalegre, “Defining Virtual Worlds Usability Heuristics,”
  2012 Ninth International Conference on Information Technology - New
  Generations, pp. 690–695, apr 2012. [Online]. Available:
  <http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6209172>


Inostroza2012
R. Inostroza, C. Rusu, S. Roncagliolo, C. Jimenez, and V. Rusu, “Usability
  heuristics for touchscreen-based mobile devices,” in Information
  Technology: New Generations (ITNG), 2012 Ninth International Conference on,
  April 2012, pp. 662–667.

Jimenez_EAST2012
C. Jimenez, C. Rusu, V. Rusu, S. Roncagliolo, and R. Inostroza, “Formal
  specification of usability heuristics: How convenient it is?” in
  Proceedings of the 2Nd International Workshop on Evidential Assessment
  of Software Technologies, ser. EAST '12.1em plus 0.5em minus
  0.4emNew York, NY, USA: ACM, 2012, pp. 55–60. [Online]. Available:
  <http://doi.acm.org/10.1145/2372233.2372249>


Inostroza201640
R. Inostroza, C. Rusu, S. Roncagliolo, V. Rusu, and C. A. Collazos,
  “Developing smash: A set of smartphone's usability heuristics,”
  Computer Standards and Interfaces, vol. 43, pp. 40 – 52, 2016.
  [Online]. Available:
  <http://www.sciencedirect.com/science/article/pii/S0920548915000926>


carrare2015usability
A. Carrare, C. Hernandez, C. Kochi, I. Silveira, and C. Longui, “Usability
  heuristics for clinical case-based learning assessment systems applied to
  medical education,” IEEE LATIN AMERICA TRANSACTIONS, vol. 13, no. 3,
  pp. 892–898, 2015.

Jaferian2011
P. Jaferian, K. Hawkey, A. Sotirakopoulos, M. Velez-Rojas, and K. Beznosov,
  “Heuristics for evaluating IT security management tools,” SOUPS
  '11: Proceedings of the Seventh Symposium on Usable Privacy and Security,
  pp. 1–20, 2011.

diaz2013cultural
J. Díaz, C. Rusu, J. A. Pow-Sang, and S. Roncagliolo, “A
  cultural-oriented usability heuristics proposal,” in Proceedings of
  the 2013 Chilean Conference on Human-Computer Interaction.1em plus
  0.5em minus 0.4emACM, 2013, pp. 82–87.

solano2013evaluating
A. Solano, C. Rusu, C. A. Collazos, and J. Arciniegas, “Evaluating interactive
  digital television applications through usability heuristics/evaluando
  aplicaciones de televisión digital interactiva a través de heurísticas de
  usabilidad,” Ingeniare: Revista Chilena de Ingenieria, vol. 21,
  no. 1, p. 16, 2013.

Berry2003
B. Berry, “Adapting heuristics for notification systems,” Proceedings
  of the 41st Annual ACM Southeast Conference (ACMSE '03), pp. 144–149, 2003.

tsui2010developing
K. M. Tsui, K. Abu-Zahra, R. Casipe, J. M. Sadoques, and J. L. Drury,
  “Developing heuristics for assistive robotics,” in Human-Robot
  Interaction (HRI), 2010 5th ACM/IEEE International Conference on.1em
  plus 0.5em minus 0.4emIEEE, 2010, pp. 193–194.

Alroobaea2013
R. Alroobaea, A. H. Al-badi, and P. J. Mayhew, “A Framework for Generating
  Domain-Specific Heuristics for Evaluating Online Educational Websites,”
  International Journal of Information Technology & Computer Science,
  vol. 8, no. 2, pp. 75–84, 2013.

Neto2013
O. M. Neto and M. D. G. Pimentel, “Heuristics for the Assessment of
  Interfaces of Mobile Devices,” Proceedings of the 19th Brazilian
  symposium on Multimedia and the web - WebMedia '13, pp. 93–96, 2013.
  [Online]. Available: <http://dx.doi.org/10.1145/2526188.2526237>


Bertini2006
E. Bertini, S. Gabrielli, and S. Kimani, “Appropriating and assessing
  heuristics for mobile computing,” AVI '06: Proceedings of the working
  conference on Advanced visual interfaces, no. April 2016, pp. 119–126,
  2006. [Online]. Available:
  <http://portal.acm.org/citation.cfm?id=1133265.1133291>


Pinelle2009
D. Pinelle, N. Wong, T. Stach, and C. Gutwin, “Usability heuristics for
  networked multiplayer games,” Proceedings of the ACM 2009
  international conference on Supporting group work, pp. 169–178, 2009.

Kuparinen2013
L. Kuparinen, J. Silvennoinen, and H. Isomäki, “Introducing Usability
  Heuristics for Mobile Map Applications,” Proceedings of the 26th
  International Cartographic Conference (ICC 2013)., 2013.

Inostroza2012_JCC
R. Inostroza, C. Rusu, S. Roncagliolo, C. JimÃ©nez, and V. Rusu, “Usability
  heuristics validation through empirical evidences: A touchscreen-based mobile
  devices proposal,” in Chilean Computer Science Society (SCCC), 2012
  31st International Conference of the, Nov 2012, pp. 60–68.

Singh2009
A. Singh and J. Wesson, “Evaluation criteria for assessing the usability of
  ERP systems,” Proceedings of the 2009 Annual Research Conference of
  the South African Institute of Computer Scientists and Information
  Technologists, vol. 125, no. October, pp. 87–95, 2009. [Online]. Available:
  <http://portal.acm.org/citation.cfm?id=1632149.1632162&coll=ACM&dl=ACM&CFID=65944897&CFTOKEN=53873172>


Mankoff2003
J. Mankoff, A. K. a.K. Dey, G. Hsieh, J. Kientz, S. Lederer, and M. Ames,
  “Heuristic Evaluation of Ambient Displays,” Proceedings of the
  SIGCHI conference on Human factors in computing systems, no. 5, pp.
  169–176, 2003. [Online]. Available:
  <http://portal.acm.org/citation.cfm?id=642611.642642>


desurvire2009game
H. Desurvire and C. Wiberg, “Game usability heuristics (play) for evaluating
  and designing better games: the next iteration,” in Online communities
  and social computing.1em plus 0.5em minus 0.4emSpringer, 2009,
  pp. 557–566.

Somervell2005
J. Somervell and D. S. McCrickard, “Better discount evaluation: Illustrating
  how critical parameters support heuristic creation,” Interacting with
  Computers, vol. 17, no. 5, pp. 592–612, 2005.

Conte2009
T. Conte, J. Massolar, E. Mendes, and G. Travassos, “Web usability inspection
  technique based on design perspectives,” IET Software, vol. 3,
  no. 2, p. 106, 2009. [Online]. Available:
  <http://digital-library.theiet.org/content/journals/10.1049/iet-sen.2008.0021>


petersenAl:ease2008
K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic mapping
  studies in software engineering,” in Proceedings of the 12th
  International Conference on Evaluation and Assessment in Software
  Engineering, ser. EASE'08.1em plus 0.5em minus 0.4emSwinton,
  UK, UK: British Computer Society, 2008, pp. 68–77. [Online]. Available:
  <http://dl.acm.org/citation.cfm?id=2227115.2227123>


kitchenham:east2012
B. A. Kitchenham, “Systematic review in software engineering: Where we are and
  where we should be going,” in Proceedings of the 2Nd International
  Workshop on Evidential Assessment of Software Technologies, ser. EAST
  '12.1em plus 0.5em minus 0.4emNew York, NY, USA: ACM, 2012, pp.
  1–2. [Online]. Available: <http://doi.acm.org/10.1145/2372233.2372235>








§ DETAILED DESCRIPTION OF 



In this section we present a detailed explanation of each of the 8
stages of . We describe the purpose of each stage, specify
input and output artifacts, and discuss their relevance. As shown in
Figure 2, the eight stages of are:



  * Search for specific information

  * Search for usability heuristics

  * Heuristic specificity

  * Heuristic normalization

  * Heuristic prioritization 

  * Detailed description of heuristics

  * Validation 

  * Refinement




 §.§ Stage 1: Search for specific information



The first stage is exploratory in nature, and its main objective is to
determine the characteristic dimensions of a specific domain,
. The characteristic dimensions proposed are the following:



  * Usage contexts, .

  * Interactive logic devices, .

  * Interactive physical devices, .

  * User Profiles, .


Input The name or description of domain is required to
begin to determine the characteristic dimensions. For example: “grid
computing”, “e-learning”, etc.

Output This step produces two artifacts. The first is a list
of relevant keywords for the bibliographic search in the next
step (<Ref>). The second is the list of
characteristic dimensions and their initial specificity
  indices, which is summarized in tables, such as <Ref>, one
for each characteristic dimension. The specificity values follow a
standard 5-level Likert scale, where 0 is not specific and 4 is
completely specific. These specificity scores are used in step 5
(<Ref>) for calculating the final specificity
index.



An immediate goal for future work is to make a list of pre-selected
choices for each characteristic dimension. For example; suggest
physical devices such as desktops, tablets, phones, touchscreen, etc.,
to help complete this stage.







 §.§ Stage 2: Search for usability heuristics



The objective of this stage is to conduct a bibliographical search to
identify sets of usability heuristics related to the domain under
study. In general, and as suggested research practice, it is
recommendable to perform a systematic mapping or a
systematic review of
literature <cit.>.

Input At this stage the keywords related to the
domain are used, from Stage 1 (<Ref>).

Output At this stage sets of heuristics are obtained
S_1, …, S_n, which are potentially applicable to the domain. A
unique identifier must be assigned to each heuristic, for example,
H_1^S_1 represents the first heuristic of the set S_1.

If there is a set of heuristics validated for the domain and which
suits the researchers’ needs, it is possible to terminate the
application of at this point. These heuristics can also
serve as control heuristic in Stage 7 (<Ref>), in the
event that new heuristics for the domain are defined.

In general, Stages 1 and 2 specify how to carry out the process of
extracting information, as reported by Hermawati and Lawson
<cit.>, giving specific recommendations on how to
conduct the search, and taking into account the determining aspects of
the domain under study. As mentioned in <Ref>,
and in this appendix, there is considerable potential for improvement
in these stages.







 §.§ Stage 3: Heuristic specificity



In order to perform an initial filtering, at this stage we assign an
initial specificity index , associated with each heuristic
found in the previous stage.

Input Heuristic sets S_1, …, S_n, identified in the
previous stage, are required.

Output At this stage a tabulation of heuristics is obtained
with their initial specificity indices, as shown
in <Ref>. We say that these heuristics are
denormalized, since they don’t resolve duplication and/or
overlap conflicts between the heuristics yet. However, it is possible
to rank the heuristics and eliminate those considered unlikely to be
applied to the domain.









 §.§ Stage 4: Heuristic normalization



The objective of this stage is to resolve possible cases of overlap
or duplication in the heuristics obtained so far.

Input The denormalized heuristics are received as input,
along with their initial specificity indicators.

Output At this stage a preliminary set of heuristics
applicable to the domain is produced, ensuring that there are no
overlap-ping or duplication problems between heuristics. Each
heuristic will have an score, possibly revised after
normalization.

In general, cases of multiplicity can be solved through any of the
following:



  * Keep one of the heuristics that have similarity and discard the
  others.


  * Discard similar heuristics and reformulate a new heu-ristic that
  combines the characteristics of the heuristics that make it up.


Thus, cases of overlapping can be solved by any of the fol-lowing
strategies:



  * Maintain a general heuristic that groups together the
  overlapping heuristics.


  * Separate the heuristic overlapping into several individual
  heuristics.


This process of resolving overlap and multiplicity should be carried
out iteratively until obtaining a set of normalized heuristics, in
other words, a set without any overlaps or multiplicities. Note that
if new heuristics are created, following the aforementioned
strategies, these must also have a unique identifier. Finally, it is
recommended to review and reconsider the values of the specificity for each one the normalized heuristics.







 §.§ Stage 5: Prioritization of Heuristics



The objective of this stage is to synthesize the applicability of the
normalized heuristics, considering the specificity of each one with
respect to context of use, as well as the initial specificity
indicators, to have a rankable list of heuristics that can be applied
to the domain.

Input This stage receives the normalized heuristics obtained
in the previous stage, together with the respective indicators. In addition, it works with characteristic dimensions, ,
, , ; obtained in Stage 1.

Output The principal product for this stage is the
specificity matrix, as per the example
in <Ref>, which facilitates a final process
of ranking and selecting the heuristics that can be considered most
applicable for the domain.



This matrix contains, for each heuristic H_j, a summary of the
following specificity indicators:



  * : initial specificity indicators, created in Stage 1 and
  refined in Stage 4.


  * , , and : global specificity indicators
  of each characteristic dimension. To exemplify the calculation of
  these indicators, we consider the dimension, for which we
  created a tabulation such as <Ref>. For each usage
  context, a specificity score was assigned, and the score is
  the average of each row. Similarly, tables were created for the
  other characteristic dimensions.





Finally, the final specificity index synthesizes the specificity
and applicability of the different heuristics to the domain in a
single value, considering the initial evaluation and the evaluation of
each characteristic dimension.

For a heuristic H_j, the index is calculated with the following
formula:


    _H_j = 4 * _H_j * ∑𝐺𝑆𝐼_{, , , }/64


The final value of varies between 0 and 4; in general, a high
value of is expected to indicate high applicability of that
heuristic, although the final selection criterion is always within the
discretion of the researchers involved, with the methodological
backing granted by 



 §.§ Stage 6: Detailed description of heuristics



At this stage the description of the heuristics selected in the
previous stage are formalized, in order to design the experimental
validation for Stage 7. This formal description can provide
information necessary to understand and apply the domain heuristics
when performing a heuristic evaluation, particularly for inexperienced
evaluators, as evidenced
in <cit.>. <Ref> presents a
template format that can be used at this stage. This template takes
the basis proposed by  <cit.>, and adds a new
field, Checklist, with the intention to further facilitate its
application, following the suggestions in <cit.>.

Input The selected set of heuristics for application to the
domain.

Output The formal description of each heuristic, based on
the standard template.









 §.§ Stages 7 & 8: Validation and Refinement



The ultimate purpose of culminates in the stages of
validation and refinement, in which it seeks to empirically validate
that the selected domain heuristics are more effective than a control
group of heuristics. As explained in <Ref>,
defines quantitative quality indicators for this
comparison, and to guide the process of iterative refinement if
necessary. Of course, it is always advisable to perform additional
validations, especially qualitative; provides a solid
foundation for validating domain heuristics.

Input The selected domain heuristics, along with their
formal description. Additionally, the set of control heuristics to be
used. By default Nielsen’s heuristics <cit.> should
be used as a control group, unless there are more specific heuristics
and they are already validated.

Output After the experimental evaluation, the following
artifacts are obtained:




  * Set of problems detected in the heuristic evaluation(s). The
  problems are classified into common problems, ,
  problems specific to the domain heuristics, ,
  and problems specific to the control heuristics,
  . Each problem is associated with a particular
  heuristic.


  * Each problem’s severity is specified, using a 5-level Likert
  scale.


  * For each problem in
  = ∪,
  a specificity coefficient, with the same kind of scale as for
  severity.



In addition, with this information the following quality indicators
are calculated:




  * Unique problem rate .


  * Problem dispersion rate .


  * Problem severity rate .


  * Problem specificity rate .



Regarding the indicators, those were described
in <Ref>, and we only need to define the rate of
specificity. Let us recall that this indicator is defined as follows:


    =  /

 
where and are the
specificity averages for the problems found, respectively, by the
domain heuristics and the control heuristics.

The point we must explain is that the and
calculations are different, although they
aim to quantify the same phenomenon: how specific the problems
encountered are. The problem is that when the control heuristics are
very general, for example Nielsen’s, it is difficult to decide when a
problem is domain-specific. For that reason, the problems in
need to be qualified according to specificity. That
is to say, corresponds to the simple
specificity average for each problem. Nevertheless, on the other hand,
the domain heuristics have been designed and selected based on their
specificity indices – in particular , which synthesizes this
characteristic on a heuristic level. Therefore,
corresponds to a weighted sum of the number
of problems in each heuristic, multiplied by the associated with
it. In summary, we have that:


    = ∑_H_j (|P ∈ H^D_j| * _j)/||
       = ∑_i = 1^n ε (P_i)/||


for all domain heuristics H_i, where:




  * |P ∈ H_j| is the number of problems associated with said heuristic.


  * || and || correspond to
  the number of domain and control problems, respectively.


  * _j is the final specificity index for said heuristic
  H_j.


  * n is the total number of problems in .


  * ε (P_i) is the individual specificity of a given
  problem P_i, and associated with the control group problems.





§ INTERVIEW FOR THE INITIAL EVALUATION OF 



In this section we present the interview conducted as part of the
initial validation of , designed to collect information in
respect to the perception of as a methodology to create
usability heuristics, based on the following aspects and whose
questions are summarized in <Ref>.




  * Method of application

  * Interest or final product

  * Stages applied 

  * Easiest and/or most difficult stages

  * Usefulness  

  * Quantification of application time

  * Validation mechanisms of the generated products

  * Future plans for application of the process






