Analyzing control questions data for a survey I have an experimental study with a list of demographic and related questions and in order to identify data from participants that were potentially just answering the questions at random (to get through them more quickly I would assume), I've included two very similar 7-point likert scale questions at different points in the survey. My assumption would be that since the questions are reflective, the answers participants should give will be at least somewhat similar between the two questions (eg, it should be very unlikely that a participant answers 7 to one question yet 1 to the other).

I haven't yet collected the data, however I would like to have a method for determining which sets of data are suspicious (might be considered for exclusion in analysis) based on these control questions. One method might be to simply determine where the data fit on a Gaussian distribution. However, I think that the limited discriminating power of a 7-point scale would make this an improper test. My other idea was to do a cluster analysis on the data, looking for five groups: three along the line of correlation (between the questions), and two to examine unusually high/low and low/high values. I thought this could provide better suggestions for which data sets might be unusual since it wouldn't use somewhat arbitrary comparisons, it would only use the data given.

I'd really appreciate any suggestions for a better method, or improvements I could make as well as any comments toward more "standard" practices in this area, since I'm somewhat new to research.
You seem to be concerned with reliability, and more specifically internal reliability. Internal reliability is the degree to which different questions are measuring the same construct. This concept is used often in psychology and is usually measured using Cronbach's alpha. However, it is typically used to measure the reliability of a test, and not the reliability of an individual.

As Jeromy Anglim points out, I think it's important to consider the goal here. Using a two question Likert scale is probably not good enough to reliably detect outliers: What if the respondent checked all '4s' on a 7-point Likert scale? Reversing the scale would have no effect.

One alternative approach is to employ an instructional manipulation check (Oppenheimer et al., 2009). The gist of the technique is to trap participants into answering a question in a specific way that they could only have done by reading the instructions carefully. Here is an example from a survey administered by Facebook:



While this technique may throw out a few good participants, it will almost certainly raise the signal-to-noise ratio of your data by only including participants who followed instructions and read questions before answering.

Another tried and true technique is to use a computer-administered test and look at reaction times. You may be able to throw out a few responses (or whole participants) by simply looking for outliers in response time that are below the mean.


  Oppenheimer, D. M., Meyvis, T., &amp; Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.