overall MOS, evaluated for all sequences, is a good indicator of their reliability in comparison with the remaining test participants. Advanced models While rating stimuli, humans are subject to biases. These may lead to different and inaccurate scoring behavior and consequently result in MOS values that are not representative of the “true quality” of a stimulus. In the recent years, advanced models have been proposed that aim at formally describing the rating process and subsequently recovering noisiness in subjective ratings. According to Janowski et al., subjects may have an opinion bias that generally shifts their scores, as well as a scoring imprecision that is dependent on the subject and stimulus to be rated. Li et al. have proposed to differentiate between subject inconsistency and content ambiguity. Standardized testing methods There are many ways to select proper sequences, system settings, and test methodologies. A few of them have been standardized. They are thoroughly described in several ITU-R and ITU-T recommendations, among those ITU-R BT.500 and ITU-T P.910. While there is an overlap in certain aspects, the BT.500 recommendation has its roots in broadcasting, whereas P.910 focuses on multimedia content. A standardized testing method usually describes the following aspects: how long an experiment session lasts where the experiment takes place how many times and in which order each PVS should be viewed whether ratings are taken once per stimulus (e.g. after presentation) or continuously whether ratings are absolute, i.e. referring to one stimulus only, or relative (comparing two or more stimuli) which scale ratings are taken on Another recommendation, ITU-T P.913, gives researchers more freedom to conduct subjective quality tests in environments different from a typical testing laboratory, while still requiring them to report all details necessary to make such tests reproducible. Examples Below, some examples of standardized testing procedures are explained. Single-Stimulus ACR (Absolute Category Rating): each sequence is rated individually on the ACR scale. The labels on the scale are "bad", "poor", "fair", "good", and "excellent", and they are translated to the values 1, 2, 3, 4 and 5 when calculating the MOS. ACR-HR (Absolute Category Rating with Hidden Reference): a variation of ACR, in which an original unimpaired source sequence is shown in addition to the impaired sequences, without informing the subjects of its presence (hence, "hidden"). The ratings are calculated as differential scores between the reference and the impaired versions. The differential score is defined as the score