text:
Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data
Repeated measures analyses of variance are the method of choice in many studies from experimental psychology and the neurosciences. Data from these fields are often characterized by small sample sizes, high numbers of factor levels of the within-subjects factor(s), and nonnormally distributed response variables such as response times. For a design with a single within-subjects factor, we investigated Type I error control in univariate tests with corrected degrees of freedom, the multivariate approach, and a mixed-model (multilevel) approach (SAS PROC MIXED) with Kenward–Roger's adjusted degrees of freedom. We simulated multivariate normal and nonnormal distributions with varied population variance–covariance structures (spherical and nonspherical), sample sizes (N), and numbers of factor levels (K). For normally distributed data, as expected, the univariate approach with Huynh–Feldt correction controlled the Type I error rate with only very few exceptions, even if samples sizes as low as three were combined with high numbers of factor levels. The multivariate approach also controlled the Type I error rate, but it requires N ≥ K. PROC MIXED often showed acceptable control of the Type I error rate for normal data, but it also produced several liberal or conservative results. For nonnormal data, all of the procedures showed clear deviations from the nominal Type I error rate in many conditions, even for sample sizes greater than 50. Thus, none of these approaches can be considered robust if the response variable is nonnormally distributed. The results indicate that both the variance heterogeneity and covariance heterogeneity of the population covariance matrices affect the error rates.
KeywordsAnalysis of variance Robustness Nonnormality Small sample settings Repeated measurements Correlated data Multivariate Mixed model analyses Multilevel model Simulation study Type I error rate Central limit theorem Monte Carlo
Repeated measures designs in which each experimental unit (e.g., subject) is tested in more than one experimental condition are very common in psychology, the neurosciences, medicine, the social sciences, and agricultural research. The data from such experiments are often analyzed with analyses of variance (ANOVAs). The statistical assumptions that must be met in order for repeated measures ANOVAs to be valid are stronger than for data from a completely randomized design (also termed an independent-groups design) in which each experimental unit is tested only under one single experimental condition. For example, when testing for main effects and interactions with more than one numerator degree of freedom, the variance–covariance structure of the data is important for the validity of the tests (cf. Huynh & Feldt, 1970; Keselman, Algina, & Kowalchuk, 2001; Rouanet & Lépine, 1970). The extents to which several different approaches for the analysis of repeated measures data are robust, in the sense that they control the Type I error rate (i.e., the probability that the null hypothesis will be rejected, given that it is true), have been studied extensively for certain repeated measures designs (for reviews, see Keselman et al., 2001; Keselman, Algina, & Kowalchuk, 2002).
The aim of the present study was to investigate the control of the Type I error rate exhibited by different types of repeated measures ANOVAs in situations characterized by very small sample sizes (N as low as three) and nonnormality. Such data are frequently encountered in controlled experiments in the fields of experimental psychology, the neurosciences, and in certain clinical studies. However, the robustness of the different procedures for the analysis of this specific type of repeated measures data has not yet been investigated very systematically. We illustrate three characteristics of the data type examined in our study with example studies from the relevant fields that have each been cited between 300 and 2,000 times, according to the Science Citation Index (http://isiknowledge.com/), and can thus be considered as relevant and accepted contributions to the respective fields of research. The three characteristics are (a) small or very small sample sizes (typically, N = 3 to 30), (b) nonnormally distributed dependent measures, and (c) the use of a completely within-subjects design (i.e., there are no between-subjects factors).
Characteristic (a) can be attributed to economic considerations. In many experiments—for example, in visual psychophysics or psychoacoustics—the experimentation time required for each subject is high. In these fields, typically large numbers of trials are collected per subject and experimental condition, because experimenters wish to minimize the effect of, for instance, day-to-day fluctuations on detection thresholds, or because they want to compute measures of sensitivity, such as d´ from signal detection theory (cf. Green & Swets, 1966), that require several hundreds of trials. Additionally, researchers are often interested in the best possible performance that subjects can attain, in order to explore the limits of the sensory and cognitive systems. Therefore, highly trained subjects are required, and training periods of up to 5 h are not unusual (e.g., Duncan & Humphreys, 1989). As a consequence, to keep experimentation time at a manageable level, small numbers of subjects are tested (N = 2 to 10; e.g., Duncan & Humphreys, 1989; Eriksen & Eriksen, 1974; Ernst & Banks, 2002). If physiological responses are collected, there are additional reasons to restrict the number of subjects tested. For example, in electroencephalography (EEG) experiments, which nowadays often use up to 128 electrodes, the attachment of the electrodes is time consuming and also poses a certain risk of infection to the subjects, so that often small samples are used (N = 5 to 20; e.g., Sams, Paavilainen, Alho, & Näätänen, 1985). In neuroimaging studies using fMRI, operating the MRI scanner is expensive, not least because qualified medical and technical personnel are required. Additionally, the time available for research use is often very limited in clinics, due to the priority of medical use. Beyond that, the data analysis for a high number of subjects would be time consuming, because the fMRI data have to be mapped to anatomical structures on an individual basis (cf. Brett, Johnsrude, & Owen, 2002). Therefore, small sample sizes are again common (N = 10 to 20; e.g., Kanwisher, McDermott, & Chun, 1997). Other factors imposing limitations on the sample size are limited access to rare populations—such as, for example, persons with special types of synesthesia or patients with specific neurological disorders—or the ethical policy of minimizing the use of animals for research.
Concerning the distributional form of the response variables [characteristic (b)], it should be noted that not only in the research fields discussed here, but also in many other areas in psychology, strong deviations from normality are frequently encountered (Micceri, 1989). In psychophysics and cognitive neuroscience, two very important response measures are nonnormally distributed. Response times show a skewed and heavy-tailed distribution that is often successfully modeled as an ex-Gaussian distribution, which is a convolution of the Gaussian and exponential distributions (Luce, 1986; Van Zandt, 2000). In first approximation, response times follow a (shifted) log-normal distribution (Heathcote, Brown, & Cousineau, 2004; Ulrich & Miller, 1994). Proportions like error rates or the proportion of correct responses are binomially distributed. Other important measures, such as detection thresholds, d´ values, or amplitudes of evoked EEG responses, however, can be considered to be normally distributed.
Finally, the use of designs containing only within-subjects factors [characteristic (c)] is due to the fact that, for example, in basic research on vision or audition, interindividual differences are often considered less important because the aim is to understand the basic functioning of these sensory systems. Therefore, designs containing only within-subjects factors (i.e., each subject is tested under all experimental conditions) are advantageous because they provide comparably high power even with small sample sizes.
To investigate the robustness of repeated measures analyses for data showing the three characteristics discussed above, we simulated a design with a single within-subjects factor, no between-subjects factors, small sample sizes, and both normally and nonnormally distributed response variables. Due to the substantial computation time required for our simulations, we focused on Type I error rates (i.e., probability of false positives) and did not obtain Type II error rates (i.e., power; Algina & Keselman, 1997, 1998; Potvin & Schutz, 2000). In fundamental research, often more emphasis is placed on avoiding Type I errors than on avoiding Type II errors. Therefore, our data provide a basis for identifying procedures that control the Type I error rate in the specific situations that we studied. Additional simulation studies will be required to compare the procedures that we identified as suitable in terms of the Type I error rate with respect to statistical power (Type II error rate). Following many previous studies on the empirical Type I error rate for repeated measures designs, we adopted the "liberal" criterion of robustness proposed by Bradley (1978), according to which a procedure can be considered robust if the empirical Type I error rate \( \widehat{\alpha } \) is contained within the interval \( 0.5\alpha \leqslant \widehat{\alpha}\leqslant 1.5\alpha \), where α is the nominal Type I error rate (i.e., level of significance).
Analysis approaches
Before explaining the design of our study, we will briefly introduce three different approaches that are widely used for the analysis of repeated measures data,1 and that we therefore included in our study. A detailed description of these procedures can be found in Keselman et al. (2001).
Univariate approach with df correction
In the one-factorial, completely within-subjects design that we simulated, each subject (i = 1 . . . N) is measured once under all levels (k = 1 . . . K) of the within-subjects factor, and there are no between-subjects (grouping) factors. In the univariate approach, the F-distributed test statistic is the ratio between the mean square for the within-subjects factor W and the mean square for the W × Subject interaction. This test statistic depends on the assumptions of normality, independence of subjects, and sphericity of the population variance–covariance matrix (Huynh & Feldt, 1970; Rouanet & Lépine, 1970). Sphericity means that all orthonormalized contrasts (e.g., K − 1 treatment differences) have the same population variance (Huynh & Feldt, 1970; Rouanet & Lépine, 1970). If the assumption of sphericity is violated, the F test will result in too many Type I errors. Box (1954) showed that in this case, the test statistic is approximately distributed as F[α; ε(K − 1); ε(N − 1)(K − 1)], where the population parameter ε quantifying the deviation from sphericity is in the interval [1/(K − 1), 1.0] (Geisser & Greenhouse, 1958) and ε = 1.0 for a spherical population variance–covariance matrix (Huynh & Feldt, 1970).
Two popular sample estimates of ε have been proposed by Greenhouse and Geisser (1959) and Huynh and Feldt (1976). Both variants were included in our study, and will be denoted by GG and HF in the following. Other variants of df-adjusted univariate tests are discussed by Quintana and Maxwell (1994) and Hearne, Clark, and Hatch (1983).
Multivariate approach
The multivariate test of the effect of the within-subjects factor is performed by first creating K − 1 difference variables between pairs of factor levels, or more generally, K − 1 orthonormalized contrasts (for an excellent introduction, see Maxwell & Delaney, 2004). Next, Hotelling's (1931) multivariate T2 statistic is used for testing the hypothesis that the vector of population means of these K − 1 difference variables equals the null vector. Unlike the univariate test, the multivariate test (denoted by T2 in the following) does not require sphericity, but only that the covariance matrix be positive definite. However, like the univariate approach, it is of course based on the assumptions of normality and the independence of observations across subjects. An important aspect is that the multivariate test requires N ≥ K, which prevents its use if a small sample size is combined with a high number of factor levels, as was the case in most of the example studies cited above.
Mixed-model analysis
A third approach for testing the repeated measures main effect has gained importance during the last decade, partly due to the increasing availability of powerful computer hard- and software. This so-called mixed-model analysis2 is often also referred to as multilevel models (especially in educational psychology and the social sciences), hierarchical linear models, or random coefficient models (especially if used with continuous covariates—that is, in multiple regression problems) (cf. Maxwell & Delaney, 2004, p. 763).
Population covariance structures used for data simulation
Compound symmetric (CS) (2 parameters) \( {\sigma^2}\left[ {\begin{array}{*{20}c} 1 \hfill & \rho \hfill & \rho \hfill & \rho \hfill \\ {} \hfill & 1 \hfill & \rho \hfill & \rho \hfill \\ {} \hfill & {} \hfill & 1 \hfill & \rho \hfill \\ {} \hfill & {} \hfill & {} \hfill & 1 \hfill \\ \end{array}} \right] \)
Random coefficients (RC) (4 parameters) \( \left[ {\begin{array}{*{20}c} 1 & 0 \\ 1 & 1 \\ 1 & 2 \\ \vdots & \vdots \\ 1 & {K-1} \\ \end{array}} \right]\left[ {\begin{array}{*{20}c} {{\sigma_{11 }}} & {{\sigma_{12 }}} \\ {{\sigma_{12 }}} & {{\sigma_{22 }}} \\ \end{array}} \right]\left[ {\begin{array}{*{20}c} 1 & 0 \\ 1 & 1 \\ 1 & 2 \\ \vdots & \vdots \\ 1 & {K-1} \\ \end{array}} \right]+\left[ {\begin{array}{*{20}c} {\sigma^2 } & {} & {} & {} \\ {} & {\sigma^2 } & {} & {} \\ {} & {} & \ddots & {} \\ {} & {} & {} & {\sigma^2 } \\ \end{array}} \right] \)
Unstructured (UN) (\( {{{K\left[ {K+1} \right]}} \left/ {2} \right.} \) parameters) \( \begin{array}{*{20}c} {\sigma_{11}^2} & {{\sigma_{12 }}} & {{\sigma_{13 }}} & \cdots & {{\sigma_{1K }}} \\ {} & {\sigma_{22}^2} & {{\sigma_{23 }}} & \cdots & {{\sigma_{2K }}} \\ {} & {} & {\sigma_{33}^2} & \cdots & {{\sigma_{3K }}} \\ {} & {} & {} & \ddots & \vdots \\ {} & {} & {} & {} & {\sigma_{KK}^2} \\ \end{array} \)
Heterogeneous first-order autoregressive [ARH(1)] (K + 1 parameters) \( \begin{array}{*{20}c} {\sigma_1^2} & {\sigma_1 {\sigma_2}\rho } & {\sigma_1 {\sigma_3}{\rho^3}} & \cdots & {\sigma_1 {\sigma_K}{\rho^{K-1 }}} \\ {} & {\sigma_2^2} & {\sigma_2 {\sigma_3}\rho } & \cdots & {\sigma_2 {\sigma_K}{\rho^{K-2 }}} \\ {} & {} & {\sigma_3^2} & \cdots & {\sigma_3 {\sigma_K}{\rho^{K-3 }}} \\ {} & {} & {} & \ddots & \vdots \\ {} & {} & {} & {} & {\sigma_K^2} \\ \end{array} \)
Unlike the univariate and multivariate approaches, which use least-squares procedures, a maximum-likelihood or restricted maximum-likelihood approach (cf. Jennrich & Schluchter, 1986; Littell, Milliken, Stroup, Wolfinger, & Schabenberger, 2006) is used for parameter estimation for the mixed model, with the potential consequence of numerical problems.
Note that the (restricted) maximum-likelihood-based PROC MIXED analysis can be used even if there are missing values (cf. Padilla & Algina, 2004)—more precisely, if the missing data mechanism is missing completely at random (MCAR) or missing at random (MAR) (Rubin, 1976). Missing values usually play no role for well-controlled laboratory experiments, so we did not consider this issue in our study.
Previous simulation studies
Not many simulation studies measuring Type I error rates exist for completely within-subjects designs with nonnormal data and/or very small sample sizes. It has been suggested, however, that simulation results for designs with a combination of within- and between-subjects factors, identical group sizes (i.e., balanced designs), and equal covariance matrices across groups should be similar to what can be expected for a completely within-subjects design (e.g., Keselman et al., 2001, p. 11; Keselman, Kowalchuk, & Boik, 2000, p. 55). For "split-plot" designs containing both between- and within-subjects factors, many simulation studies are available (for reviews, see Keselman et al., 2001, 2002), due to the importance of this type of design for educational and clinical psychology and for the social sciences. Still, even if studies containing between-subjects factors are considered, it has to be stated that Type I error rates have not yet been obtained very systematically for extremely small sample sizes and nonnormal data. We will first discuss studies simulating small samples and nonnormal data, and then consider studies in which small sample sizes were simulated but the data were normally distributed.
For N = 6 or 9 and a design with equal group sizes and equal covariance matrices across groups, Keselman et al. (2000) reported frequent liberal error rates if the data were χ2(3) distributed. The multivariate approach showed a stronger tendency toward liberal error rates than did a univariate approach with df correction. For normally distributed data, both approaches controlled the Type I error rate (Huynh & Feldt, 1976; Lecoutre, 1991).
Berkovits, Hancock, and Nevitt (2000) simulated a design with a single within-subjects factor (K = 4), no between-subjects factors, and varied the sample size between 10 and 60. For normal data and N = 10, HF, GG, and T2 controlled the Type I error rate. With increasing skewness and kurtosis of the simulated response variable, the three procedures produced conservative Type I error rates for a spherical population covariance matrix and showed liberal behavior at small values of ε. For strong deviations from normality (skewness = 3.0, kurtosis = 21.0), the results were robust only at the highest sample size (N = 60).
Wilcox, Keselman, Muska, and Cribbie (2000) reported both conservative and liberal Type I error rates in a design with one within-subjects factor (K = 4, N = 21) if the distribution of the dependent variable was nonnormal. For strongly heavy-tailed distributions (g-and-h distribution with h = 0.5; cf. Headrick, Kowalchuk, & Sheng, 2010), the univariate approach with Huynh–Feldt df correction often produced conservative Type I error rates, especially if the variances under the four factor levels were equal. If the variances were unequal, the multivariate approach produced conservative results for symmetric and heavy-tailed distributions, but highly liberal results for asymmetric and heavy-tailed distributions. In the latter case, HF also occasionally produced liberal Type I error rates.
Muller, Edwards, Simpson, and Taylor (2007) simulated normal data from a design without between-subjects factors (K = 9), studied sample sizes between 10 and 40, and varied the sphericity parameter ε of the population covariance matrix. At an α level of .04, GG and HF always met Bradley's liberal criterion. PROC MIXED fitting a UN or CS type of covariance matrix and using the Kenward–Roger (1997) adjustment produced liberal Type I error rates if N = 10 and ε < 1.0. At N = 20, PROC MIXED produced robust results if a UN rather than a CS type of covariance matrix was fitted.
Skene and Kenward (2010a) simulated a design with two groups, five subjects per group (i.e., N = 10), and a continuous within-subjects covariate with five levels (K = 5). The response variable was normally distributed. PROC MIXED fitting an unstructured covariance matrix and using the Kenward and Roger (1997) adjustment controlled the Type I error rate.
Gomez, Schaalje, and Fellingham (2005) simulated normal data, with one continuous within-subjects factor (three to five levels), one between-subjects factor with three levels, and total sample sizes between nine and 15. For a balanced design with identical covariance matrices across groups, PROC MIXED with the Kenward–Roger adjustment controlled the Type I error rate if the correct covariance structure was fitted, except for a single type of population covariance structure. If the covariance structure was selected via the Akaike (AIC; Akaike, 1974) or Bayesian (BIC; Schwarz, 1978) information criteria, several liberal error rates were reported.
Taken together, for samples sizes as low as N = 6, the univariate approaches with df correction and the multivariate approach seem to control Type I error rates for normal data, which is the expected result. With some exceptions, this is also true for PROC MIXED with the Kenward–Roger adjustment. However, this pattern seems to change, sometimes even dramatically, if the distribution of the response variable is nonnormal.
To summarize the previous results, for repeated measures designs, no empirical Type I error rates have been reported for N < 6, the influence of sample size has only been investigated in most studies by comparing two or three different sample sizes, and the effects of nonnormality have also not been tested for very small sample sizes combined with higher numbers of levels of the within-subjects factor. Thus, our study for the first time provides empirical Type I error rates for (1) samples sizes as small as three and varied in small steps, (2) normal and nonnormal data, (3) numbers of factor levels ranging from four to 16, and (4) a rather wide variety of population covariance matrices.
Simulated experimental design
Characteristics of the simulated data sets
The simulated data sets varied in (a) the population distribution, (b) the population covariance structure, (c) the number of levels of the within-subjects factor, and (d) the sample size (i.e., the number of simulated subjects). These variables were factorially combined, and each resulting scenario was simulated 5,000 times. It should be noted that PROC MIXED requires substantial computation time, especially at large values of K and N when many parameters have to be estimated.
Population distributions
The following distributional forms of the simulated response variable were chosen for this study: (1) a normal distribution with μ = 0 and σ = 1, (2) a log-normal distribution3 with μ = 0 and σ = 1, and (3) a chi-squared distribution with two degrees of freedom.
The selected log-normal distribution had skewness and kurtosis values of 6.18 and 113.94, respectively. Skewness is a measure of asymmetry, and a distribution with positive kurtosis has a heavier tail and a higher peak than the normal distribution (see DeCarlo, 1997). Thus, the simulated log-normal distribution exhibited a degree of asymmetry and heavy-tailedness that, according to previous studies, might negatively affect the performance of the analysis procedures (Sawilowsky & Blair, 1992; Wilcox et al., 2000). Most previous simulation studies had used a log-normal distribution with μ = 0 and σ = 0.5 (e.g., Algina & Oshima, 1994), which has smaller values of skewness and kurtosis (1.75 and 8.89, respectively).4 We selected a distribution showing a rather extreme deviation from normality in order to test the robustness of the data analysis approaches in such cases. Note that skewed distributions are frequently encountered in psychological studies (Micceri, 1989; Wilcox, 2005), although it is unclear which values of, for instance, skewness are representative for a particular field of research (Keselman, Kowalchuk, Algina, Lix, & Wilcox, 2000; Wilcox, 2005). As we discussed above, the log-normal distribution can be considered a first approximation to the distribution of response times (cf. Heathcote et al., 2004), which is a dependent measure very frequently used in cognitive psychology. Recently, Palmer, Horowitz, Torralba, and Wolfe (2011) reported empirical values of skewness and kurtosis for response time distributions from a visual search experiment. They found skewness values greater than 4.0 and kurtosis values greater than 40.0 in some conditions, which again justifies our decision to include a log-normal distribution with rather high values of skewness and kurtosis.
The χ2(2) distribution exhibits extreme asymmetry (Micceri, 1989), because its mode coincides with the minimum. Although the values of skewness = 2 and kurtosis = 6 are moderate, the χ2(2) distribution is not only asymmetric but has a monotonically decreasing probability density function (PDF). Note that a χ2(2) distribution multiplied by a constant, as is included in the covariance matrices with unequal variances, is strictly speaking no longer a χ2(2) distribution, although in the literature it is usual to ignore this issue and to refer only to the base distribution used when generating the random variates (e.g., Keselman, Carriere, & Lix, 1993). Incidentally, a binomial distribution with success rate p = .05—as would, for example, occur for error rates in many experiments in cognitive psychology—also shows a strictly decreasing PDF if each proportion is based on fewer than 20 trials, as in, for example, the study by Meiran (1996) cited above.
Population covariance structures
As we discussed in the introduction, an important concern for repeated measures ANOVAs is the sphericity (or, rather, lack of sphericity) of the population covariance matrix (e.g., Huynh & Feldt, 1970; Rouanet & Lépine, 1970). For example, the univariate ANOVA approach assumes sphericity (Huynh & Feldt, 1976), and the correction factors for the degrees of freedom (e.g., Greenhouse & Geisser, 1959; Huynh & Feldt, 1976) attempt to correct for deviations from a spherical pattern, as quantified via Box's ε (Box, 1954), which is a population parameter. For this reason, we included a compound symmetric (CS) population covariance structure (see Table 1), which is spherical (i.e., ε = 1.0). In addition, three population covariance structures with ε = .5 were studied, thus showing a strong deviation from sphericity. As is shown in Table 1, these structures were of the unstructured (UN), random-coefficients (RC), and heterogeneous first-order autoregressive [ARH(1)] types (cf. Wolfinger, 1996). The reason to include different types of covariance structures with ε = .5 was to evaluate the performance of PROC MIXED with different fitted covariance structures. Beyond that, we were interested in whether the Type I error rates of the other ANOVA procedures would depend on the specific covariance structure for equal values of ε.
For K = 4 and ε = .75, covariance matrices of types UN, RC, and ARH(1) were specified by Keselman, Algina, Kowalchuk, and Wolfinger (1999a). These matrices were subsequently used in a substantial number of simulation studies. To construct covariance matrices for a higher number of factor levels and with ε = .5, we created covariance matrices with the appropriate structures, using randomly selected values for the parameters. For each pairing of covariance structure and number of factor levels, we then selected a covariance matrix with ε close to the desired value of .50 (squared deviation ≤ 10–4). For the power-transform method (Headrick, 2002) that we used to generate multivariate nonnormally distributed data (see the Data Generation section), positive definite covariance matrices are required because the method involves a Cholesky decomposition (Commandant Benoit, 1924). Additionally, the intermediate covariance matrices used by the power-transform method (see the Data Generation section) are also required to be positive definite. We selected covariance matrices to meet these criteria.
As can be seen in Table 1, the RC covariance structure needs four parameters: σ, σ11, σ22, and σ12. This is true for all numbers of factor levels. The common standard deviation σ was randomly selected from an interval from 0.1 to 4. The parameters σ11, σ22, and σ12 were sampled uniformly from an interval from −0.9 to 0.9.
The ARH(1) covariance structure has only one correlation parameter (ρ) and a variance parameter for every factor level (σi2). Therefore, the ARH(1) covariance structure needs K + 1 parameters. The standard deviations σi were first uniformly sampled from an interval from 1.0 to 4.0, and then the correlation parameter ρ was selected to produce the desired ε = .5.
The UN covariance structure has \( {{{K\left( {K+1} \right)}} \left/ {2} \right.} \) parameters. While K of these parameters represent the variances of the K variables, the remaining parameters represent the covariances. The standard deviations σii were first sampled uniformly from an interval from 0.5 to 6. Then the covariances σij were sampled uniformly from the interval [0.30σiσj, 0.97σiσj].
The CS covariance structure needs only two parameters, independent of the number of factor levels. This covariance structure is distinct from the other three, because by design the Box epsilon is ε = 1.0 for all CS covariance matrices (i.e., the matrix is spherical) and because all variances are equal. We used the parameter values σ2 = 20.0 and ρ = .8.
Supplement B (available online with this article) shows the covariance matrices used for data generation. Note that the nonspherical matrices also exhibited heterogeneous variances for the K variables.
Numbers of levels of the within-subjects factor
For the number of levels of the within-subjects factor (K), the following values were chosen: 4, 8, and 16. While a value of K = 16 might seem high, experiments using, for example, a large number of presentation levels (e.g., Florentine, Buus, & Poulsen, 1996) are not unusual in psychophysics. Additionally, a test with 16 − 1 = 15 numerator degrees of freedom would arise when testing for the interaction between two within-subjects factors that have four and six factor levels, respectively. The number of factor levels has been demonstrated to affect the Type I error rate and the power of repeated measures ANOVAs (e.g., Algina & Keselman, 1997; Keselman, Keselman, & Lix, 1995). Additionally, some of the analysis procedures cannot be applied if a high number of factor levels is combined with a small sample size. For example, the multivariate approach requires that the number of subjects N be equal to or greater than the number of factor levels of the within-subjects factor (N ≥ K).
Sample sizes
As we were especially interested in the performance of the procedures for extremely small sample sizes, we varied the sample size in steps of one between N = 3 and 10, and we also studied nine larger sample sizes (N = 16, 17, 18, 19, 20, 30, 50, 70, and 100). The small steps between 16 and 20 were included because pilot simulations indicated that PROC MIXED needs slightly more subjects than the number of factor levels for convergence when fitting a UN covariance structure.
Data analysis approaches
The empirical Type I error rates were analyzed for four different repeated measures ANOVA procedures. These were (1) the univariate approach with Greenhouse–Geisser correction for the degrees of freedom (GG), (2) the univariate approach with Huynh–Feldt df correction (HF), (3) the multivariate approach (T2), and (4) the mixed-model approach as computed by SAS PROC MIXED (see Keselman et al., 2001, for a description of the different approaches). Note that for the univariate approaches, the df correction was unconditionally applied (Keselman, Rogan, Mendoza, & Breen, 1980), instead of first conducting a test for lack of sphericity (Mauchly, 1940). The primary reason for this was that the Mauchly test requires N ≥ K(K − 1)/2, which rendered it useless for a considerable proportion of our simulated designs. Beyond that, the Mauchly test is sensitive to departures from normality (Huynh & Mandeville, 1979).
The analyses in PROC MIXED were conducted using the Kenward–Roger (1997)5 adjustment (option ddfm=KR in SAS 9.2), which has been demonstrated to be superior to alternative methods of computing the degrees of freedom (Arnau, Bono, & Vallejo, 2009; Fouladi & Shieh, 2004; Kowalchuk, Keselman, Algina, & Wolfinger, 2004; Schaalje, McBride, & Fellingham, 2002; Skene & Kenward, 2010a). Additionally, for the SAS PROC MIXED analyses, several different model covariance structures were fitted (cf. Keselman et al., 1999b; Kowalchuk et al., 2004; Littell et al., 2000)—namely UN, CS, ARH(1), CSH, HF, and RC (see Table 1 and Wolfinger, 1996, for detailed specifications of these structures). It should be noted that PROC MIXED offers a large amount of flexibility for specifying covariance structures. For example, the CS structure can be fit in two different ways (cf. Littell et al., 2000). The SAS syntax that we used for the six variants is available as supplemental material (Supplement A) in the journal's electronic supplementary archive.
Data generation
We simulated multivariate normally distributed data using the method of Kaiser and Dickman (1962), and nonnormal correlated data via the power method transformation proposed by Headrick and coworkers (Headrick, 2002; Headrick & Kowalchuk, 2007; Headrick, Sheng, & Hodis, 2007). The power method is based on work by Fleishman (1978) and by Vale and Maurelli (1983). Fleishman proposed a moment-matching approach to simulate nonnormally distributed data. First, normal deviates are generated. Then a polynomial transform of order three is applied, with the polynomial coefficients selected so that the first four moments of the transformed data match the corresponding moments of the target distribution. Headrick (2002) enhanced this approach by using fifth-order polynomials. Therefore, it is possible to match the first six moments of a distribution. It is also possible to match a wider range of distributions (e.g., combinations of skewness and kurtosis).
Following Headrick (2002) and Headrick et al. (2007), our simulations started by generating K independent standard normal deviates (X = X1 . . . XK; μ = 0, σ = 1) with the SAS IML function RANDNORMAL. From these independent normal deviates, K correlated normal deviates Z = Z1 . . . ZK with population correlation matrix R* were generated using the method of Kaiser and Dickman (1962). For the K × K matrix R*, the Cholesky decomposition U'U = R* (where U is an upper triangular matrix and U' is the transpose of U) yielded K(K + 1)/2 coefficients. These coefficients were used to compute K linear combinations of each Xk, Z = UX, which resulted in Z, correlated with the population correlation matrix R* (for an example, see Headrick et al., 2007, p. 11; Kaiser & Dickman, 1962). As Vale and Maurelli (1983) have shown, if a polynomial transformation is applied to these correlated normal deviates, the correlations between the resulting nonnormal deviates will not be equal to the correlations between the normal deviates that they were based on. Therefore, to produce K correlated nonnormal deviates with the intended population correlation matrix R, the Zk are generated with an intermediate correlation matrix R* that was selected so that, after applying the polynomial transformations to the Zk, the target correlation matrix R results. Headrick (2002) noted that if the correlation between a pair of standard normal deviates (Zi and Zj) is ρZiZj and then a polynomial transformation is applied to each of the two correlated standard normal deviates, so that two power-transformed variables (Yi and Yj) with mean 0 and variance 1 result, then the correlation ρYiYj is given by the expected value of the product of the latter two random variables, \( {\rho_{{Y_i {Y_j}}}}=E\left( {Y_i \cdot {Y_j}} \right) \) (see Eq. 26 in Headrick, 2002). To determine the intermediate correlation matrix R*, this equation is solved for ρZiZj, separately for each of the pairwise correlations, and given the to-be-applied polynomial coefficients (see Eq. 2 below) and the target correlations ρYiYj. We solved the equation numerically with the Wolfram Research Mathematica function NMinimize. The values of ρZiZj constitute the intermediate correlation matrix R*.
The polynomial coefficients c1i through c6i were chosen so that the first six standardized cumulative moments of Yk matched the first six moments of the target distribution—χ(2) or LogNormal[0, 1] (Headrick, 2002). The moments of a power-transformed normal deviate are available in closed form (cf. Headrick, 2002). We used a numerical procedure (Wolfram Research Mathematica NMinimize) to find the coefficients minimizing the sum of squared deviations between the six moments of the power-transformed normal deviate and the corresponding six moments of the target distribution [e.g., χ(2)], with an accuracy goal of 10−20.
The Yk are nonnormal deviates with mean 0 and variance 1, correlated with the desired population correlation matrix R. To create correlated nonnormal deviates with the desired population covariance matrix Σ, in the final step, each Yk was multiplied by the desired standard deviation. Note that we simulated data corresponding to the null hypothesis μ1 = μ2 = . . . = μK = 0.
To check the quality of our simulations, we computed the empirical covariance matrix, as well as the empirical moments of the individual random variables, for each of the 36 simulated multivariate distributions (i.e., for each combination of distribution of the simulated response variable, covariance structure, and number of factor levels). For each of these conditions, 1,959,999 data sets were simulated. The maximal deviation of the empirical Box (1954) ε from the target ε (1.0 or .5) was 0.51 %. The average root-mean squared deviation of the variances and covariances in the upper diagonal of the empirical covariance matrix, as compared to the intended covariance matrix (see Supplement B), was 0.0047, with a maximum of 0.101.
The first four empirical standardized moments (mean, variance, skewness, and kurtosis) were also found to be within a narrow range around the target values. Only for the log-normal distribution did the kurtosis show rather large variability, which is to be expected, because the fourth sample moment about the mean is strongly affected by extreme values.
Taken together, the empirical covariance matrices and the empirical moments showed that the simulation algorithm worked as expected.
Simulation program
The simulation program was written in the SAS MACRO and SAS/IML languages and was run on version 9.2 of SAS.
Type I error rates
For each simulated data set, the p value produced by a given analysis approach was compared to the nominal α level of .05, and the empirical Type I error rate \( \widehat{\alpha } \) for a condition (e.g., N = 3, K = 4, normally distributed data, CS population covariance structure) was computed as the number of tests in which p < α, divided by the total number of p values for this condition (i.e., 5,000, if all tests produced a p value). The empirical Type I error rates are binomially distributed; thus, the 95 % Wald confidence interval for the proportion \( \widehat{\alpha } \) is \( \widehat{\alpha}\pm 1.96\sqrt{{{{{\widehat{\alpha}\left( {1-\widehat{\alpha }} \right)}} \left/ {N} \right.}}} \) where N is the number of simulations. If, for example, the observed Type I error rate was .05 (i.e., 250 tests out of 5,000 were significant at the nominal α level of .05), then the true Type I error rate could be expected to be in the interval [.0456, .0544] with a probability of 95 %.
For the PROC MIXED analyses, we analyzed the empirical Type I error rates in five different cases: (1) fitting the correct covariance structure—for instance, fitting an UN structure if the data were generated using a population covariance matrix of type UN; (2) selecting the best PROC MIXED model in terms of the AIC (Akaike, 1974); (3) selecting the model on the basis of the BIC (Schwarz, 1978); (4) unconditionally fitting an unstructured (UN) model covariance matrix; or (5) unconditionally fitting a covariance matrix of the "heterogeneous compound symmetry" (CSH) type. In the following discussion, PROC MIXED Variants 1–5 are denoted by PMCC, PMAIC, PMBIC, PMUN, and PMCSH, respectively. Thus, as, for example, in Kowalchuk et al. (2004), one variant of the PROC MIXED approach used prior knowledge about the population covariance structure. It is, of course, unlikely that a researcher will know in advance which covariance structure his or her data will exhibit, but fitting the correct covariance structure should represent the best possible performance of the mixed-model approach. The two variants selecting the best covariance structure using likelihood-based information criteria attempted a trade-off between global goodness of fit and the number of model parameters (cf. Keselman et al., 2002). Thus, at least in theory, these information criteria should select the most parsimonious model still showing an acceptable level of goodness of fit. Finally, one of the two remaining variants always assumed a UN covariance structure, which should incorporate all possible empirical covariance structures, because it places no constraints on the structure of the covariance matrix. This flexibility comes at the cost of having to estimate a large number of parameters, making it impossible to use if a high number of factor levels is combined with a small sample size, and also potentially causing problems with numerical convergence (cf. Gomez et al., 2005). The CSH structure, on the other hand, is in a sense located between the UN structure and the CS structure (for a discussion, see, e.g., Wolfinger, 1996). The CS structure has equal variances and equal correlations. The CSH structure allows for unequal variances (i.e., entries on the main diagonal), while still assuming a constant correlation between all pairs of factor levels (Wolfinger, 1996). Note that the CSH structure is somewhat similar to the HF structure (Wolfinger, 1996), the latter corresponding to the spherical "Type H" structure discussed by Huynh and Feldt (1970). We opted for the CSH rather than the HF structure because, in our experience with analyzing real data sets, PROC MIXED models using the HF covariance structure frequently show problems with convergence, while fitting the CSH structure seems less problematic. This assumption was clearly supported by the convergence rates observed in our simulations (see the Convergence Rates section below).
Normally distributed data
Across the 204 combinations of population covariance structure [ARH(1), CS, RC, and UN], number of factor levels, and sample size, the univariate approach with Huynh–Feldt df correction (HF) controlled the Type I error rate according to Bradley's (1978) liberal criterion, except for four cases characterized by K = 16 and N < 7, for which the error rate was slightly higher than .075. At the smaller samples sizes, the error rates had a small tendency toward liberal error rates for the nonspherical population covariance structures, and a smaller tendency toward conservative error rates for the spherical CS structure. As expected, the GG df correction frequently resulted in conservative error rates at the smaller sample sizes, especially for the spherical CS population covariance matrix (open circles in Fig. 1; cf. Huynh & Feldt, 1976). The multivariate approach (T2) always showed almost perfect control of the error rate, which was expected because its assumptions were all met. However, T2 is applicable only for N ≥ K. When fitting a UN covariance structure, PROC MIXED required N > K for convergence (see the Convergence Rates section below) and produced liberal error rates for N = K + 1. The latter result was also obtained by Muller et al. (2007). Note that except for these conditions, the error rates were identical to the error rates produced by the multivariate approach, which is the expected result (Skene & Kenward, 2010a). When fitting the CSH structure, PROC MIXED frequently produced conservative error rates at small sample sizes and some liberal error rates at higher sample sizes. Selecting the covariance structure via AIC or BIC worked reasonably well for normally distributed data. At K = 4, both variants produced some liberal error rates at N < 10. Additionally, selection via AIC or BIC produced liberal error rates if N = K + 1, which can be attributed to the liberal behavior of PROC MIXED fitting a UN structure in this condition. This problem did not occur for selection via BIC if K = 16. Some error rates were conservative at N = 3 or 4, especially for selection via BIC. Finally, if the correct covariance structure was fit, the error rates were close to the nominal values, except for the two smallest samples sizes when the population covariance structure was ARH(1), and of course liberal rates for the UN population covariance structure at N = K + 1.
Taken together, for normally distributed data the HF test controls the Type I error rate with only very few exceptions and can therefore be recommended, especially if extremely small sample sizes are combined with high numbers of factor levels of the within-subjects factor. The multivariate approach worked perfectly if N ≥ K. For these sample sizes, future studies should compare the power of the HF and the T2 tests. Results by Algina and Keselman (1997) have suggested that the multivariate approach would be more powerful than HF if the sample size was large, the deviation from sphericity was large (i.e., ε < .85), and the number of factor levels was small. PMAIC, PMBIC, and the benchmark variant PMCC performed reasonably well, but all of them occasionally produced conservative or liberal Type I error rates at sample sizes smaller than 20. For N > K + 1, the error rates for PMUN were identical to those produced by the multivariate approach. PMCSH produced many nonrobust results and can therefore not be recommended.
χ2(2)-distributed data
In contrast to this generally conservative behavior observed for the CS population covariance matrices, for the nonspherical population covariance matrices the error rates were often liberal if the data were χ2(2) distributed. In fact, at K = 4 all procedures produced liberal error rates, except for the univariate df-adjusted approaches at the highest sample sizes. At K = 8, the univariate df-adjusted procedures controlled the Type I error rate except at the smallest samples sizes, where the GG test again erred on the conservative side, and HF produced some liberal error rates. The multivariate approach and PROC MIXED produced liberal results. At K = 16, the HF test showed robust behavior for the nonspherical population covariance matrices, while the GG procedure often behaved conservatively, and the error rates produced by the remaining procedures were frequently liberal. It should be noted that even at the three highest sample sizes (N ≥ 50), only GG and HF behaved robustly, while the problems with nonrobust Type I error rates remained for the other procedures. Across all conditions, the error rates were closest to the nominal value for procedure HF, but even this procedure produced many liberal or conservative error rates according to Bradley's liberal criterion.
Log-normally distributed data
Nonspherical population covariance structures with homogeneous variances
The results showed a pronounced effect of the population covariance structure for χ2(2)-distributed data. Can the observed differences between the error rates for the CS structure and for the remaining structures be attributed to the deviation from sphericity (i.e., ε = 1.0 vs. ε = .5), or to the fact that the K variances were equal for the CS population covariance matrix but unequal for the other matrices? To gain a preliminary insight into this question, we decided to conduct additional simulations for population covariance structures with homogeneous variances but ε = .5. We denote these structures by HV, for "homogeneous variances." For K = 8 or 16, we used a first-order autoregressive correlation structure. The correlations in this structure follow the same pattern as for the ARH(1) matrix depicted in Table 1, but the variances were all identical and equal to 16.0. For K = 8 and K = 16, the matrix has ε = .5 if ρ = .760 or .632, respectively. For K = 4, it is not possible to construct an AR(1) matrix with ε = .5. Therefore, we created a matrix with all variances set to 16.0 and with the correlations randomly sampled from the intervals [−.9, −.3] and [.3, .9]. We denote this matrix by UNs, for "unstructured–same variance." The population covariance matrices are displayed in Supplement B. The simulation results for the nonspherical matrices with equal variances are shown by the filled circles in Figs. 1–3. For normal data (Fig. 1), the Type I error rates did not differ strongly between the nonspherical matrices with homogeneous versus heterogeneous population variances. At K = 4, PMAIC, PMBIC, and PMCSH produced more liberal error rates at small sample sizes with the HV population covariance structure than with the remaining structures. At K = 8, procedure HF produced a liberal result at N = 4 or 5 for the AR(1) population covariance structure, but not for the nonspherical structures with heterogeneous population variances. PMCSH was liberal at larger sample sizes for the AR(1) but not for the UN population covariance structure, and PMAIC and PMBIC both produced one additional liberal error rate with the AR(1) structure. At K = 16, the AR(1) structure resulted in one additional liberal error rate for HF (at N = 3), and in liberal error rates for PMCSH at large sample sizes.
For the χ2(2)-distributed response variable (Fig. 2), at K = 4 GG, HF, T2, PMUN, and PMCC showed robust behavior for the UNs population covariance structure, with only four exceptions, in contrast to the generally liberal error rates observed with the population covariance matrices with heterogeneous variances. PMAIC and PMBIC also controlled the Type I error rate with the UNs structure if N > 10. At K = 8, where the error rates of most procedures were liberal with heterogeneous population variances, the AR(1) structure removed this tendency toward liberal decisions, resulting in robust behavior for HF, T2, PMUN, PMAIC, and PMBIC in most conditions. Essentially the same pattern was observed at K = 16. Thus, for the χ2(2)-distributed response variable, nonspherical population covariance matrices with homogeneous variances removed the strong tendency toward liberal results observed for the population covariance structures with heterogeneous variances. However, the error rates for the nonspherical population covariance matrices with homogeneous variances were not identical to the results for the CS structure, which shows variance homogeneity and covariance homogeneity, and for which the error rates were often conservative.
For the log-normal data (Fig. 3), the Type I error rates generally tended to be lower for the nonspherical population covariance matrices with homogeneous variances, as compared to the conditions with heterogeneous population variances. This resulted in more frequent conservative error rates at smaller sample sizes, but also in the absence of liberal error rates at the three largest sample sizes. Again, the error rates for the nonspherical population covariance matrices with homogeneous variances were not identical to the results for the CS structure.
Taken together, these results indicate that both variance heterogeneity and correlation heterogeneity affect the Type I error rates for nonnormal data, and this finding is compatible with the results by Wilcox et al. (2000). Heterogeneous variances probably present a stronger problem than do heterogeneous covariances. Additional studies will be desirable here. On a more general level, our study shows that it is worthwhile to test a wider variety of population covariance structures than those included in previous studies.
Convergence rates
The univariate and multivariate approaches are based on least-squares estimation and always produce a p value for the test of the omnibus hypothesis, given that the sample size is high enough to provide sufficient residual dfs. In contrast, the mixed-model analyses based on (restricted) maximum-likelihood can exhibit problems with convergence, and thus fail to produce a p value for a given data set. While most previous studies have ignored this problem (but see Gomez et al., 2005), for researchers planning to apply this specific type of analysis, it is important to know how likely these convergence problems will be for a given condition. For this reason, we report the proportions of tests in which a particular analysis procedure failed to produce a p value.
As can be seen in Supplement C, PROC MIXED always converged when fitting the CS model covariance structure. For the ARH type, the convergence rate was higher than .98 in all conditions. With the RC type, we observed some convergence rates smaller than .95 at N = 3, but with only a single exception, the convergence rates were higher than .90 even at this smallest sample size. For the CSH type, convergence rates higher than .95 were obtained except at the smallest samples sizes (N ≤ 4 for K = 4, N ≤ 5 for K = 8, and N ≤ 6 for K = 16). For the UN model covariance structure, the data indicated that the convergence rate was close to 1.0 as soon as the sample size was larger than the number of factor levels (N > K). Compatible with our own experience when analyzing real data, the problems with convergence were most pronounced when fitting the HF type model covariance structure. Here, at least twice as many subjects as factor levels seemed to be required for convergence rates higher than .9, and even much larger sample sizes for the log-normally distributed data.
In sum, fitting the more complex covariance structures with PROC MIXED can be a problem when sample sizes are small. In most cases, however, the convergence rates are very high. Note that our results on Type I error rates when selecting the model covariance structure via AIC or BIC indirectly consider problems with convergence, because if one PROC MIXED variant fails to converge, another model covariance structure will be selected. Thus, if Figs. 1–3 show that the Type I error rate for model selection via AIC or BIC is acceptable, then there were no problems with convergence.
In a simulation study, we obtained the empirical Type I error rates of different procedures for analyzing data from a repeated measures design with a single within-subjects factor and no grouping factor, with a focus on extremely small sample sizes and nonnormal data. In fact, we studied for the first time the behavior of several different analysis approaches for sample sizes smaller than six combined with a high number of levels of the within-subjects factor (K = 8 or 16). We also studied two distributions of the response variable showing stronger deviations from normality than did the nonnormal distributions included in some previous simulation studies.
We begin our discussion with the case of normal data. Here, as expected, the exact multivariate test (T2) controlled the Type I error rate in all conditions (according to Bradley's liberal criterion). This approach requires N ≥ K, however, so it cannot be used in typical experiments from, for example, psychophysics, where the number of subjects is often smaller than the number of factor levels. With only very few exceptions, the univariate approach with Huynh–Feldt correction for the degrees of freedom controlled the Type I error rate even at N = 3, and can thus be recommended for designs with extremely small sample sizes, given that the response variable is normally distributed. Note that we only analyzed Type I error rates but did not compare the power of the T2 and HF procedures. This is an important task for future research. Previous results for higher sample sizes have suggested that the relative power of the two approaches depends on the combination of N and K and on the deviation from sphericity of the population covariance matrix (Algina & Keselman, 1997). While the PROC MIXED variants fitting a UN model covariance structure or selecting the model covariance structure via information criteria (AIC and BIC) controlled the Type I error rates in most conditions for normal data, they performed less convincingly than the two traditional approaches. Therefore, we do not recommend the use of PROC MIXED unless there are missing data, which should not be the case for controlled experiments. In any case, it can be concluded that several procedures for the analysis of repeated measures data show excellent control of the Type I error rate for normal data, even for extremely small sample sizes.
In contrast, for nonnormally distributed data, our data indicate serious problems with the control of Type I error rates for all analysis procedures, which is compatible with previous studies simulating small sample sizes (see the introduction). This is visualized in Figs. 5 and 6, where most data points are located outside the gray area that indicates robust Type I error rates.
For χ2(2)-distributed data, we observed a marked difference between the spherical (CS) population covariance structure and the nonspherical structures. For the CS population covariance matrix, error rates tended to be conservative, while for the remaining structures, most values were liberal (see Fig. 2). For the nonspherical covariance structures with homogeneous variances, the results were in between those in the two former cases. It would be interesting to study the effects of nonsphericity versus variance heterogeneity in greater detail. For log-normally distributed data, we observed conservative Type I error rates in the majority of conditions, especially at smaller sample sizes. At sample sizes greater than 50, several liberal error rates were observed. For both distributions, not a single procedure showed acceptable control of Type I error rates across a wider range of settings, so that, unfortunately, none of the procedures can be recommended.
It is interesting to compare our results to the effects of nonnormality in a completely randomized (independent-groups) design containing only between-subjects factors. In that case, many studies have shown that for ANOVAs and t tests, departures from normality are not critical (e.g., Glass, Peckham, & Sanders, 1972; Harwell, Rubinstein, Hayes, & Olds, 1992; Lix, Keselman, & Keselman, 1996; Schmider, Ziegler, Danay, Beyer, & Bühner, 2010). For t tests, the Type I errors still meet Bradley's liberal criterion even with extreme deviations from normality, at least if the sample size is higher than about 20, if there are (approximately) equal numbers of observations per group (balanced design) and if two-tailed tests are conducted (Kubinger, Rasch, & Moder, 2009; Sawilowsky & Blair, 1992). Therefore, for between-subjects designs, textbooks do not recommend against the application of the general linear model for nonnormal data (e.g., Maxwell & Delaney, 2004, pp. 111ff). For between-subjects designs, the standard ANOVA is also known to be robust to moderate violations of the homogeneity of variances, unless the sample sizes are not identical between groups (Lix et al., 1996; Maxwell & Delaney, 2004). Our results show that—unfortunately—the relatively high robustness against nonnormality of the general linear model reported for a completely randomized design does not generalize to a repeated measures design.
One of the most surprising findings was that for nonnormal correlated data, some analysis approaches produce liberal or conservative Type I error rates even at samples sizes as high as 100 (see Figs. 5 and 6). Thus, simply increasing the sample size does not guarantee robustness when the data are nonnormal. In a sense, this indicates that the intuition of experimenters from the field of psychophysics is correct: These experiments typically use only a small number of subjects but collect many trials per subject and experimental condition, rather than testing a large number of subjects but collecting only few trials. If many trials are available per subject and condition (e.g., m = 100), then according to the central limit theorem (see Le Cam, 1986, for a historical review) the mean of the observations across trials will be approximately normally distributed, even if the underlying population distribution of the response measure is nonnormal. Textbooks on statistics typically state that the sample mean will be approximately normally distributed if the number of observations is greater than 30 (e.g., Hays, 1988). For the log-normal distribution that we studied, the skewness and kurtosis of the sample mean with m = 30 observations are 1.1 and 6.0, respectively, which indicates a distribution much closer to the normal distribution than the underlying distribution of the response measure. It would be interesting to investigate for which values of skewness and kurtosis the analysis procedures produce robust Type I error rates. Given this information, and some information concerning the distribution of the response measure, it would then be possible to decide how many trials per condition are be required.
Besides averaging across a rather large number of trials per subject and experimental condition in order to make use of the central limit theorem, there are four other potential solutions to the problem of nonrobust Type I error rates for nonnormally distributed data from a repeated measures design.
First, the data values could be transformed prior to conducting the statistical tests. For a log-normal distribution, or for an approximately log-normal distribution such as response times, taking the logarithm of the observed values would be the obvious choice. In our experience, researchers analyzing response times seem reluctant to apply this transformation. One reason might be that the transformation would change the interpretation of interaction effects, which are often important in factorial designs. Several transformations are discussed or recommended in textbooks on statistics, such as, for example, an arcsine-square root transform for proportions (e.g., Maxwell & Delaney, 2004; Winer, Brown, & Michels, 1991). However, simulation studies investigating the effects of transformations for repeated measures analyses are missing, and even for data from completely randomized designs, a debate has concerned whether or not transformations should be used (Games, 1983, 1984; Levine & Dunlap, 1982, 1983). It is also not always the case that researchers will know what distribution their data will exhibit. However, at least for a completely randomized design, it might be possible to select the appropriate transform on the basis of the sample characteristics (Rasmussen, 1989). Again, no corresponding simulation studies seem to exist for repeated measures designs.
Second, the parametric procedures designed for the analysis of normal repeated measures data can be combined with "robust" estimators like trimmed means (Berkovits et al., 2000; Keselman et al., 2000; Wilcox et al., 2000) or M estimators (Wilcox & Keselman, 2003). The existing simulation studies evaluating the Type I error control of this approach do not cover a range of sample sizes, numbers of factor levels, and population covariance structures comparable to the present study, and they also indicate that the use of robust estimators will not in all cases result in sufficient control of the Type I error rate.
Third, statistical software is increasingly available for fitting generalized linear models or nonlinear models that are applicable to the data from repeated measures designs (e.g., Breslow & Clayton, 1993; Davidian & Giltinan, 1998; Hu, Goldberg, Hedeker, Flay, & Pentz, 1998; Jaeger, 2008; Lee & Nelder, 2001; McCullagh & Nelder, 1989; Vonesh & Chinchilli, 1997). If the distribution of the response variable is known from previous research or can be estimated from the sample, then a model designed for the specific distribution can be used. For instance, for log-normal data, software including SAS PROC NLMIXED is available for fitting nonlinear mixed-effects models (e.g., Davidian & Giltinan, 2003; Jin, Hein, Deddens, & Hines, 2011; Littell et al., 2006). For binomially distributed data (e.g., error rates), several multiple logistic regression approaches accounting for the correlation structure of the data have been proposed (Hu et al., 1998; Kuss, 2002; Lipsitz, Kim, & Zhao, 1994; Neuhaus, Kalbfleisch, & Hauck, 1991; Pendergast et al., 1996; Spiess & Hamerle, 2000). However, not much is known about how these approaches behave if small sample sizes are combined with high numbers of factor levels and with different types of population covariance matrices (e.g., Austin, 2010). Specifying the correct model can also present a challenge to researchers, due to either the high flexibility or the restrictions of the software.
Finally, nonparametric approaches might be an alternative, such as, for example, the framework based on rank-score tests proposed by Brunner and colleagues (e.g., Brunner, Munzel, & Puri, 1999; Brunner & Puri, 2001). This method is available in SAS PROC MIXED, but it requires N ≥ K and has not been tested extensively in simulation studies.
Therefore, it has to be concluded that while there are several promising alternatives to parametric approaches assuming normality, it remains for future research to show whether these procedures will indeed solve the problems with nonrobust Type I error rates for nonnormal data in a repeated measures design that we have identified in our study.
Summary and recommendations
In this study, we obtained empirical Type I error rates for several procedures available for the analysis of data from a repeated measures design. We simulated a design with a single within-subjects factor and no grouping factors. Our focus was on specific designs often encountered in experimental psychology and the neurosciences, where high numbers of factor levels of the within-subjects factor(s) are studied in small samples, and several important response measures, such as response times, are nonnormally distributed.
Several analysis approaches showed good control of the Type I error rate for normal data, while none of the procedures was found to be robust against nonnormality.
In the following summary, we propose some guidelines for selecting an analysis procedure for completely within-subjects designs.
Figures 4–6 visualize the control of the Type I error rate for the eight analysis procedures that we studied. Each data point shows the number of nonrobust Type I error rates at a nominal α level of .05, according to Bradley's (1978) liberal criterion, which considers an empirical Type I error rate \( \widehat{\alpha } \) as being acceptable if it is contained within the interval \( .025\leqslant \widehat{\alpha}\leqslant .075 \). While researchers often have some idea about whether their response measure is approximately normally or nonnormally distributed, it is rather unlikely that reliable information about the population variance–covariance structure will be available. Therefore, we studied five quite different population variance–covariance structures. Each data point in Figs. 4–6 shows the number of nonrobust Type I error rates across the five population covariance structures. If this number is 0 (i.e., the data point is located within the gray area), our study indicates that for a given number of factor levels (K) and sample size (N), the procedure adequately controls the Type I error rate across a wide range of possible population covariance structures.
Our simulation results show that it is primarily important to distinguish between the cases of normally and nonnormally distributed response measures.
For normal data, Fig. 4 shows that the multivariate approach (T2) controls the Type I error rate. However, this procedures requires N ≥ K, and therefore cannot be used if a small sample is studied under a high number of factor levels. For N < K, the univariate approach with Huynh–Feldt correction for the degrees of freedom can be recommended, which produced only very few nonrobust error rates. PROC MIXED unconditionally fitting a univariate covariance structure performed identically to the multivariate approach for N > K + 1. It could therefore be used in the case of missing data (MCAR or MAR), but missing data are typically not a problem in controlled laboratory experiments. The remaining procedures (GG and PROC MIXED with selection of the model covariance structure via information criteria) did not show acceptable control of Type I error rates across conditions. Note that PMCC is only of theoretical interest here, because the population covariance structure will typically be unknown. Finally, our study only obtained Type I error rates. Previous studies have shown that in terms of statistical power, the recommended procedures HF and T2 can differ quite substantially, depending on the sample size, K, and the nonsphericity of the covariance matrix. Simple rules for deciding between the two procedures can be found on page 215 in Algina and Keselman (1997), albeit only for sample sizes greater than K + 4.
For nonnormally distributed response measures, Figs. 5 and 6 show that none of the procedures that we studied was able to control the Type I error rate across a larger range of conditions. Notably, even at a sample size of 50 or 100, liberal or conservative error rates were observed in a substantial number of cases. Although several potential solutions exist to the problem of nonnormal data in a repeated measures design (see the Discussion section), it is currently unclear which of these alternative procedures can be recommended. Therefore, our recommendation for researchers studying a nonnormal response variable is simply to collect a higher number of trials per subject and experimental condition (i.e., level of the within-subjects factor), and then for each experimental condition and subject compute the mean across trials. According to the central limit theorem, with increasing numbers of trials, this sample mean will approach a normal distribution, even if the underlying response measure strongly deviates from normality. Therefore, if a sufficient number of trials is collected per subject and experimental condition, the dependent variable in the repeated measures ANOVA can always be considered normally distributed, and therefore either the HF or the T2 procedure can be used. The necessary number of trials per condition will of course depend on the deviation from normality of the "raw" response measure. Additional research is necessary for providing exact guidelines concerning the minimum number of trials for which robust Type I error rates can be obtained with nonnormal response measures. At present, the number of 30 trials typically recommended in statistics textbooks (Hays, 1988) could be used as a lower limit.
Several other available procedures are targeted at analyzing the data from unbalanced designs (cf. Keselman et al., 2001; Vallejo & Livacic-Rojas, 2005; Vallejo Seco, Izquierdo, Garcia, & Diez, 2006). As we studied a completely within-subjects design without grouping factors, we did not include these procedures.
Note that early texts referred to the univariate approach as a "mixed model," because it contains a fixed effect of the within-subjects factor and a random effect of the subject (e.g., Geisser & Greenhouse, 1958).
The probability density function for the log-normal distribution is \( P(x)= {\frac{1}{{\sigma \sqrt{{2\pi }}x}}} \; {e^{{{{{-{{{\left( {\ln\,x-\mu } \right)}}^2}}} \left/ {{\left( {2{\sigma^2}} \right)}} \right.}}}} \)
Several articles have erroneously claimed that a log-normal distribution with μ = 0 and σ = 0.5 has a kurtosis of 5.90 rather than 8.89. This error can be traced back to Keselman, Algina, Kowalchuk, and Wolfinger (1999b, p. 71).
Kenward and Roger (2009) recently suggested an improved variant of their procedure. However, because we are not aware of a statistical package incorporating the improved algorithm, the results reported in this article apply to the original procedure by Kenward and Roger (1997), which was used in SAS Version 9.2.
Author Note
T.F. is now at the German Federal Statistical Office. This work was supported by a grant from Johannes Gutenberg-Universität Mainz ("Förderstufe I") to the first author. We are grateful to Marieke Hölle for helping with the figure and table preparation.


text:
Items in Cart:
Sell Your Property Today
Reach hundreds of thousand of potential buyers who visit bidorbuy every month. List your property for sale now!
Sell Now
Signs You Are Eating Too Much Sugar
Do you often feel sluggish and tired throughout the day? Do you regularly crave sugar, even after having a meal?
Are You a Sugar Addict?
Choosing The Right Braai
We have everything you need to know about becoming the ultimate braai aficionado. If you are in the market for a new braai, we have weighed up the pros and cons of kettle, flat top and gas braais to make the search a lot easier.
The Right Braai for You
Should You Make The Switch?
As of right now, the Nintendo Switch is available for purchase in South Africa - but is this the answer to the powerhouse consoles on offer from Sony and Microsoft?
The Nintendo Switch
Open Length: 160mm
Pocket Knife Folding Knife Stainless Steel Knife Outdoor Knife

subset: commoncrawl


text:
Click on a thumbnail to go to Google Books.
Drought (edition 2011)
by Pam Bachorz
Work details
Drought by Pam Bachorz
No current Talk conversations about this book.
Felt a little preachy. And I'm still not sure why they were there in the first place.
Ruby is in no way your average 17 year old girl. This is perhaps most obvious in the fact that she is actually 200. She and her mother and the rest of the Congregation have all lived for centuries. Ruby's father, Otto, led them and gave them long life through the power of his blood, but then he left. In his absence, they have been forced into servitude, made to gather Water so that others may share in the magic. They lied though about what turns water into Water; Darwin West, the man who enslaves them, does not know that there is Water only because Ruby drops some of her blood into it. Like her father's, Ruby's blood has amazing healing powers. Only they may not be strong enough to rescue the Congregation from Darwin West. And Ruby is starting to suspect that her father may never return to save them.
Drought is a really strange book. Partly because I thought it was going to be a dystopia, which it did seem to be for a while, but then it turned out to be something else. I would not have picked up on the point of the story at all, if not for an observation by one of the characters; I would discuss it, because it's an interesting theme, but the reveal is a part of what makes the story interesting, so I won't. Suffice it to say that it is covering fairly new ground in teen lit.
The story is fairly slow moving without much real action; the characters spend much of the book gathering water and getting beatings. While a whipping may sound like action, it felt more like drudgery, because it happened to the Congregants most every day. Just because the book is slow doesn't mean it was hard to get through; it actually flowed along at a slow pace, like a sluggish river.
The characters were a bit of a weak point. Ruby was definitely most likable, which is good since you see from her perspective. I couldn't get caught up in her romance or her worries about the Congregation though, because I did not see why either one really deserved such devotion.
Religious themes are hugely important to the plot of this novel. The Congregation is so named because they all worship together. They worship Otto, their savior who promised to return (sound somewhat familiar?). They believe in him because of the miracles he performed (now?). A couple of clever things are done with this, such as the scene where Ruby and her love interest both insult one another's beliefs: she his in the Holy Trinity and he hers in Otto.
Overall, I really do not know how I feel about this book. It was just so strange! For that reason alone, I think I am happy to have read it, because it is definitely walking some fresh ground. My plans to read Candor, Bachorz' other book have not changed; this book was well-written enough and interesting enough to earn her another try.
The first chapter or two were enticing, so I was looking forward to uncovering this story. Unfortunately absolutely nothing new is revealed after the first 50 pages or so and the ending is drawn out and insipid. What a shame.
For 200 years, Ruby and the Congregation have been controlled by Darwin, a ruthless Overseer who forces them to collect water from the leaves of plants every morning and harvest it in great cisterns. This magical water stops aging, but there's something Darwin doesn't know - it's not the water itself, but the blood in the water that makes it special. Ruby's blood. Conflicted between leaving her family and all she's ever known or escaping and making her way in a new world, Ruby must choose between the two: her family, and Ford, the only man she's ever loved, and an Overseer to boot.
Though skeptical about this book at first, I was drawn in by the first few pages. The book seemed to take a magical hold over me, compelling me to read it in every spare second. I finished in about three and a half hours, and let me tell you, it was three and a half hours well-spent. The description was amazing, and the way the plot unfolded was truly astounding.
I loved this book and would recommend it to anyone - especially those who enjoy romances and slightly dystopian themes. :)
I was rather interested in this book based on the plot alone. Yes, it has bad reviews on Amazon and Goodreads, but it does not live up to the promise the plot gives. In fact, it falls flat thanks to 1.) stupid characters, 2.) implausible plot developments, and 3.) an ending that wrapped up 400 pages worth of action in 10 minutes.
Ruby has lived her entire life living practically as a slave, gathering Water (dew collected daily from plants mixed with her blood which gives extended life and healing powers). She's 200 years old but looks and acts like she's 16. One of the Overseers, Ford, is the designated love interest of this tale. He's handsome, 18, and working as a slave driver to provide for his mother's cancer treatment. Ruby's mother claims to be looking out for the interests of the community while serving as a reverend for a Congregation that worships Otto, the original man whose blood gave life before he up and left the community to hide in the woods to continue on, but only after impregnating Ruby's mother. The main villain is a man named Darwin West, who is in love with Ruby's mother and, scorned, follows the Congregation into the woods and enslaves them in order to procure and sell the Water to the Visitor. The Congregation has lived there for 200 years and still lives as if it is 1810, waiting for Otto while Darwin manages to live on as a member of the outside community, recruiting Overseers to serve as guards from convicts and the desperate unemployed.
Some tropes employed in this book include Instalove (Ford falls in love with Ruby at first sight, and she pretty much does as well). More importantly than that, everyone in the Congregation is apparently dumb. They make Ruby their leader and then when they realize she is in love with an Overseer they cast her out - by the end, Ruby's mother has become a secondary villain, even trying to kill Ford and tells Ruby she is an embarrassment and not worth love because she "betrayed the community" by trying to save them and not wait for Otto (who is now a Jesus-like figure).
The ending seems like the author realized she needed to wrap it up and after 300 pages of harvesting Water, talking about Ruby's power, love, and how Darwin is evil, the Visitor shows up. I still don't get the Visitor - he just showed up and then left but managed to do so much in the process that concluded the novel. On top of this, Ford is also guilty of religious insensitivity, saying he can't love Ruby because worshiping Otto is heresy and a sin. Ruby, meanwhile, is guilty of "I love him, but I can't love him" in every chapter. We get it, we don't need to be reminded every five pages.
In the end, this book had promise, but just fell flat. Ruby was dumb, her mother was a b*tch (all she had to do was marry Darwin - it was obviously she cared about him, but she was saving herself for Jesus - I mean Otto), and Ruby and Ford fell in love despite them having nothing in common besides googly eyes. Oh, and there were several obvious editing problems (they're versus their came up at least twice, and punctuation usage was off). A good premise does not make a good book.
References to this work on external resources.
Wikipedia in English
No descriptions found.
Ruby's blood holds the secret to the Water that keeps her and her fellow Congregants alive and enriches Darwin West, who has enslaved them for two centuries, but when her romance with an Overseer, Ford, brings her freedom in the modern world, she faces a terrible choice.… (more)
(summary from another edition)
Amazon Kindle (0 editions)
Audible (0 editions)
CD Audiobook (0 editions)
Project Gutenberg (0 editions)
Google Books — Loading...
Popular covers
Is this you?
Become a LibraryThing Author.


text: Skate Dance Dream has announced two new tour dates & cities! The first stop will be Pittsburgh, PA on July 27, 2013 and followed by Dimondale, MI on November 9, 2013. The Pittsburgh performance will be headlined by Courtney Hicks, reigning U.S. pewter medalist; Ashley Cain, U.S. Junior silver medalist; and, Comfort Fedoke, an All-Star Dancer & Top Eight Finalist from the Fox hit show "So You Think You Can Dance." Joining the Stars will be a cast of top local and regional figure skaters and dancers. For tickets, updates and announcements on the Star casts and more, visit their website at http://www.SkateDanceDream.com


text:
Wellllllllllllllll...I am a new proud owner of a 2012 WRX Limited. Had zero intention of buying one but someone I got bit by the bug. I called the local dealership looking for a STI hatch and he basically told me that its impossible to keep anything like that in stock and they could special order anything I wanted. So we set up a meeting for me to come in so we could discuss everything. I got there and started bull****ting with him about cars and stuff. I currently have a 2010 Genesis Coupe 3.8 also and a 2001 Dodge Ram 1500. So we started talking about the Subby's. He then told me, "I know this isn't what your looking for but trust me, take it for a test ride and I gaurentee you will not be dissapointed. They had one WRX left, three on the lot and two sold just hours before. It had everything I was looking for, minus the STI. Heated seats, sun roof, back seat, nice size trunk, manual tranny, and the list goes on. This car was ordered for someone else but they test drove it and decided they didn't like it. So it only had 17 miles on it. It also came with the SPT Exhaust. So to sum it all up. I test drove it, and couldn't of been happier. We got back started punching numbers, I traded in my Ram and joined the WRX Club. I've had the car for one week today and I just crossed the 500 mile mark. I have a lot of plans in the future for this car, intake, BOV, extra rims for winter tires, debadging, etc. I am also a very passionate photographer so I'm sure I will be posting a lot of my photos and joining in on some wrx talk.
Quick Photo
before and after
Subaru WRX Limited | Flickr - Photo Sharing!
Thanks for taking the time to introduce me


text:
Family: Wife, Lori; daughter, Hailey, 19; and son, Blaine, 15.
Pets: None right now.
Hobby: Sports, plays golf and basketball.
Book last read: "The Advantage," "Who Is This Man?" and on his list, "That's Why I'm Here."
Favorite movie: "Hoosiers."
Favorite TV show: Sports Center.
Favorite restaurant: Jake's and Green Leaf.
Favorite website: Yahoo.
Favorite vacation: North Myrtle Beach.
My pet peeve is: People driving 55 mph in the left lane.
The best record album ever is: Anything by Chris Tomlin.
The ringtone on my cell is: Vibrate.
The talent or superpower I wish I had is: To be able to know what people are thinking.
Least favorite chore: Cleaning the toilet.
If I could change one thing about myself, it would be: To be more patient.
Favorite sports team: Cleveland or Ohio State.
First job: Summer job in the Elms pro shop.
Best advice you ever received: Work ethic from grandparents.
Nobody knows I: Struck out three times in a T-ball all-star game.
Who, living or dead, would you most like to have dinner with: My family because our schedules are so busy.


text:
These treats are delicious and socially responsible – GMO-free and fair-trade. At just 1.5 calories a pop, you don't have to feel guilty about polishing off an entire tin. Guilt-free chocolate, what a novel concept!
Sweetriot Cacao nibs are a great snack for movie nights – at home or in the theater (instead of a 2,000 calorie tub of popcorn). Chocolate also beats PMS and boredom binging for when you need a bit of a chocolate pick-me-up.
Amazon.com or sweetriot.com
April 30, 2012
these are the best. Just enough to satisfy me. great idea!!!!
Receive curated news, tips, and product reviews
Need help or have a question?
We're available M-F 9AM - 5PM ET - send us a note!
Conscious Box member? Click here for support.
© 2017 Bestowed.


text:
View Single Post
View Public Profile
Send a private message to Hropkey
Find More Posts by Hropkey
Join Date: Sep 2008
Posts: 378
Saturday, November 17, 2012 at 06:24PM
They're not equal aggressors; I said that pretty specifically, Israel has more power in this situation.
Also Israel isn't targeting civilians. They're targeting militant leaders, but Gaza is so densely populated and Hamas specifically puts militant leaders amongst civilians because they know the IDF isn't allowed to target civilians. (An explanation but NOT a justification for the deaths.)
Also Israeli actions were a direct result of a lot of the fear and uncertainty that came from the second intifada (again, an explanation but not a justification). Just because Hamas is being supported by Palestinians does not mean they're good. Honestly, it's a Catch-22- they have a terrorist organization in power so Israel commits what I see as human rights violations against them, which gets them justifiably more angry and more willing to commit violent and extremist acts and then Israel puts more restrictions and takes away more human rights... I don't foresee an end unless there's a mediated peace process. I just want a ceasefire in Gaza and for this to be done with.
Mod CG as of 10/18/08
Using: Suave Naturals, L'oreal Vive Pro Nutri Gloss, LA Looks Sports Gel.
"Well not live like this. They will try to bury us with false manifestos, inscribe us in wars against false enemies but well sing songs about dying from loving the wrong cowboy and gospel; our bodies will burn in effigies of promise. I swear."
-Ibi Kaslik


text:
Susan Sassano talks about her son, Jason, and 18p-, a rare syndrome that causes vision, heart and other significant medical issues. As a result of this syndrome, Jason also found it difficult to concentrate at school. He cried a lot. And simple incidents could throw him into a rage. These behaviors ultimately resulted in the loss of his friends, his schooldays and the joys of childhood.
Then, at Crotched Mountain School he began learning skills to reengage with the world.
See what happened next in this video.
Your support today makes hopes and dreams become a reality at Crotched Mountain.


text:
Click to enlarge
The famous exotic porn star Katsuni wants to play with you at present! A precise reproduction of Katsuni's vagina is contained within this Lotus fashion Fleshlight, which is presented in a deluxe canister for simple storage. Made with SuperSkin materials that's delicate to the contact - and thrust. The inner sleeve blossoms into a more comfy inside house which caresses you as you pump away.An inverted node on the within simulates brushing the cervix throughout intercourse, to really help tip you over that edge. Perfect.
Product Information:
- Material: Superskin
- Size (Imperial): life size
- Size (Metric): life size
- Power: None Required
Write Review
Your Name:
Your Review: Note: HTML is not translated!
Rating: Bad Good
Enter the code in the box below:
Your Review: Note: HTML is not translated!
Rating: Bad Good
Enter the code in the box below:
There are no additional images for this product.


text:
Tip: Put a + in front of words that must be included. To search for Monster Hunter and only receive results with both words, use +Monster +Hunter
i liked you costume..
great job
Awesome Eriol! I love that hat <3
wow so cute!!! XD
Wai~ You make such an adorable Eriol <3
nice eriol!!
United States


text:
Jet set to join Geldof's Live 8 party and rock to end poverty
The Sun-Herald
Page Tools
Jet are set to perform at Live 8, 20 years after Live Aid at
Photo: Janie Barrett
Australian band Jet have been asked to join the line-up of Sir Bob Geldof's Live 8 concert event, which will be the biggest live music show the world has seen.
Jet's management has confirmed the band have been approached to perform at the Philadelphia concert for the Live 8 event on July 2, with other concerts being staged simultaneously in London, Paris, Berlin and Rome.
Australian country music star Keith Urban will also play at the US concert in Philadelphia.
Apart from the audiences attending the concerts in each country, the event is expected to be televised to 1.5 billion people. Acts including U2, Coldplay and Sir Elton John have already signed on to perform free of charge at the concerts, which will be staged by Geldof to raise awareness of third-world debt and to promote fair trade.
"Jet have been approached about the US Live 8 and they are working out now whether they can work it into their touring and recording schedule," a spokeswoman from EMI, the band's record label, said.
Rock guru Molly Meldrum, meanwhile, revealed his plan to ask Geldof to stage a full-scale Live 8 concert in Sydney, so acts from the southern hemisphere could be included in the massive global concert event.
"I've been dying to say, 'Bob, pull your finger out of your arse and do something here'," Meldrum said.
As the only Australian who worked closely with Geldof in the lead-up to the original Live Aid concerts in 1985 - when he spearheaded the local "Oz for Africa" broadcast - Meldrum has a strong enough relationship with Geldof to push the notion of an Australian concert.
"It'd be great to have Sydney, Australia, on that list ... or Australia somewhere," Meldrum said. "I'm hoping he'll do it, we've certainly got the acts."
Apart from Jet's likely appearance overseas, Meldrum has revealed that if Kylie Minogue wasn't recovering from breast cancer surgery, she would have been among the artists in the London concert. "She would have definitely been involved," he said.
Australia's three commercial networks and the ABC are locked in a battle to secure the rights to screen Live 8 and will be told this week what the host broadcaster, the BBC, which is co-ordinating the coverage of the 16-hour event has decided.
"We are in serious negotiations for the rights," an ABC spokeswoman said. "It's a marathon production but it's one of those events which will go down in history."
While no one is commenting on how much the event will cost the highest bidder, international sales of the broadcast rights will raise millions towards eradicating world poverty.
The ABC screened the original Live Aid concert in 1985.
Geldof is calling upon world leaders attending the G8 Summit in Scotland on July 6 to increase international aid, promote fair trade and cancel the debt of the world's poorest countries.
Live 8 is being held 20 years after the Live Aid concerts organised by Sir Bob Geldof for famine relief. Musicians including Freddie Mercury (from Queen), Adam Ant and the Beach Boys performed for Live Aid in 1985.
Artists including U2, Madonna and Sting performed at the original event and will appear at Live 8 two decades later.
The July 2 concert is being staged a few days before the G8 summit in Edinburgh, and aims to put pressure on world leaders to reduce poverty and increase fair trade.
Five simultaneous concerts have already been planned for the event; in London, Philadelphia, Berlin, Paris and Rome. The 16-hour broadcast of live music is expected to be watched by 1.5 billion people.
Get the SMH delivered for as little as $3 a week - SAVE 21%


text:
Home >> Furniture Cupboard Closet Dresser Zinc Alloy Pull Handle Grip Silver Tone 15pcs
- Silver Tone Chrome Plated surface, Zinc Alloy door handle knob is a great tool for door open, easy to grip and install.
- Can replace your old or broken pull knob, is suitable for cabinet door, dresser, drawer, cupboard, etc.
|Chrome Plated, Zinc Alloy
|Package Content
|15 x Pull Handle
|Product Name
|Pull Handle
|Main Color
|Silver Tone
|Overall Size
|62 x 19 x 17mm/2.4" x 0.75" x 0.67"(L*W*T)
|Thread Hole Dia
|4mm/ 0.16"
Other Choice (in stock)
- L Women Lady Shapewear Firm Control Tummy Slimmer Full...
- $11.21
- Free Shipping
- Size L Women Body Shaper Tummy Trimmer Waist Control Girdle...
- $5.76
- Free Shipping
- 15pcs Pro Makeup Brushes Set Eyeshadow Eyeliner Lip Brush...
- $11.66
- Free Shipping


text:
Hunter Hogan Photographer
STALIkrisModel Female Vancouver, British Columbia, Canada
My MM URL: http://www.modelmayhem.com/STALIkris
Mayhem # 683214
About Me
Hello my name is Kristalee!
Modeling is something I really enjoy and is something I want to pursue and am very serious about. At this time I am interested in completing TFCD/TFP shoots with differnt photographers to try all kinds of looks. Eventually I would like to complete paid shoots. Absolutely no nudity, so please don't ask... I will always bring an escort to every shoot no matter what!
Please leave a public Aknowledgement if you like my look. Contact me if you desire to work with me.
Thank-you for checking out my portfolio
Have you worked with STALIkris? Add Credits for STALIkris! >>
Credit Notes
Hunter Hogan Photographer
Born8thCre... by MHP Photographer
M M P Photography Photographer
FrizzyCube Photographer
spankydesi... Clothing Designer
Sire Tu Nathaniel Teh Xiae Photographer
Loading Content


text:
Steve McCurry
Gerelateerde onderwerpen verkennen
Photographer Steve McCurry took this photo of an Afghan girl. It appeared on the cover of National Geographic in the mid-80's. He is one of the greats.
Steve Mc Curry - portraits - Taringa! Más
Dust storm - Rajasthan, 1984 | Steve McCurry, photographer.
Steve McCurry India Photography-7
Learn how to Turbo Charge Read and actually remember what you have read. http://youtu.be/LyO3EkP1TdY great for researching authors and students. India | Steve McCurry.
Taken by one of my fave photographers Steve McCurry of #NatGeo
This famous Steve McCurry photograph of a young Afghan girl that adorned the cover of National Geographic magazine and went on to become one of the most famous faces in the world. Here eyes are so haunting.
Peshawar, Pakistan Afghan refugee, Terri Gold.
Che i colori ti saltino addosso o che sia tu a riempirti di colori l'importante è saperne cogliere la ricchezza, saper vedere ogni sfumatura, viverla. foto di Steve McCurry
by Steve McCurry. I dislike how he's rather exclusively known for his photo of Sharbat Gula-The Afghan Girl. It is an extraordinary portrait, however, he has such an extensive portfolio.


text:
Artists /Black ink and book paper are her materials of choice. Meet Denise Kupferschmidt, the New... read more
Artists /"Art is eternal, but life is short," painter Evelyn De Morgan once wrote.... read more
feed me
Artists / Bill Henson
Anna Harrison
Every woman, every man, every object, every heavy breath on a freezing night, is a defiance against the pervasive...
Artists / Sandy Kim
Paul Bui
Sandy Kim is one of NYC's rising art stars. Her photographs flitter on the edge of confronting to...
Artists / Graham Shearer
Anna Harrison
You remember. A fresh faced, bushy-browed Alison Brahe. A tousled haired, big-toothed Kristy Hinze. A steely-eyed...


text:
The use of shared folders as a means of allowing multiple persons to access documents may seem like an efficient approach but in a world where decisions taken minutes too late could cost millions there is a greater need for more efficient solutions.
Enterprise Content Management (ECM) Systems provides a more efficient means of storing, searching and capturing hard and soft copies of your documents. An ECM also allows greater collaboration by allowing the assignment of business rules and workflows to your documents. Unlike traditional shared folders an ECM allows owners of content such as Word files to be able to share their content without the intervention of a Systems Administrator.
Lets say you work in insurance and your business process dictates the following. The collections department collects applications from potential customers and reviews them for completeness. If the collected application is complete route to your risk department, if the application is incomplete reject the application. The risk department will either approve the application and assign an insurance cost or reject the application. The risk departments' decision can be communicated via email to both the customer and the collections department.
By assigning business rules to these applications, if-then-else logic can be applied to these documents. Metadata can be added to documents to store the validity, the decision taken on a application and any other Metadata deemed useful. This Metadata can be used as conditions in your business logic.
At LTGJAMAICA we want to help your company become as efficient as possible.


text:
I thought I would give the nut pattern a try but I'm stuck on the first row. I am completely bewildered by the knit 1, yarn forward, knit 1 into the same stitch instruction. Can anyone explain how on earth I do this?
Many thanks!
I don't know what nut pattern you're talking about but to do k yf k - knit into the stitch and leave it on the needle, yf (really a yarn over) then knit into the stitch again and you have 3 new stitches instead of 1.
Aha that makes sense as in line 4, I have to P3tog and was worried that I would then have too few stitches.
Thanks for such a prompt response, much appreciated!
Is it possible to link to the pattern or an image of it? I'd like to see what you're doing and if you need help again, it might help figure out what you're stuck on.
Hmm I've been trying to attach the photo I have taken from the book I am using but its not working. Will try to type up the pattern today. Thank you for the offer of help.
Don't post the pattern, please because there are problems with copyrights. If you tell us the name of the pattern and the book it's from, we'll be able to find it. Jan has some infromation here on how to posst a photo that should help.
Its called nut pattern and its from a book called The Knitter's Handbook (Craft Library): A Guide to Patterns, Yarns, Stitches & Techniques byEleanor Van Zandt.
Well, that looks like a very handy book! I couldn't find the pattern though. It isn't on Ravelry. Is the pattern actually a nut?