%pylab inline
from scipy import stats
Ns=np.arange(20,200,4); 
K=10000;
ps=np.zeros((Ns.size,3))
res=np.zeros(4)
cs=np.zeros((Ns.size,8))
i=0
for N in Ns:
    for k in range(K):
        x1=np.zeros(N);x1[N/2:]=1
        x2=np.mod(range(N),2)
        y= 42+x1+x2+x1*x2+np.random.randn(N)*2
        tot=np.square(y-y.mean()).sum()
        
        x=np.ones((N,4))
        x[:,1]=x1*x2
        x[:,2]=x1*(1-x2)
        x[:,3]=(1-x1)*x2
        res[0]=np.linalg.lstsq(x,y)[1]
        
        x=np.ones((N,2))
        x[:,1]=x1
        res[1]=tot-np.linalg.lstsq(x,y)[1]
        
        x[:,1]=x2
        res[2]=tot-np.linalg.lstsq(x,y)[1]
        
        res[3]=tot-res[0]-res[1]-res[2]
        
        mss=res/np.float32(np.array([N-4,1,1,1]))
        F=mss[1:]/mss[0]
        p=1-stats.f.cdf(F,1,N-4)
        p=p<0.05
        ps[i,:]+=np.int32(p)
        cs[i,p[0]*4+p[1]*2+p[2]]+=1
    i+=1
ps/=float(K)
cs/=float(K)
for k in range(ps.shape[1]): plt.plot(Ns/4, ps[:,k])
plt.legend(['A','B','X'],loc=2)
plt.xlabel('N per cell')
plt.ylabel('expected power');

"""
Explanation: I quote myself from the last post:

The number of tests and the probability to obtain at least one significant result increases with the number of variables (plus interactions) included in the Anova. According to Maxwell (2004) this may be a reason for prevalence of underpowered Anova studies. Researchers target some significant result by default, instead of planning sample size that would provide enough power so that all effects can be reliably discovered.

Maxwell (2004, p. 149) writes: 

a researcher who designs a 2 $\times$ 2 study with 10 participants per cell has a 71% chance of obtaining at least
one statistically significant result if the three effects he or she tests all reflect medium effect sizes. Of course, in
reality, some effects will often be smaller and others will be larger, but the general point here is that the probability of
being able to find something statistically significant and thus potentially publishable may be adequate while at the same
time the probability associated with any specific test may be much lower. Thus, from the perspective of a researcher who
aspires to obtain at least one statistically significant result, 10 participants per cell may be sufficient, despite the fact that a methodological evaluation would declare the study to be underpowered because the power for any single hypothesis is only .35. 

What motivates the researcher to keep the N small? Clearly, testing more subjects is costly. But I think that in Anova designs there is additional motivation to keep N small. If we use large N we obtain all main effects and all interactions significant. This is usually not desirable because some of the effects/interactions are not predicted by researcher's theory and non-significant main effect/interaction is taken as an evidence for a lack of this component. Then the researcher needs to find some N that balances between something significant and everything significant. In particular the prediction of significant main effects and non significant interaction is attractive because it is much easier to achieve than other patterns. 
Let's look at the probability of obtaining significant main effects and interaction in Anova. I'm lazy so instead of deriving closed-form results I use simulation. Let's assume 2 $\times$ 2 Anova design where the continuous outcome is given by $y= x_1 + x_2 + x_1 x_2 +\epsilon$ with $\epsilon \sim \mathcal{N}(0,2)$ and $x_1 \in {0,1}$ and $x_2 \in {0,1}$. We give equal weight to all three terms to give them equal start. It is plausible to include all three terms, because with psychological variables everything is correlated (CRUD factor). Let's first show that the interaction requires larger sample size than the main effects.
End of explanation
"""


plt.figure(figsize=(7,6))
for k in [0,1,2,3,6,7]: plt.plot(Ns/4, cs[:,k])
plt.legend(['nothing','X','B','BX','AB','ABX'],loc=2)
plt.xlabel('N per cell')
plt.ylabel('pattern frequency');

"""
Explanation: Now we look at the probability that the various configurations of significant and non-significant results will be obtained.
End of explanation
"""


