What does it mean statistically for there to be significant differences in average IQ scores between racial groups? Charles Murray (in his book, "The Bell Curve") reported that there are real differences between the mean IQ scores of different racial groups. He also points out that IQ variation within said groups is greater than the variation between the groups. I'm struggling to reconcile these two statements.

If by "real differences", he means statistically significant differences between group averages--doesn't that entail the variation is greater between groups than within them?

That is, isn't the statistical test for inter-group mean differences (e.g. ANOVA) based on a comparison of the variances within vs between groups?--(if the inter-group variance is comparatively large enough, this constitutes a statistically significant difference between the groups?) 
If not, then what does it mean to say that the groups are significantly different from eachother?
Statistical significance in this context is determined by:


the underlying size of the group differences, 
the group sample sizes, and
your threshold for statistical significance (traditionally .05).


Even if there were very small group differences, if you had a large sample size (say 1000s of people from each race doing an intelligence test), then you would be highly likely (i.e., high statistical power) to find a significant group difference.

However, Murray's point is not  about statistical significance. I.e., it's not about ruling out the null hypothesis. Observed  differences in intelligence test scores are often fairly large, and this is a heavily studied area with very large sample sizes. So ruling our the null-hypothesis is not a problem. The differences in a given study are statistically signifiant.

Murray's point is about translating the quantitative differences in group means into something that is linguistically meaningful. There are various ways of doing this. For variables that are intrinsically meaningful, we often talk about the raw scales (e.g., you could talk about the gender pay gap in dollars earned per year or you could compare races in height using centimetres or inches).
However, in psychology, we often have scales that lack intrinsic meaning, which leads to the use of standardized measures of effect.

The most common standardized measure of group differences is often labeled Cohen's d. I.e., it is the difference between group means in terms of the standard deviation. So for example, IQ scores have a standard deviation of 15. So if one group has a mean of 110 and another has a mean of 95, then that's a cohen's d of 1.0 (i.e., (110 - 95) / 15 = 1.0). Rules of thumb have been proposed based on an examination of the psychological literature which suggests that .2 is small, .5 is medium, and .8 is large in terms of an effect. 

From memory, some observed differences in intelligence test scores between races are in that 0.8 to 1.2 range (see for example IQ knowns and unknowns). So by conventional rules of thumb for effect size, this would be labelled a large effect or a large group difference.

However, this is not a finding of mere academic interest. This finding can have profound negative consequences for people's lives. Specifically, the fear is that this finding will reinforce negative stereotypes and that people in the group that tends to scores lower on IQ tests will be given fewer opportunities in life (e.g., education, work, immigration, political policy, etc.). And that the finding would support bigotry. 

In particular, there is a real danger that people will rely on a group stereotype rather than judging a person on their merits. For example, it is reasonable to hire a person for a job because they show the most aptitude for the job based on measures of the competence and experience. It is unreasonable to hire a person because they come from a racial group that happens to on average perform better on the job.

Murray uses an alternative index of effect size to say that between group variance is less than within-group variance. Thus, if you do a regression with group as the predictor variable, then only when you explain more than 50% of variance will group explain more variance than there is variance within groups. I think you need a cohen's d of 2.0 to get 50% of variance explained.

Here's a simulation in R:

x &lt;- data.frame(group = "a", dv = rnorm(100000,0,1))
y &lt;- data.frame(group = "b", dv = rnorm(100000,2,1))
df &lt;- rbind(x, y)
fit &lt;- lm(dv ~ group, df)
summary(fit)


Basically, I'm simulating data for two groups and their group means differ by two standard deviations and the result is 50% of variance explained (and 50% variance unexplained; i.e., within group variance):

Coefficients:
              Estimate Std. Error t value Pr(&gt;|t|)    
(Intercept) -0.0007719  0.0031664  -0.244    0.807    
groupb       1.9974176  0.0044780 446.050   &lt;2e-16 ***
Multiple R-squared:  0.4987,    Adjusted R-squared:  0.4987 


In general, a group difference of 2.0 is very large.
Here's another article that talks about different indexes for group differences (Table 1 is particularly interesting).

So, in general, Murray is presumably making the point that even though group differences are fairly large by conventional standards, there are many people in the lower scoring group that score higher than the higher scoring group. So please don't use this finding to stereotype or marginalize people.

Follow-up

You asked in the comments:


  Given the group differences, is it technically accurate to say that a randomly selected individual of a lower scoring group is more likely to have a lower score than a randomly selected individual of a higher scoring group? 


Yes, that is accurate.

To answer that question see Table 1 here.

"Probability that person from experimental group will be higher than person from control, if both chosen at random (=CLES)"


no group differences: 50% chance
0.5 SD difference: 64% chance
1 SD difference (i.e., what has sometimes been found historically for some race IQ differences): 76% chance
2 SD difference: 92% chance