Why does adding LIMIT to my query make it crawl? Simple query:

select sum(score) total,name,gender,dob,country 
from users join scores on users.id = scores.user_id
where date between '2012-01-01' and '2012-01-31 23:59:59'
group by scores.user_id having sum(score)&gt;=1000 order by sum(score) desc limit 50


So trying to get a list of accumulated scores for Jan 2012, order them by scores descending and paginate them.

Without limit: slowish but OK: searches 69348 rows. (Be nice to work out how to avoid the temporary table but I can't). Explain says: 

1, 'SIMPLE', 'scores', 'range', 'user,date,user+date', 'date', '8', '', 69348, 'Using where; Using temporary; Using filesort'
1, 'SIMPLE', 'users', 'eq_ref', 'PRIMARY', 'PRIMARY', '8', 'scores.user_id', 1, 'Using where'


With limit: it's the same but the rows search are now 1806794 and it takes forever.

It's a partitioned InnoDB if that makes any difference, with all data being on one partition.
Use correct ANSI group by (not the MySQL abomination extension) and see what happens
select sum(score) total,name,gender,dob,country  
from users join scores on users.id = scores.user_id
where date between '2012-01-01' and '2012-01-31 23:59:59'
group by name,gender,dob,country
having sum(score)&gt;=1000
order by sum(score) desc limit 50

Why?
GROUP BY in MySQL implies ORDER BY which causes the filesort here
Changing the query to be more standard may help the optimiser
More on the MySQL abominations extensions:

https://stackoverflow.com/a/6642253/27535
https://stackoverflow.com/a/10894876/27535
https://stackoverflow.com/a/6457144/27535

And from the MySQL Docs on ORDER BY Optimization

In some cases, MySQL cannot use indexes to resolve the ORDER BY, although it still uses indexes to find the rows that match the WHERE clause. These cases include the following:
...
You have different ORDER BY and GROUP BY expressions.