Avoiding performance hit from GROUP BY during FULLTEXT search?
You could try looking at tmp_table_size variable and increasing it. The group by must do a file sort using a temporary table. It's probable the temp table exceeds the size allowed and is dumped to disk for the final sort processing (very slow). 

If you do:

SHOW VARIABLES LIKE 'CREATED_TMP%' 


And look at the ratio of Created_tmp_tables vs created_tmp_disk_tables, this should tell you if the value is too small. 

see also: http://dev.mysql.com/doc/refman/5.0/en/internal-temporary-tables.html