What is the dominant factor for evaluating performance for TTS & ASR?
I would expect TTS and ASR to be very memory-intensive as they process input (which needs to be stored and transformed) and generate output.  So slow memory would probably be a limiting factor with a fast CPU.  There should also be a little I/O -- loading voice samples for comparison and playback -- but I would not expect it to be as much of an issue.  I can't get any more specific since I don't know how the default engine works (I suspect few people outside Google do).

You might get a better answer on StackOverflow, since this is more of a question about the specifics of an algorithm and not really about Android use.