Why are cache queries so slow?
I don't know this for a fact but I think it's because the serialised objects contained in the data column are pretty large.

I just checked a production site of mine and the data column for views_data:en contains a serialised string that is 572824 characters long. I kid you not! Imagine loading that on each page request (fortunately that doesn't happen in my case).

The main cache table on the same website is currently at 1.4MB, with just 16 records in it.

On another, less-than-a-week-old dev site, the cache_form table is at 6.3MB, with just 142 records.

I know 1.4MB and 6.3MB aren't large amounts of data as such, but if you take into account the fact that it has to be processed and passed from MySQL to PHP I think that's where you'll find the bottleneck.