Why does small cache memory take less time to index
At least two factors:


bigger caches means longer wires, so it takes more time for the information to  travel (remember, at 1Ghz, light travel 30cm per cycle, and electrical signals are slower), both from the address to the memory cells, and from the memory cells to the place where the data is used.
if your cache is fully associative, that means that you have to address more comparators, increasing the load on the address lines and thus reducing the speed more than just what the wire length increase would hint.