SPI modes of operation: Word size
It is very likely that this time is due to the micro-controller having to put the next byte in memory, either on a software or hardware level.

You can check the code and see if there is any bottleneck at this level, if for instance you send the data byte by byte with an interrupt.

Some micro-controllers have DMA buffer channels that can speed up transfer.

There are 8, 16 and 32bit micro-controller, depending on your application they can also speed up data transfer, although it is also related to the core frequency.