Arduino Serial Queue
It sounds like you figured most of your question out already.

(Someone correct me here if I'm wrong). The Atmega 328p uC, I believe, has an 8 byte hardware buffer for RX serial data. In software, the Arduino core code defines a maximum ring buffer of 64 bytes that acts as a FIFO queue.

So, what does his mean? Let's just focus on the hardware buffer since it's fairly easy. Say you send your data 1,2,...,10 at 10ms intervals, and you read your data at 100ms intervals (for illustration purposes). The problem is that you send 10 bytes of data but only have room for 8 bytes. So, your first two values (1,2) are pushed out of the buffer and lost. This idea also illustrates a FIFO queue, first in first out. 

The same idea can be extended to a ring buffer.

Anyway, like you said, this could explain your problem. If you have 64 bytes, which will obviously be full since your inevitably losing data (20ms read with a 10ms write), you'll be seeing the effects for 64 * 20ms = 1.28s. There's probably some processing delay too which will extend this delay, but it's more than likely minimal if it's efficient code.