Understanding USB processing and polling delay in USB host controller
The datasheet of the CP210x mentions large-ish FIFOs on both transmit and receive side.

I strongly suspect that they also optimize usb (and host cpu) utilisation by "collecting" uart data in larger USB packets. 

This would mean that data is hold back until there are either 64 byte in the FIFO or the receiver was not active for a few bit times.

Note that bulk 64 byte packets are special: They don't finish a transaction in the host (unless a buffer was too small). 

You can actuall see a "high dip" for the 64 byte transactions, as these could be executed as one 64 byte packet followed by a zero byte packet.

Side note: It might be worthwhile to re-check timings with an USB 2.0 high speed hub between the PC host and the CP210x. This configuration could use the 0.25ms 0.125ms mircoframe timing available in high speed.