Better way to code download speed calcuation
Your summary complaint is that you're measuring short term xfer rate from system buffers to user space, rather than long term xfer rate from distant server to local network interface.

After a few roundtrips, TCP opens the congestion window and can have many KiB of data in flight. I don't know what your buffer.length is, but you make it sound like it is short enough that you're not blocking on read. That is, ns.Read() finds quite a lot of data buffered at the OS level, and returns a buffer's worth of it at once, so that your timing represents local CPU / memory bandwidth rather than network bandwidth.

The simplest remedy is to record initial starting time and don't touch it, report cumulative bytes read over cumulative elapsed time.

If your residential ISP does traffic shaping, and you want the display to reflect the fact that the first 30 seconds goes at 3 Mbps but longer downloads go at 1 Mbps, then keep a circular buffer of the last dozen or so timestamps, and use that to report on recent progress. Old measurements will age out, so after a while they will no longer affect the display.