Elapsed time in SSMS is consistently higher than SQL Server Execution Time - connection issue?
If you're after a system that takes 0 ms to return and render results, good luck. 

This doesn't mean that SQL Server is 'just slow', though I'm not sure what your expectations for grid rendering are. 

The only way to see what your server is waiting on is to look at its wait stats. If you'd like a free script that does that, head over here and check out sp_BlitzFirst (full disclosure, I contribute to this open source project).

You can run it this way to look at your server's wait stats since it started up:

EXEC sp_BlitzFirst @SinceStartup = 1

Or this way to get a sampling during a window length of your choosing:

EXEC sp_BlitzFirst @Seconds = 30, @ExpertMode = 1

And if you find a system that returns and renders results in 0 ms, drop me a line! I'd love to hear about it.