Click app in dock to show open instances
While generally most applications allow only for one instance of an application to be running, I believe that you mean the multiple windows open for an application.

By default OS X does not show icon or window previews on the Dock.  However, there are two options for allowing to see the windows:


Mission Control (built-in)
Mission Control gives you the overview on the entire screen with all open application windows grouped together, but there is also an integrated function, called "Application Windows", which just shows you all the windows open for the currently active (foreground application).  You can assign keyboard shortcuts and and trackpad gestures inside System Preferences. Additionally, if you prefer to use the mouse for this feature, you can drag the Mission Control application into the Dock for you to click on.
3rd party application (e.g. HyperDock)
There are some 3rd party applications, which mimic the Aero window preview of Windows 7.  The one I used to have was called HyperDock, which looks almost like the real thing, but there are others as well.