Dock indication about different application instances
There's a Multi-Touch gesture that can show you all the instances of one app. Swiping downwards with either three or four fingers (configurable in System Preferences > Trackpad > More Gestures) opens "App Exposé", a handy birds-eye view of all your open windows in the frontmost app. It's not the same as an indicator in the always-present Dock, I know, but I think this can be one of the best options you'll get without 3rd-party software (assuming such software exists).