How to Determine What's Using Filling GPU's Video Memory?
There is a tool available to check a lot of graphic card parameters: OpenGL Drive Monitors

You may download it at Apple developer. You need a developer account though. Search for "Graphics Tools" and then download "Graphics Tools for Xcode - Xcode 6.1". Open the .dmg and copy "OpenGL Drive Monitors" to your Application folder.

You can't track down graphic memory usage of individual apps though.

Screenshot: