Is it possible to take a screenshot in vector format?
Solution 1: Using a vector GUI widget API with PDF backend:

Actually, it is possible. Consider the fact that on screen objects are created by a widget API, and if this widget API is based on vector elements and have appropriate vector output backend, then it can output to vector formats like PDF.

Please look at the similar question, which also contains some working samples:


Can I take a screenshot as vector graphics or at a higher resolution
than native?


This needs Gtk+ widget library and Cairo as its GUI display and PDF generation backend, which both are available for Mac OS X. But as few applications in Mac OS X use Gtk+, this is not a general solution. I wanted to show the possibility of creating vector screenshots. If you are a developer, this may be useful for you.



Solution 2: Creating vector mockups:

For general applications, you can create mockups. Here you can find "Free Yosemite Vector GUI for Illustrator", which can be used to create mockups of your GUI. It includes all main UI Components:

http://codemotionapps.com/yosemite-gui/