Does anyone know why I can't screen cast play mode in BGE?
Here are some possible options:


You use BGEs internal screenshot function, either in python or with logic bricks. Here is a logic brick test setup which quits the game after 120 ticks and makes a screenshot on every frame:

The good thing: You end up with high quality PNGs without compression artifacts. The bad thing: On my laptop (and probably any other non-high end machine), this heavily reduced framerate as I would expect from every software-based screen recorder which tries to compress/dump a massive amount of frames. Edit: The release log of 2.77 says, it already uses another tread. However, my laptop still can't handle Full HD output without losing framerate. 
You buy one of these hardware game capture devices, which will capture the HDMI output in a standalone system. Ask amazon for "game capture HDMI". The good thing: It doesn't eat your machines resources. The bad thing: You may have no control about compression. And it is not for free, obviously. 
Your graphics card supports hardware accelerated screencapture. Modern NVIDIA cards have shadowplay.
You try one of these broadcasting softwares and stream to another machine, where you record your gameplay. With this solution you outsource the file operations to another machine.
On recent Win10 OS, you can use Win-Key + G and follow the User interface. Microsoft has built a screencapture feature which dumps recordings in the users "videos" folder.


For work, I stuck with solution 2 because my laptop could not handle the others.