what graphics card features effect NVIDIA NVENC hardware encoding speed?
There are two main features in NVIDIA NVENC encoding:


Video memory if you need to transcode many streams.
NVENC SIP - than better this SIP than better encoding performance
is. It depends on GPU. You can find the best explanation related GPU
and NVENC SIP generation on Wikipedia website. Maxwell Gen 2 is the best at the moment(for 2016).


According to the license limitation: it's possible to run only 2 encoding threads simultaneously on the consumer level NVIDIA cards (any GTX card). It's regulated on the driver level, however it’s possible to remove this limitation. In this case, the value of maximum transcoding threads will depend on  video memory size and video engine utilization.
Video memory size which is necessary for one transcoding stream is various and depends on a video card model. For encoding SD stream on QUADRO K4200(4GB) is necessary 100 MB of video memory, but for encoding the same stream on GTX 980TI(6GB) we need 170MB.

My results in transcoding SD sources in real time are:


QUADRO K4200(4GB): one transcoding thread costs 100 MB and we can run in parallel about 36 threads, but the bottleneck is the video engine utilization. I can run about 30 parallel threads with "-preset hp -vcodec nvenc_264" 
GTX 980TI(6GB): the bottleneck is the video memory. I can run about 32
threads (32*170=5440) in parallel. I did it for my education, of
course. But the video engine results is better then on K4200 in 2.5
times. These 32 threads with "-preset slow -vcodec nvenc_264 -vf
yadif=0" and video engine utilization are only 80%.
I also tested it on GTX 660(2GB, Maxwell Gen 2). It was about 15
parallel threads due to video memory.


My conclusion(for 2016): If you need to transcode no more than 2 threads in parallel, then  GTX 960 is a good variant. Also,  you can save some money to another hardware and look for another video card with Maxwell Gen1. If you go to the hack way, then GTX 960 is a good variant but only with 4GB of video memory.

UPDATE FROM 2018:
Nowadays the situation in this area has changed in  better direction. The card with the best encoder is Tesla V100, but it's too expensive. The best work variant is the video cards based on Pascal CHIPs. If you prefer the hack way then GTX1050TI(4GB), it will be perfect for your budget in 800$. Otherwise, Quadro P2000 doesn’t have license limitations and also very comfortable for the same budget.