Is there a way to improve h264_nvenc output quality?
h264_nvenc is able to encode in constant quality, at least from Pascal onwards. I had to do quite some digging around the internet to figure out how to properly use it, but finally found a working formula.
A command like below will record in constant quality, and the CQ value 26 will produce about the same but slightly better quality, measured with VMAF, as a libx264 encode with the following options: -vcodec libx264 -level 4.1 -profile:v high -tune film -preset medium -crf 23. The H264_Nvenc bitrate was about 19% higher than the libx264 encode, but it had a slightly better VMAF score. I made these tests on some 1080p60 game recording, but also on some 1080p movie trailers.
ffmpeg -i input.mkv -c:v h264_nvenc -preset hq -profile:v high -tune hq -2pass 0 -rc-lookahead 8 -bf 2 -rc vbr_hq -cq 26 -b:v 0 -maxrate 300M -bufsize 600M output.mkv

Important: Unless you set the Max rate and buffer size via -maxrate 300M -bufsize 600M, the CQ mode doesn't behave accordingly and lowering its value won't yield better results.
The Constant Quality mode is activated by specifying Variable Bitrate rate control together with a CQ parameter, a bitrate set to zero, and the above two parameters set to some large value which won't limit the output. Don't be afraid of those values, as bitrate will be quite good.