How can I make a video such that the quality degrades if someone tries to record it using screen recording software?
Rovi does try to add an element of security but the simple truth is that once it is being played, there is actually nothing you can do to prevent it being copied. Unless you control everything (including the screen) then an attacker can just wait until the encryption has been removed and copy the output. So this could be in memory, on screen, by intercepting HDMI etc.

This has been discussed at great length on security.stackexchange.com as it is one of the unsolved issues in security.