Mechanism of Screenshot
Sounds like you've been barking up the wrong tree for a little while. A screen buffer is a generic programming concept which does not describe the situation in macos well. Hardware overlay will likely only affect what appears on the monitor, not what appears in a screenshot. And Netflix's method is no more interesting that DVD Player's method.

Fortunately, this question has been asked and answered on the sister Stack Overflow site. Basically there is a private API that allows this, but being private you're best to talk directly to Apple about it via a DTS request.