How can I turn off responses to Calendar invites?
This is not possible using Calendar.app, including both macOS (OS X) and iOS.

Your best bet is to submit feedback on Apple's website: https://www.apple.com/feedback/macos.html

If enough of us do it hopefully they'll add the functionality.