Dedupe rule to reject record rather than creating new record
I am not aware of anything in CiviCRM core that would allow you to do so. Depending on your use case you could consider setting up forms that can only be used by authentictaed users.