Event registration with participant approval: missing e-mail notification when rejected
I don't know of any magic that happens if you use the "Rejected" status.  I think it's probably no different than if you created "Flaky" or "In absentia" as statuses: most participant statuses are just there for you to keep track of attendees and cause no special behavior.

Consequently, CiviCRM won't really do anything different whether you are saying someone is "rejected" versus "attended" or "no show" or whatever.  In particular, there is no "rejection notice" email template in the System Workflow Templates that would be triggered.  (In contrast, there is a "cancellation notice" for canceled registrations.)

Unfortunately, that means that the default registration email is what would automatically go out, just as if you had approved the registration.

I would suggest that you leave the box unchecked and manually send an email to the rejected applicant(s) explaining the situation.  I suspect that the reason there is no "rejection notice" template is that few organizations would want to send out generic, antiseptic notices to the effect of

Your registration for the Rain-forest Cup has been rejected.


Sending an email manually would let you be cordial and detailed.  You could send the same message to everyone you cancel (all at once, and they'll receive individual messages), or you could be as specific as you'd like for each rejected participant.

As for the different templates, here goes:


Events - Pending Registration Expiration Notice: if you run the scheduled job to process pending registrations, and the pending participant expiration field is set on an event, this will let people know they've been dropped by the scheduled job.
Events - Receipt only: this is what you get if the site has the event registration in cart mode and you register for one or more events.
Events - Registration Cancellation Notice: automatically sent when the status is "canceled" and (I'm pretty sure) the event is in the future.
Events - Registration Confirmation and Receipt (off-line): generated if you register someone as an admin and check the receipt box.
Events - Registration Confirmation and Receipt (on-line): generated by regular online event registration.
Events - Registration Confirmation Invite: if you're on the waitlist, and a spot opens up, when the scheduled job to update participant statuses automatically bumps you to "pending from waitlist" status, this lets you know you're now allowed to go register.