Are there ethical guidelines prohibiting penalties against potential research subjects who have not yet provided consent?
From the US Code of Federal Regulations (45 CFR 46), in the section on seeking consent:


  An investigator shall seek such consent only under circumstances that provide the prospective subject or the representative sufficient opportunity to consider whether or not to participate and that minimize the possibility of coercion or undue influence.


Note the use of the phrase "prospective subject" - this clause refers to the investigator's conduct towards potential subjects who have not yet given consent. Presumably a penalty or threat of a penalty would constitute "coercion or undue influence."

Specifically regarding research participation for credit, the APA Code of Conduct Section 8.04 (a different section of which was mentioned by Stephan Kolassa in his answer) says


  (a) When psychologists conduct research with clients/patients, students or subordinates as participants, psychologists take steps to protect the prospective participants from adverse consequences of declining or withdrawing from participation.
  
  (b) When research participation is a course requirement or an opportunity for extra credit, the prospective participant is given the choice of equitable alternative activities.