Are mathematics REUs absolutely limited to undergraduates?
I quote from the NSF REU guideline (emphasis mine):


  Eligible Student Participants: Undergraduate student participants
  supported with NSF funds in either REU Supplements or REU Sites must
  be U.S. citizens, U.S. nationals, or permanent residents of the United
  States. An undergraduate student is a student who is enrolled in a
  degree program (part-time or full-time) leading to a baccalaureate or
  associate degree. Students who are transferring from one college or
  university to another and are enrolled at neither institution during
  the intervening summer may participate. High school graduates who have
  been accepted at an undergraduate institution but who have not yet
  started their undergraduate study are also eligible to participate.
  Students who have received their bachelor's degrees and are no longer
  enrolled as undergraduates are generally not eligible to participate.
  For REU Sites, a significant fraction of the student participants
  should come from outside the host institution or organization. Some
  NSF directorates encourage inclusion in the REU program of K-12
  teachers of science, technology, engineering, and mathematics. Please
  contact the appropriate disciplinary program officer for guidance.
  Within the framework of the basic eligibility guidelines outlined
  here, most REU Sites and Supplements further define recruitment and
  selection criteria, based on the nature of the particular research and
  other factors.


So this means that (a) your host will not be able to receive funding for you from the NSF if s/he accepts you into the program (b) on the other hand if you and/or the PI of the grant is able to secure funding otherwise, there's generally no rule saying that a student in your position cannot be involved in research in some way. 

You will need to individually contact the REUs you are interested to find out whether they'd be willing to grant you the leeway.