Does being a poorly-performing undergraduate researcher mean that I have little chance of being successful in graduate school?
To address the titular question, no, being a "poor undergraduate researcher" does not mean you are not cut out for graduate school.  First, many people who go to grad school in math have no research experience in advance.  Second, having one bad research experience does not mean you are a poor researcher--you're not even in your third year yet.

Having bad research experiences is pretty common, and I know several professional mathematicians who've had bad experiences.  I was part of a summer program and a workshop where I felt I was in over my head and everyone else seemed to know the topic much more than me.  Fortunately they weren't my first research experiences, and they didn't last as long as an REU, so they weren't too discouraging as a result.  These experiences also provided no gauge as to whether or not I could be a successful researcher.

Research takes a lot of time, usually more than you expect (even when you expect it's more than you expect), as well as adequate preparation.  It's also hard to find good problems that are suitable for undergraduates to do in 6-10 weeks.  Part of the problem is, before you work on a project, you don't know how hard it will be or exactly what it will involve.  Therefore lots of REU projects don't amount what was hoped for.  It sounds a least like this project wasn't appropriate for you at that time.  Whether that was due to the supervisor being too ambitious or you not having the expected background, I can't say.  (There could be other reasons for your admission besides your "minority status.")  What I can say is that is easy to get the impression that everyone around you understands or knows more than you just because they're more vocal or they know different things from you.

To address what seems to be your principal concern now, the primary reason to go to grad school in math should be because you want to learn and do more math.  Getting a master's or a PhD in math does not mean you will be a math researcher, and you often don't even need to do any research to get a master's.  It's unfortunate if this experience turned you off from math, but I suggest you wait a little while to see how you feel about math.  Maybe after this experience fades from your memory, you'll get more excited about math again.  Or maybe you'll find something else that excites you, which is good too.

By the way, if you've already taken all the math classes offered at your school by your second year, then possibly your school doesn't have an environment as conducive to developing mathematical maturity as other schools.  If you do want to pursue math further, you might look into transferring, a semester abroad, or taking classes at other nearby schools.