Do LOR writers need to be someone who's taught you a class?
Whether they taught you a class or not is irrelevant. What you need is people who can confidently and honestly predict your future success. This implies that having worked with someone in a research environment is probably more valuable than having them as an instructor.
In fact, given that many places now have very large classes and student interacting mostly with TAs, instructors may not be nearly as valuable as they once were.
So, no, it won't hamper you provided that your letter writers can make that prediction honestly and enthusiastically.