How should you deal with a teacher who is difficult to understand?
I was in a similar position as your classmates in my language course. I'll describe the situation so you can "step into my shoes" and see how it looks from the other side. Hopefully, this will help you understand your situation a bit more.

I'm fluent in spoken Chinese, but being illiterate in would prevent me from passing the equivalency exam, so I took the Intro 101 course with most of the class not being able to speak any Chinese.

Our professor was a grad student originally from China. Her English was acceptable as she was a native Chinese speaker, and she had only come to the US after college.

I had no issues with other students (the class was probably ~15 people) asking for directions to be repeated again more slowly.

Reason? Because, from experience, I know languages are hard to learn. I may be fluent in Chinese, but if I were to take a French class, I'd be in the same boat.

The other students in your class should be able to recognize that you are a new learner. Now, leaving the class might not have been the best response, but you can't change that now. I would recommend not leaving class in the future. That's counter-productive.

Have you tried talking to the other students or asking them for help? I'm sure they're willing to help you learn Japanese as well. Certainly, if they already know the material, they might not even want to be in a rush to go through all the material. 

I guess a final concrete example is in order. 


  If you were an English teacher with a college mastery of English, and teaching a 1st grade English class, would you be annoyed when the 1th graders asked 1st grade English questions?