Redundancy of letters of recommendation for scientific and mathematical majors
Most universities want some independent advice about candidates. This should come from people who know the candidate and can attest to their suitability and likely successful outcome. It is hard to gauge that with material provided only by the candidate, even in an interview. 

And there is more to graduate school (or a job, for that matter) than raw competence. There is a social aspect and so a desire to accept people who will get along well in the program or job. (That is why I asked the question in the comment, actually.)

But it is what it is, so in your case, you just need to live with it. I'm surprised that few teachers know you, actually. Think about others who can help. But I would keep trying with the three that you mention in the comment. Let them know of your need and the difficulty of finding others to help.