Is getting an oral recommendation from a professor who has known me since childhood a conflict of interest during graduate admissions?
I suspect @Buffy is right, that you did not do anything wrong. However, you did receive an advantage you did not earn because of who your parents are and who do they know. I suspect that that advantage is much more subtle and much more substantive than the oral recommendations (compare your experience with this answer Does politics and perception play a role in higher education? )
In short, you were lucky, and it is not moraly wrong to be lucky. However, it is understandable (not right, but understandable) that people who had to earn their position the hard way require extra evidence before they are willing to accept you as an equal. It is your challenge to deal with this situation gracefully. Demonstrate your competence, as Buffy suggested, but also learn how priviledged you are, and how challenging university can be for those who do not come from a middle class or higher background. A nice place to start is here: https://www.asanet.org/sites/default/files/attach/journals/feb15asrfeature_0.pdf