What does the 呢 in "你听什么呢？" mean?
呢 is a final particle. It denotes interrogative form in your example sentence. (it has other usages in other context)

You can just use a question mark e.g. "你在听什么？" (what are you listening ) to mark the sentence as a question 

"呢" in "你在听什么呢？” asks the question of "what" are you listening to

"嗎" in "你在听什么嗎？” asks the question of "are" you listening to something