What does the literature say about why women aren't attracted to nice guys?
This isn’t a literature answer but a qualitative clarification. “Nice guy” is a conflation between two distinctly-separate behavioral patterns:

1) Passively begging for approval points via virtue-signalling in order to become her most favored suitor. This signals weakness, insecurity and/or cowardice because the male lacks thr courage to take action, risk rejection or be vulnerable by making his intentions clear.

2) Being kind, generous or gallant without expectation or necessity, as a king banking social capital and goodwill with his subjects as a demonstration of his abundant power and good character to lead and provide.

Clearly, these are two vastly-different approaches that should be considered entirely different strategies. One is genuine, while the other is wishful and deceptive.

Furthermore, considerations of how much sexual (genetic) vs. social/status (partner) value is demonstrated in order to elicit various respect and/or attraction responses. For example, males with mostly sexual value but little partner value will quickly gain mating opportunities, whereas low sexual value but high partner value males will be expected to invest much more time, money and energy before mating opportunities and be held to higher behavioral standards than the former group. Social value only demonstrated males without enough sexual or status value will be friend-zoned. Too high of status men also risk friend-zoning to protect her ego, ie auto-rejection. High sexual value men may also be initially pseudo-rejected, but it’s unlikely that a female would pass on an opportunity to gain superior biological father genes considering the time and energy commitment risks involved in child-rearing.