Why do hot water baths affect only males' fertility? Why not females'?
Well, the answer by @An J has already explained the prima facie of your question which is: sperms need lower temperature for development. But I will be explaining in my answer: Why do the sperms actually need a lower temperature for their development?

This is a debatable subject, but there have been several models or hypothesis about this. I found this literature on it, and it summarises the thing excellently. Hope it helps:


  Throughout the Cenozoic, the fitness benefits of the scrotum in placental mammals presumably outweighed the fitness costs through damage, yet a definitive hypothesis for its evolution remains elusive. Here, I present an hypothesis (Endothermic Pulses Hypothesis) which argues that the evolution of the scrotum was driven by Cenozoic pulses in endothermy, that is, increases in normothermic body temperature...
  
  The model argues that stabilizing selection maintained an optimum temperature for spermatogenesis and sperm storage throughout the Cenozoic at the lower plesiomorphic levels of body temperature that prevailed in ancestral mammals for at least 163 million years. Evolutionary stasis may have been driven by reduced rates of germ‐cell mutations at lower body temperatures...
  
  The fitness advantages of an optimum temperature of spermatogenesis outweighed the potential costs of testes externalization and paved the way for the evolution of the scrotum. The scrotum evolved within several hundred thousand years of the K‐Pg extinction, probably associated initially with the evolution of cursoriality, and arguably facilitated mid‐ and late Cenozoic metabolic adaptations to factors such as climate, flight in bats and sociality in primates.


About the function of the scrotum from the article


  The scrotum is a sac‐like thermoregulatory structure that houses the testes at a temperature lower than the core body temperature (Tb; Moore, 1926; Wislocki, 1933; Ruibal, 1957; Setchell, 1998). The cremasteric muscles in the scrotum contract and relax to draw the testes closer to or allow them to dangle further away from the body in order to maintain the testes at an ‘optimal’ temperature for spermatogenesis (sensu Moore, 1926; Tsperm) and sperm storage of 34–35 °C (Setchell, 1998; Gallup et al., 2009; Mawyer et al., 2012). In humans, the scrotal temperature is maintained about 2.7 °C lower than the Tb (Momen et al., 2010). Notwithstanding the relative fitness benefits of a cooled epididymis vs. the cooled testis, four nonmutually exclusive hypotheses currently dominate explanations for the evolution of the scrotum: the Cool Spermatogenesis Hypothesis (Moore, 1926), the Galloping Hypothesis (Frey, 1991), the Mutation Hypothesis (Short, 1997) and the Activation Hypothesis (Gallup et al., 2009).


Cool Spermatogenesis Hypothesis


  ...argued that Tsperm and the maintenance of sperm viability during storage in the epididymis is about 34–35&nbsp;°C (Moore, 1926; Appell et al., 1977). Scrotal temperatures that approach those of the core Tbcompromise fertility (Moore, 1926; Bedford, 1978b, 2004; Setchell, 1998), particularly if evaporative cooling of the scrotum is impaired (Momen et al., 2010).


The Galloping Hypothesis


  ... (Frey, 1991)proposes a trade‐off between testes vulnerability (fitness cost) and the avoidance of strong intra‐abdominal pressure fluctuations during galloping which impair spermatogenesis in abdominal testes (fitness benefit).


The Mutation Hypothesis


  ... maintains that the testis is a ‘hot spot’ for germ‐cell mutations and that the lower temperatures of the scrotum reduce the rates of mutation on the Y chromosome through mutagenic metabolites (Short, 1997).


The Activation Hypothesis


  ... storage of sperm at a lowered temperature ensures that they undergo ‘thermal shock’ during ejaculation into the higher temperatures of the female, which increases their motility and hence the probability of a successful insemination (Gallup et al., 2009).