While the contributing role of testosterone to bone health is rather modest compared to other factors such as estradiol levels, male hypogonadism is associated with low bone mass and fragility fractures. Along with stimulating physical puberty by achieving virilization and a normal muscle mass and improving psychosocial wellbeing, the goals of testosterone replacement therapy in male hypogonadism also include attainment of age-specific bone mineral density. We report on a 37-year-old man who presented with multiple vertebral compression fractures several years following termination of testosterone replacement therapy for presumed constitutional delay in growth and puberty. Here, we discuss the management of congenital hypogonadotropic hypogonadism with hyposmia (Kallmann syndrome), with which the patient was ultimately diagnosed, the role of androgens in the acquisition of bone mass during puberty and its maintenance thereafter, and outline specific management strategies for patients with hypogonadism and high risk for fragility fractures.