In non-osteoporotic postmenopausal women with breast cancer, aromatase inhibitors (AIs) negatively affected bone mineral density (BMD), lumbar spine trabecular bone score (TBS) as a bone microarchitecture index, and hip geometry as a bone macroarchitecture index. AIs increase the risk of fracture in patients with breast cancer. Therefore, we aimed to evaluate the long-term skeletal effects of AIs in postmenopausal women with primary breast cancer. We performed a retrospective longitudinal observational study in non-osteoporotic patients with breast cancer who were treated with AIs for a 3 years (T-score >-2.5). Patients with previous anti-osteoporosis treatment or those who were given bisphosphonate during AI treatment were excluded from the analysis. We serially assessed BMD, lumbar spine TBS, and hip geometry using dual-energy X-ray absorptiometry. BMD significantly decreased from baseline to 5 years at the lumbar spine (-6.15%), femur neck (-7.12%), and total hip (-6.35%). Lumbar spine TBS also significantly decreased from baseline to 5 years (-2.12%); this change remained significant after adjusting for lumbar spine BMD. The annual loss of lumbar spine BMD and TBS slowed after 3 and 1 year of treatment, respectively, although there was a relatively constant loss of BMD at the femur neck and total hip for up to 4 years. The cross-sectional area, cross-sectional moment of inertia, minimal neck width, femur strength index, and section modulus significantly decreased, although the buckling ratio increased over the treatment period (all P < 0.001); these changes were independent of total hip BMD. Long-term adjuvant AI treatment negatively influenced bone quality in addition to BMD in patients with breast cancer. This study suggests that early monitoring and management are needed in non-osteoporotic patients with breast cancer who are starting AIs.