Middle-age stress and working account for some of the psychological causes of impotence in older men. Physical causes for impotence in older men include: 1  diabetes. 2  thyroid problems. 3  kidney issues. 4  blood vessel damage. 5  nerve damage. 6  high blood pressure.