What Some Research Shows About Stress and Low T. Over the years, some studies have suggested that the stress hormone cortisol and the sex hormone testosterone work against each other. This has led to the theory that stress may cause infertility and loss of libido in men by inhibiting testosterone.