Effect of the health of the male parent at the time of conceiving
It can matter, though at this point we are just starting to understand how much. 

Men can typically have children much later in life than women, but there are costs as time goes on. 

Epigenetics has shown us that what happens to the parents in their lifetime can affect his offspring.  Obesity especially has been shown to affect the offspring, producing a higher risk of diabetes for two generations. 

It is thought that this is not going to affect fathers who develop diabetes at a later age, but primarily those who are obese around 9-12 y.o. when the spermatogenic cells are produced in the testes.

What happens to men as they get older though is that their ability to produce healthy sperm starts to degrade.  They can lose fertility and produce more birth defects typically as early as age 35.  Given this there are probably a lot of habits or events that can adversely affect fertility. 

Lifestyle choices as simple as tight underwear which might block blood flow to the testes is sometimes cited.   

Given all this DNA or cellular damage from chemicals or diet or poor health certainly would also affect male fertility.