Why do we have such a wide range of responses to pathogens and carcinogens?
This answer will address only cancer.

There is no such thing as standardized exposure to carcinogens in humans; while it would go a long way to answering your question, it would be unethical.

Also, it is unlikely that two people who do consume identical amounts of a carcinogen have identical environments or identical genetics.

Cancer used to be thought of as a carcinogen-host interaction; expose a (rodent, in my studies) to adequate levels of a carcinogen and cancer will occur. We tried to figure out exactly why. Then it was thought of as a host + environment (This was decades ago.)  The immune system was found to play a role; another level of complexity was added.

Mapping the entire human genome has made us aware of how simplistic these (and most) models were. There are tumor-suppressor genes. There are genes that matabolize carcinogens differently from other genes. There are genes that direct repair of DNA better than other genes. The possibilities are just beginning to surface.

To answer your question more directly,


  Although many studies have focused on carcinogen metabolism and cancer risk, more recent studies are considering DNA repair. 


I would not go so far as to label DNA repair as Immunology. 

I don't know why my father, who had a 120 pack year history of smoking, had absolutely no ill effects (not a trace of COPD, and certainly not cancer, no heart disease, CVD or other) and why my mother had COPD at a 40 pack-year history, but I'm sure it was genetic.


  Sporadic cancers are caused by genen-environmentn interactions rather than a dominant effect by a specific gene, environmental exposure, or gene-environment interaction. New paradigms, where we categorize genes as caretaker or gatekeeper genes, will allow for new hypotheses to be tested and will require advanced methods of analysis.


And, in another study,


  Three meta-analyses report 35% to 50% increases in breast cancer risk for long-term smokers with N-acetyltransferase 2 gene (NAT2) slow acetylation genotypes.


Cancer Risk and Low-Penetrance Susceptibility Genes in Gene-Environment Interactions
Integrative Epidemiology: From Risk Assessment to Outcome Prediction
Active smoking and secondhand smoke increase breast cancer risk: the report of the Canadian Expert Panel on Tobacco Smoke and Breast Cancer Risk (2009)