How can I use annotations to remove variants not relevant to cancer risk?
While your question is specific to cancerous germline mutations, I'd suggest you look at the COSMIC database of somatic mutations to include in your analysis. 

There are other factors to include in this kind of analysis you're suggesting, such as predictive deleterious effects (PolyPhen for example can perform such predictions).

If you have 10M variants/indels, for the same form of cancer, then look for common variants, or maybe the frequency of variants identified across exons of a gene.

What you've asked is highly broad and we'd need more specific details about your cohort to suggest more specific approaches.