How can I use annotations to remove variants not relevant to cancer risk? I currently have ~180 whole germlines and around 10M SNPs/indels. I would like to build a predictive model using Machine Learning (ML) techniques to predict cancer risk according to these germline variants. The thing is, most of these 10M variants are not relevant to cancer, so first I have to annotate them and remove non-relevant SNPs/indels.

Which tools/measurements do you think are important to use in order to filter out irrelevant variants and keep only, let's say, around 10,000 SNPs/indels?

I guess the most obvious is CADD score, so I could keep those SNPs/indels with PHRED >= 10.
Which one would you suggest taking into account that downstream analysis would be ML algorithms that take as features variants that really contribute to cancer?
While your question is specific to cancerous germline mutations, I'd suggest you look at the COSMIC database of somatic mutations to include in your analysis. 

There are other factors to include in this kind of analysis you're suggesting, such as predictive deleterious effects (PolyPhen for example can perform such predictions).

If you have 10M variants/indels, for the same form of cancer, then look for common variants, or maybe the frequency of variants identified across exons of a gene.

What you've asked is highly broad and we'd need more specific details about your cohort to suggest more specific approaches.