using SNPs to identify mixed samples
Yes. The general term for identifying subsets from a mixed group is deconvolution.

This is easier to do when there is haplotype-level information available (e.g. sequenced reads that span multiple SNPs), but it is possible to get some indication of sample mixes from unlinked SNPs alone.

Deconvolution is helped a lot by having proportional data in the input (e.g. frequencies of genotypes, rather than a single genotype). Most SNPchip platforms have multiple probes per variant, and can provide some indication of variant proportion when digging down into the raw data. Deconvolution is also a lot easier with some knowledge about the component mixes. It's much easier to simulate mixes than to guess at possible mixes.

Here's a paper about the application of deconvolution in forensics.

Here's a paper about a program called CIBERSORT which carries out a deconvolution of immune populations based on gene counts:

Here's another paper which discusses the idea in more general terms.

You might find more information on one of these Biostars posts:


Deconvolution Methods on RNA-Seq Data
Differential gene expession analysis in cell populations of mixed tumor and normal cells