using SNPs to identify mixed samples Is there a way to identify mixed samples based on SNPs?

Example input (table of genotypes for multiple samples):

|      | sample1 | sample2 | sample3 |
|------|---------|---------|---------|
| rs1  | AA      | BB      | AB      |
| rs2  | AA      | BB      | AB      |
| rs3  | AA      | BB      | AB      |
| rs4  | AA      | BB      | AB      |


Desired output: Sample relationships/distances. In this example, sample1 and sample2 are distinct, but sample3 is likely a mix of two (biological or technical).

Based on this simple example, the calculation seems doable, but maybe it's impossible in more complex realistic scenarios.

There is a nice tool peddy that does something conceptually similar, but it expects pedigree information as well as genotype frequencies for both the input samples and population.
Yes. The general term for identifying subsets from a mixed group is deconvolution.

This is easier to do when there is haplotype-level information available (e.g. sequenced reads that span multiple SNPs), but it is possible to get some indication of sample mixes from unlinked SNPs alone.

Deconvolution is helped a lot by having proportional data in the input (e.g. frequencies of genotypes, rather than a single genotype). Most SNPchip platforms have multiple probes per variant, and can provide some indication of variant proportion when digging down into the raw data. Deconvolution is also a lot easier with some knowledge about the component mixes. It's much easier to simulate mixes than to guess at possible mixes.

Here's a paper about the application of deconvolution in forensics.

Here's a paper about a program called CIBERSORT which carries out a deconvolution of immune populations based on gene counts:

Here's another paper which discusses the idea in more general terms.

You might find more information on one of these Biostars posts:


Deconvolution Methods on RNA-Seq Data
Differential gene expession analysis in cell populations of mixed tumor and normal cells