This study focuses on STRUCTURE, but as far as I can see, it's also relevant to ADMIXTURE and FRAPPE. All of these are model-based algorithms for inferring ancestry, and suffer from very similar problems. I've already discussed some of these issues in an earlier blog post (see here), but it's great to see a formal paper on the topic.
One of the primary goals of population genetics is to succinctly describe genetic relationships among populations, and the computer program STRUCTURE is one of the most frequently used tools for doing so. The mathematical model used by STRUCTURE was designed to sort individuals into Hardy-Weinberg populations, but the program is also frequently used to group individuals from a large number of populations into a small number of clusters that are supposed to represent the main genetic divisions within species. In this study, I used computer simulations to examine how well STRUCTURE accomplishes this latter task. Simulations of populations that had a simple hierarchical history of fragmentation showed that when there were relatively long divergence times within evolutionary lineages, the clusters created by STRUCTURE were frequently not consistent with the evolutionary history of the populations. These difficulties can be attributed to forcing STRUCTURE to place individuals into too few clusters. Simulations also showed that the clusters produced by STRUCTURE can be strongly influenced by variation in sample size. In some circumstances, STRUCTURE simply put all of the individuals from the largest sample in the same cluster. A reanalysis of human population structure suggests that the problems I identified with STRUCTURE in simulations may have obscured relationships among human populations-particularly genetic similarity between Europeans and some African populations.
ST Kalinowski, The computer program STRUCTURE does not reliably identify the main genetic clusters within species: simulations and implications for human population structure, Heredity advance online publication, 4 August 2010; doi:10.1038/hdy.2010.95