I’ve referred earlier this week to the work of Ioannidis, who argues that most published medical statistics are wrong. The British Medical Journal regularly uses its Xmas issue to publish some disconcerting, off-beat papers. In a previous issue, they produced the findings of a randomised control trial which showed an apparently impossible result: praying for people whose outcomes were already decided several years ago seemed to work. The message: don’t trust randomised control trials, because they’re randomised. This year, an article, “Like a virgin”, identifies 45 women in a sample of nearly 5,500 who claim to have had a virgin birth. The message: don’t believe everything people tell you in surveys. If only medical journals applied the same rigour to some of their ‘serious’ results.
Yet another paper seems to show that our educational attainment is written in our genes. It claims that “individual differences in educational achievement are substantially due to genetic differences (heritability) and only modestly due to differences between schools and other environmental differences”. It’s been widely reported as a claim that exam grades are down to nature, not nurture.
This is based on comparisons of the figures for identical and non-identical twins. The reason why people use twin studies is because they believe that our personal characteristics are determined genetically, and so that studies of twins will confirm this. That is bad science. You’re supposed to design research so that it can disprove the proposition under test, and twin studies can’t do it. What a twin study could show, in principle, is that where monozygotic (genetically similar) twins are different, that difference cannot be genetic. That is not, however, what any of them try to do.
Genes are not blueprints for later development; the genetic structure (genotype) has to interact with the environment (phenotype). Your height depends on your genes, but it is not determined at birth; if you are starved you may be stunted. (The increase in height in successive generations has to be largely environmental, because the gene pool changes only very slowly.) If there is a genetic link between common genes and attainment, it does not necessarily mean that the level of attainment is determined by genes – it only predicts similar patterns of attainment within a given environment. So it is not possible to show that any level of GCSE scores is down to genes – it’s all about whether people from the same family, with the same home background, with the same school (and often with the same teacher), with the same experience in early years and of the same age will achieve similar results. Put that way, it would be surprising if the results weren’t very similar – the more so because the sample has been selected to exclude twins where one of them is disabled.
Heritability is supposed to examine the extent to which varability in the phenotype is attributable to the genotype. As there is no direct exploration of genes or genetics in most of these studies, what they actually look at is the way that similar characteristics occur in families, and those are attributed to the underlying genetic structure. There are lots of reasons besides genes why educational attainment might run in families – among them culture, lifestyle, language, common experiences, and so forth. The authors suppose that identical and non-identical twins all have similar home backgrounds, so that the differences between them must be down to the issue of whether they’re identical or not. That however depends on the proposition that identical twins are not treated more like each other than non-identical twins are, and that seems implausible. For example, non-identical twins may have gender differences, and children of different genders are liable to be treated differently.
While the study attributes the differences in performance to DNA, DNA was not usually examined. The results are supposed to be about the differences between monozygotic and dizygotic twins, but it makes no serious attempt to determine whether the twins it is studying are either. It states instead that “Zygosity was assessed through a parent questionnaire of physical similarity, which has been shown to be over 95% accurate when compared to DNA testing.” So what the study actually finds is that if parents think their children are really like each other, those children get more similar educational results than they do if their parents think they are different. The authors assume that the explanation for those similarities between twins must be their DNA – and not, for example, whether parents talk to them and treat them in the same way.
Having said that, there is one finding in this paper that brought me up short, and I think it does reflect on policy. The argument is that as the curriculum has become more standardised, less and less variation between results is attributable to the school, and more and more to ‘heritability’ – which really, in this case, means the home background and early years. That has deep implications for educational equality.
The reports of another supposed breakthrough in genetic research are, like so many before it, rather exaggerated. Last week, a New Scientist editorial commented that neuroscience
” is plagued by false positives and other problems. … Scientists are under immense pressure to make discoveries, so negative findings often go unreported, experiments are rarely replicated and data is often “tortured until it confesses”. … Genetics went through a similar “crisis” about a decade ago and has since matured into one of the most reliable sciences of all. “
Yesterday the newspapers were stuffed with reports from that most reliable and mature of sciences, concerning the discovery of 11 genes newly implicated in the causation of Alzheimers. This is from the Independent:
The role of the immune system in defending the brain against Alzheimer’s disease has been revealed in a study identifying 11 new genes that could help to trigger the most common form of senile dementia.
There’s more than enough there to be able to tell that the report is confused. In the first place, Alzheimer’s disease is not a single disease entity; it’s a syndrome. The term is used as a residual category for any form of dementia where there isn’t as yet a clear understanding of the process. Over the years, the size of that residuum has gradually been reduced as various specific disease entities have been identified – Pick’s, Huntington’s, Parkinsonian dementia, Lewy body, CJD and so on. The process of refinement still has a long way to go. Second, there is no evidence that Alzheimer’s is genetically determined or ‘triggered’ by particular genes. The study does not actually claim to show that the immune system defends against Alzheimer’s. All it does it to identify a group of SNPs or snips (single nucleotide polymorphisms to their friends) associated with the immune system which show some association with the diagnosis of dementia. That’s an interesting finding, because it suggests that it may be worthwhile to examine immune systems to see what connections emerge. It’s not the same thing as showing that genes cause Alzheimer’s.
However, it’s not possible to exonerate the authors of the paper altogether of blame for the misrepresentation. The title of the article, published in Nature Genetics, is: “Meta-analysis of 74,046 individuals identifies 11 new susceptibility loci for Alzheimer’s disease”. This does assume that the associations show ‘susceptibility loci’, and it emphasises that it’s a big study, which implies that it has greater authority as a result. The conclusion suggests that what needs investigating is the potential association with the risk of Alzheimer’s.
There are three common errors here: the paper commits some of the cardinal sins of statistics.
- Confusing association with causation. An association doesn’t in itself tell us what the influence of genes is or what the direction of causation is. It follows that assocation with certain genes doesn’t reveal susceptibility.
- Confusing significance with risk factors. A relationship can be highly statistically significant although its effects are very limited. (On a graph, it’s the slope of the regression line that really matters rather than the closeness of fit of the observations). It’s possible that some small part of the response is attributable to the associated factor, and in medical terms that’s potentially important – it could relate to a particular condition – but that’s not equivalent to a risk factor, and in any case the work done doesn’t identify that.
- Fishing, or data mining. In any very large body of data, there will be some unusual associations – it’s in the nature of the exercise. It doesn’t follow that those associations can be invested with meaning. This study fishes for the data in a massive pool – over 17,000 people with Alzheimer’s, over 37,000 controls and more than 7 million SNPs. Then in stage 2 there were 8572 people with dementia, 11,312 controls and 11,632 SNPs. The significance levels were strict (p < 5 per 10*-8), but the sheer size of the data sample makes the statistics more problematic, not less so. The method can’t do more than suggest that some patterns merit further investigation.
This is drawn from arguments posted on the Radical Statistics mailing list.
Genes are not a blueprint for the way we live. Biologists distinguish between genotype – the underlying pattern – and phenotype, the observable outcomes stemming from the interaction of genes, environment and the combined process of development. The argument has been made that environmental factors can make genes more important. For example, myopia, a condition rooted in genetic makeup, has been exacerbated by the development of reading. Variation in height, which is clearly governed by genotype, is nevertheless largely produced by environmental factors (which is why height has increased in succeeding modern generations). To illustrate the point, we know that two centuries ago, even if they were drawn from the same genetic pool, people were much smaller and lighter than we are now. One French study records that 79% of male recruits in 1792-9 were below 1.5 metres tall. The difference between that range and the range of heights in contemporary society is large enough to move people with a similar genetic endowment from a relatively low position to a relatively high one, depending on the developmental environment (primarily, in the case of height, on nutrition). A similar comment can be made about obesity. Estimates for the hereditability of obesity vary between 40% and 70%; but anyone who imagines that recent increases in obesity are due to changes in genetics isn’t living in the real world.
Despite nearly 150 years of trying, no-one has produced any good evidence that genes affect developed social behaviour in humans. With about 42,000 genes, it is easy to find statistical associations – at the conventional level where p<.05, there will probably be 2100 genes associated with any given character trait – but that does not demonstrate any causal link. Beyond that, however, most studies making claims about genetic origins of behaviour do not even try to show that there is a general association between the gene and the behaviour. They have simply relied on the occurrence of behaviour in specific families (1), and families have shared environments as well as shared genes. To the best of my knowledge, no study has ever shown that any social competence, personality trait or pattern of behaviour, of any kind, is shared by people with a common genotype or combination of genes while it is not present in others without that genotype. This is the minimum data that would be required to show that genes determine such issues.
Many studies rely, instead, on twin studies, in the belief that the similarity between identical twins must be genetic. This has three obvious problems. Firstly, any similarities within families may well reflect similar environmental factors. Second, identical twins generally have social environments which are very similar, and certainly more similar than fraternal twins. That’s why past studies tried to concentrate on identical twins reared apart – the problem being that (a) not enough twins are reared apart to make for a valid study, and (b) that even when twins are reared apart, social services agencies try to match their environments to the greatest possible extent. Third, identical twins are only relevant if one begins from the proposition that their genetic endowment is crucial. In other words, the studies assume the phenomenon they set out to prove.
The argument is not just bad science, It was used at the end of the 19th Century to justify the isolation of “degenerates” from the rest of the community. It was the basis for eugenics. It was closely associated with fascism, because it is an argument that was made by fascists for political reasons and offered in justification of the extermination of inferior humans. (2) The argument is sinister, and it deserves to be treated with deep scepticism.
Update, 24th November 2012. New Scientist reports this week about Mendelian randomisation, and that serves as a reminder to me that this criticism is beginning to be dated. The genetic linkage studies that were just being developed when I wrote this (e.g. Lancet, 2005 Sep 17-23;366(9490):1036-44) have started to bear fruit. A new epidemiology, described in Palmer et al’s Introduction to genetic epidemiology, has moved away from the old fallacy that behaviour is simply determined by genes; it begins, instead, with the proposition that different environments affect people with different genetic endowments differently. That makes it possible to distinguish the circumstances of people with certain genetic patterns from others – which is just what I was complaining here that studies hadn’t done to date.