Tag: social science

The one-number census is not a good way to chart what is happening

It’s census time again.  I’ve made several criticisms of the process in the past, at first in an e-mail list and subsequently in two of my books, and I hate repeating myself.  Checking my previous posts, however, I find that I haven’t made any of those comments on the blog, so it makes sense to cycle through them now.

The first thing is to acknowledge that something like a census is essential.  The census gives us the denominators – the numbers that go below the dividing line.  Without those numbers, we can’t tell how big a problem is, or how evenly it’s distributed.

‘Something like’ a census, however, doesn’t mean it has to be this  census.  There are major problems with the UK census, and most of them are avoidable. I’ve heard Ian Diamond, who’s now the head of the Office for National Statistics, lecturing on this, and he emphasises the importance of  arriving at ‘one number’.  The  preservation of the census in this form is its greatest weakness.

If we look around the world, we’ll find that Britain’s reliance on a one-number census is increasingly unusual.    They’re used in southern Europe, but in other countries they use different techniques.  The USA has the American Community Survey, a large geographic sample of about 3% of the population, to get the fine detail.  Germany has a 1% micro-census and a range of information from administrative sources.  The Nordic countries are using registers of information.  France has a rolling census, allowing for regular updates.

The British census is too long, too complex, too unwieldy, and too slow.  If we go back to the historic archives, we can see what the census used to be: a literal count of the persons in each household, comprising names, ages and addresses.  That, frankly, is all we need from a single census – and as much as we can handle.  The English census has 51 questions. Every added question is another hole in the ship.

The first problem is time.  It generally takes two years to get the first results from the census, and, because it’s such a massive exercise, it has to last us for 10 years after that.  (We last had a mid-term census in 1976.)   That means, simply enough, that the census is always out of date – by at least two, and up to twelve, years.  If we want, for practical purposes, to do something useful, like setting up a primary school, the census is at best a rough, rusty guide.  Rolling data would be much better for the purpose.

The second problem is accuracy.  Statisticians have probably learned that as the numbers get bigger, they become more reliable – that we can be ‘more ‘confident’ about the findings.  This is just not true, at least not in the real world.   What happens with big numbers is that mistakes and biases are amplified, and we are liable to invest the numbers with meaning when they may have none.  I think people will remember, for example, the large number of write-ins claiming to be members of the Jedi religion; at least we can tell that’s bogus.  It’s more difficult to pick holes in other responses, but we should be able to acknowledge at least that we couldn’t rely on previous censuses to get the numbers of young men right.  If this census gives us an accurate, reliable count of people who are disabled or those whose gender is non-binary, I for one will be astonished.

If we really want to know about these topics, the census won’t give us the information.  That is going to rely on much more detailed work, probably with a qualitative  component to clarify what the answers actually mean.  That leads me on to social science, and finding better ways to do things.  Numbers about society are indicators – that is, signposts or pointers. We do not need accurate counts of everything; we need to have enough to prepare samples, which we can look at in more depth.  The census provides us with the sort of information we need to work out how to make a sample.  The great mistake is to suppose it can do more than that.

 

 

The problems with behavioural genetics are far from being resolved

An article about behavioural genetics has attracted a certain amount of attention from people who believe this sort of thing.  The article, by Paige Harden, is trying to rehabilitate this research, which has an appalling record.  The study has been tainted by fabrication and fraud.  And, of course by mass murder.  (There’s a good history in Carlson, 2001, The Unfit.)

The search for evidence that genes determine behaviour has lasted about 150 years, and still has a long way to go.  No-one has been able to establish any real proof that any of our behaviour is genetically determined.  However, that last point is  disputed by the keepers of the faith.  Harden writes:

“A meta-analysis of results from 50 years of twin studies … concluded in 2015 that genes do, in fact, make a difference for these types of social and behavioural outcomes – for people’s cognitive ability, personality, sexual behaviour, educational attainment and income. In fact, this finding is so consistent that it’s long been enshrined as the ‘first law of behavioural genetics’: everything is heritable. That is, variation in every aspect of human psychology and behaviour, and variation in every social outcome that’s influenced by one’s behaviour, is influenced by the genetic differences among us. Despite making different assumptions from twin studies, GWAS [genome-wide association study] results converge on the same answer: which genes you happen to inherit from your parents makes a difference for socially valued life outcomes, such as how far you go in school.”

Let’s take this apart.  The first thing to note is the claim that genes ‘make a difference’.  That is plausible enough, but it slides into a different argument as the article goes on: that “genes have causal power for people’s lives.”   That’s rather more than making a difference.  In the study she cites, the authors claim to demonstrate that genetics account for up to 13% of the variance in educational attainment.  Put that another way: at least 87% of the variance is not associated with the genomics.

The central problem with the claim that this has ‘causal power’ is that genes don’t actually determine development; they only establish a pattern.  The article that the author cites on the ‘laws’ of behavioural genetics comments:  “ Development is fundamentally nonlinear, interactive, and difficult to control experimentally.”  Just so – but it’s development that really matters.  Take, for example, the case of height.  That, according to Harden, is down to “the environmental and genetic accidents of one’s birth.”  Well, not quite.   Height is not set at birth. It’s the product of a phenotype – the cumulative development of a human body, conditioned by its genome, in its interaction with the environment.   There are exceptions, at the extremes – but they are exceptions. We know that people have been getting taller in every generation; that would not be possible if height were fixed by the genes.  We also know that people who are not well fed are liable to be stunted; that would be impossible, too.  Height is influenced by genes but is not determined by them.    And we should expect nothing more than that to be true of any developmental factor – such as weight, language development, sporting prowess or academic attainment.

The second problem lies in the idea of ‘heritability’.   In animal breeding, heritability is generally put down to genes, because the main differences between animals that matter to breeders are probably those conditioned at birth.  We don’t breed humans for their physical attributes, and there are a few other differences.  Human behaviour isn’t hard-wired – humans have fewer genes than most complex animals, and we should expect (and can observe) fewer inborn, instinctive behaviours.  The relative lack of fixed, instinctive behaviour is compensated for by a different mechanism: socialisation, mainly in families.  Families share a great deal, including a more or less common environment, language, cultural practice, leisure activities and diet.  Estimates for the heritability of obesity vary between 40% and 70%.  That does not mean obesity is genetically caused, only that it runs in families.  We cannot be certain for any individual whether their obesity is generated by a genetic blueprint, but there is a clue: obesity has been rising rapidly.   As the gene pool changes only very slowly over time, we can take it that the change is social.

Third, there is the reliance on twin studies.  For more than 50 years, twin studies have been an unfailing source of bad science.  The central assumption behind twin studies is that twins who are identical (monozygotic, or MZ) will manifest the same behaviours on that account; twins who are not identical will share 50% of genes, making them more like each other than ordinary siblings would be.  There are several problems with this.

  • Any similarities within families may well reflect similar environmental factors. That point is acknowledged by Harden.  Identical twins generally have social environments which are very similar indeed, and certainly more similar than fraternal twins. That’s why past studies tried to concentrate on identical twins reared apart – the problem being that (a) not enough twins are reared apart to make for a valid study, and (b) that even when twins are reared apart, social services agencies try to match their environments to the greatest possible extent.
  • Twins are not representative of general experience.  All twins have a different experience from other babies, and an experience which is similar to each other. They are typically born prematurely; they have to share the divided attention of parents, in a  different way to siblings; and because they have been born at the same time their environment and experience will be more alike than other siblings.
  • There is the common assumption of genetic similarity.  In most studies that is based not on hard genetic evidence but (believe it or not) on the impressions of the parent.  It should be no surprise that parents who think their children are more alike treat them as more alike.
  • Nearly all studies discard information about twins where one is disabled – a not uncommon issue – even though that disability means that children with the same genes clearly have different experiences.

What, we might reasonably ask, can twin studies prove?  The standard scientific approach is not to prove a hypothesis; it is to disprove it.  Identical twins cannot show us that something is genetically caused, because of confounding factors that cannot be controlled for.  What identical twins might be able to show us are the circumstances which are not genetic in origin, because if MZ  twins were to act differently we would know that the difference between them could not be attributed to their genes. There is  some presumptive evidence, for what it’s worth, that MZ twins can and do have different sexualities – the same in two thirds of cases, different in one-third.  If that happens at all, it isn’t genetically determined – though, being of a sceptical bent, I do wonder about the sampling and the lack of specific genetics which lie behind that finding.  I don’t expect behavioural geneticists to do the work to test this properly, because of course they are already convinced that everything is driven by genes.  Unfortunately, that presumption undermines the studies they make.

Fourth, there is the use of big data.  If we were serious about identifying genetic effects, we should be able to identify behaviours which exist only when a gene or set of genes is present, and which do not exist when those genes are not present.  We can do this with current genetic medicine – the same approach which last year saved my life; I know exactly which set of genes has malfunctioned, what it has done and what the best treatment is.  The GWAS research referred to might, in principle, be able to show something of the kind about behaviour – but the signs are not good.  Take the major study of genes and educational attainment cited above.  It processed data for 1.1 million adults and identified 1271 relevant gene clusters, that is ‘genome-wide significant SNPs’ .  A previous study, done on 294,000 adults, had found that they were able to explain only 3.2% of the variance.  In this study, the authors report that “A joint (multi-phenotype) analysis of educational attainment and three related cognitive phenotypes generates polygenic scores that explain 11–13% of the variance in educational attainment and 7–10% of the variance in cognitive performance.”

Taking that result at face value, what does it tell us? The most striking finding, which I’ve already pointed to, is that genetic analysis does not explain at least 87% of the variance, and in the worst case it did not explain 96%.  Genes may have an influence, along with lots of other things, but – if we trust the figures – it’s not a big one.

Now, let’s be a bit rougher on this research.  The first objection is straightforward:  association is not causation.  Even if the association was higher, it wouldn’t actually prove a causal link.  There has to be a specific generative mechanism, and that’s never been established.

Second, the stats weren’t developed to deal with numbers this big.  People often suppose that associations in bigger data sets are more reliable than associations in smaller ones. That’s not necessarily true; the larger the number, the more systemic imbalances can be amplified.

Third, this is a fishing expedition.  Whenever you deal with massive numbers of observations, there are going to be associations arising by chance.  The bigger the numbers, the more apparently significant associations there are going to be.  The claims in this study might, I think, be compared to the work on astrology, seeking to prove that people born under particular star-signs move into certain occupations.  Austin et al report, in the Journal of Clinical Epidemiology, no less (2006 59 pp 964-969) that people born under Gemini and Libra had increased mortality after infarcts treated by aspirin.  Make the sample big enough, and you’ll find some pattern.  Just don’t try to invest it with meaning.

Last, and not least, these associations might simply mask the real influences.   There are lots of other reasons why people might have higher or lower educational attainment – class, poverty, family background, language – and a comparison of educational attainment with selected elements of the genome doesn’t bother controlling for them.

A little more on why we can't trust the statistics in published articles

I’ve referred earlier this week to the work of Ioannidis, who argues that most published medical statistics are wrong. The British Medical Journal regularly uses its Xmas issue to publish some disconcerting, off-beat papers.  In a previous issue, they produced the findings of a randomised control trial which showed an apparently impossible result: praying for people whose outcomes were already decided several years ago seemed to work.  The message:  don’t trust randomised control trials, because they’re randomised.  This year, an article, “Like a virgin”, identifies 45 women in a sample of nearly 5,500 who claim to have had a virgin birth. The message: don’t believe everything people tell you in surveys.    If only medical journals applied the same rigour to some of their ‘serious’ results.

More nonsense about our genetic destiny

Yet another paper seems to show that our educational attainment is written in our genes.   It claims that “individual differences in educational achievement are substantially due to genetic differences (heritability) and only modestly due to differences between schools and other environmental differences”.   It’s been widely reported as a claim that exam grades are down to nature, not nurture.

This is based on comparisons of the figures for identical and non-identical twins. The reason why people use twin studies is because they believe that our personal characteristics are determined genetically, and so that studies of twins will confirm this.  That is bad science.  You’re supposed to design research so that it can disprove the proposition under test, and twin studies can’t do it.  What a twin study could show, in principle, is that where monozygotic (genetically similar) twins are different, that difference cannot be genetic.  That is not, however, what any of them try to do.

Genes are not blueprints for later development; the genetic structure (genotype) has to interact with the environment (phenotype).  Your height depends on your genes, but it is not determined at birth; if you are starved you may be stunted.  (The increase in height in successive generations has to be largely environmental, because the gene pool changes only very slowly.)   If there is a genetic link between common genes and attainment, it does not necessarily mean that the level of attainment is determined by genes – it only predicts similar patterns of attainment within a given environment.    So it is not possible to show that any level of GCSE scores is down to genes – it’s all about whether people from the same family, with the same home background, with the same school (and often with the same teacher), with the same experience in early years and of the same age will achieve similar results.  Put that way, it would be surprising if the results weren’t very similar – the more so because the sample has been selected to exclude twins where one of them is disabled.

Heritability is supposed to examine the extent to which varability in the phenotype is attributable to the genotype.  As there is no direct exploration of genes or genetics in most of these studies, what they actually look at is the way that similar characteristics occur in families, and those are attributed to the underlying genetic structure.   There are lots of reasons besides genes why educational attainment might run in families – among them culture, lifestyle, language,  common experiences, and so forth.  The authors suppose that identical and non-identical twins all have similar home backgrounds, so that the differences between them must be down to the issue of whether they’re identical or not.  That however depends on the proposition  that identical twins are not treated more like each other than non-identical twins are, and that seems implausible.  For example, non-identical twins may have gender differences, and children of different genders are liable to be treated differently.

While the study attributes the differences in performance to DNA, DNA was not usually examined.  The results are supposed to be about the differences between monozygotic and dizygotic twins, but it makes no serious attempt to determine whether the twins it is studying are either.  It states instead that “Zygosity was assessed through a parent questionnaire of physical similarity, which has been shown to be over 95% accurate when compared to DNA testing.”  So what the study actually finds is that if parents think their children are really like each other, those children get more similar educational results than they do if their parents think they are different.  The authors assume that the explanation for those similarities between twins must be their DNA – and not, for example, whether parents talk to them and treat them in the same way.

Having said that, there is one finding in this paper that brought me up short, and I think it does reflect on policy.  The argument is that as the curriculum has become more standardised, less and less variation between results is attributable to the school, and more and more to ‘heritability’ – which really, in this case, means the home background and early years.  That has deep implications for educational equality.

11 more genes for Alzheimer’s? Hardly

The reports of another supposed breakthrough in genetic research are, like so many before it, rather exaggerated.  Last week,  a New Scientist editorial commented that neuroscience

” is plagued by false positives and other problems. … Scientists are under immense pressure to make discoveries, so negative findings often go unreported, experiments are rarely replicated and data is often “tortured until it confesses”. …  Genetics went through a similar “crisis” about a decade ago and has since matured into one of the most reliable sciences of all. “

Yesterday the newspapers were stuffed with reports from that most reliable and mature of sciences, concerning the discovery of 11 genes newly implicated in the causation of Alzheimers.  This is from the Independent:

The role of the immune system in defending the brain against Alzheimer’s disease has been revealed in a study identifying 11 new genes that could help to trigger the most common form of senile dementia.

There’s more than enough there to be able to tell that the report is confused.  In the first place, Alzheimer’s disease is not a single disease entity; it’s a syndrome.  The term is used as a residual category for any form of dementia where there isn’t as yet a clear understanding of the process.   Over the years, the size of that residuum has gradually been reduced as various specific disease entities have been identified – Pick’s, Huntington’s, Parkinsonian dementia, Lewy body, CJD and so on.  The process of refinement still has a long way to go.  Second, there is no evidence that Alzheimer’s is genetically determined or ‘triggered’ by particular genes.  The study does not actually  claim to show that the immune system defends against Alzheimer’s.  All it does it to identify  a group of SNPs or snips (single nucleotide polymorphisms to their friends) associated with the immune system which show some association with the diagnosis of dementia.  That’s an interesting finding, because it suggests that it may be worthwhile to examine immune systems to see what connections emerge.  It’s not the same thing as showing that genes cause Alzheimer’s.

However, it’s not possible to exonerate the authors of the paper altogether of blame for the misrepresentation.  The title of the article, published in Nature Genetics, is:  “Meta-analysis of 74,046 individuals identifies 11 new susceptibility loci for Alzheimer’s disease”.  This does assume that the associations show ‘susceptibility loci’, and it emphasises that it’s a big study, which implies that it has greater authority as a result.   The conclusion suggests that what needs investigating is the potential association with the risk of Alzheimer’s.

There are three common errors here: the paper commits some of the cardinal sins of statistics.

  • Confusing association with causation.  An association doesn’t in itself tell us what the influence of genes is or what the direction of causation is.  It follows that assocation with certain genes doesn’t reveal susceptibility.
  • Confusing significance with risk factors.  A relationship can be highly statistically significant although its effects are very limited.   (On a graph, it’s the slope of the regression line that really matters rather than the closeness of fit of the observations).   It’s possible that some small part of the response is attributable to the associated factor, and in medical terms that’s potentially important – it could relate to a particular condition – but that’s not equivalent to a risk factor, and in any case the work done doesn’t identify that.
  • Fishing, or data mining.  In any very large body of data, there will be some unusual associations – it’s in the nature of the exercise.  It doesn’t follow that those associations can be invested with meaning.  This study  fishes for the data in a massive pool – over 17,000 people with Alzheimer’s, over 37,000 controls and more than 7 million SNPs.  Then in stage 2 there were 8572 people with dementia, 11,312 controls and 11,632 SNPs.  The significance levels were strict  (p < 5 per 10*-8), but the sheer size of the data sample makes the statistics more problematic, not less so.  The method can’t do more than suggest that some patterns merit further investigation.

See also: http://blog.spicker.uk/dementia-we-shouldnt-expect-miracles-from-drug-cures/

Genetic arguments and social policy

This is drawn from arguments posted on the Radical Statistics mailing list.

Genes are not a blueprint for the way we live. Biologists distinguish between genotype – the underlying pattern – and phenotype, the observable outcomes stemming from the interaction of genes, environment and the combined process of development. The argument has been made that environmental factors can make genes more important. For example, myopia, a condition rooted in genetic makeup, has been exacerbated by the development of reading. Variation in height, which is clearly governed by genotype, is nevertheless largely produced by environmental factors (which is why height has increased in succeeding modern generations). To illustrate the point, we know that two centuries ago, even if they were drawn from the same genetic pool, people were much smaller and lighter than we are now. One French study records that 79% of male recruits in 1792-9 were below 1.5 metres tall. The difference between that range and the range of heights in contemporary society is large enough to move people with a similar genetic endowment from a relatively low position to a relatively high one, depending on the developmental environment (primarily, in the case of height, on nutrition). A similar comment can be made about obesity. Estimates for the heritability of obesity vary between 40% and 70%.  That does not mean it is genetically caused; it means only that it runs in families.  Anyone who imagines that recent increases in obesity are due to changes in genetics isn’t living in the real world.

Despite nearly 150 years of trying, no-one has produced any good evidence that genes affect developed social behaviour in humans. With about 42,000 genes, it is easy to find statistical associations – at the conventional level where p<.05, there will probably be 2100 genes associated with any given character trait – but that does not demonstrate any causal link. Beyond that, however, most studies making claims about genetic origins of behaviour do not even try to show that there is a general association between the gene and the behaviour. They have simply relied on the occurrence of behaviour in specific families (1), and families have shared environments as well as shared genes. To the best of my knowledge, no study has ever shown that any social competence, personality trait or pattern of behaviour, of any kind, is shared by people with a common genotype or combination of genes while it is not present in others without that genotype. This is the minimum data that would be required to show that genes determine such issues.

Many studies rely, instead, on twin studies, in the belief that the similarity between identical twins must be genetic. This has three obvious problems. Firstly, any similarities within families may well reflect similar environmental factors. Second, identical twins generally have social environments which are very similar, and certainly more similar than fraternal twins. That’s why past studies tried to concentrate on identical twins reared apart – the problem being that (a) not enough twins are reared apart to make for a valid study, and (b) that even when twins are reared apart, social services agencies try to match their environments to the greatest possible extent. Third, identical twins are only relevant if one begins from the proposition that their genetic endowment is crucial. In other words, the studies assume the phenomenon they set out to prove.

The argument is not just bad science, It was used at the end of the 19th Century to justify the isolation of “degenerates” from the rest of the community. It was the basis for eugenics. It was closely associated with fascism, because it is an argument that was made by fascists for political reasons and offered in justification of the extermination of inferior humans. (2) The argument is sinister, and it deserves to be treated with deep scepticism.

Update, 24th November 2012. New Scientist reports this week about Mendelian randomisation, and that serves as a reminder to me that this criticism is beginning to be dated. The genetic linkage studies that were just being developed when I wrote this (e.g. Lancet, 2005 Sep 17-23;366(9490):1036-44) have started to bear fruit. A new epidemiology, described in Palmer et al’s Introduction to genetic epidemiology, has moved away from the old fallacy that behaviour is simply determined by genes; it begins, instead, with the proposition that different environments affect people with different genetic endowments differently. That makes it possible to distinguish the circumstances of people with certain genetic patterns from others – which is just what I was complaining here that studies hadn’t done to date.

Further update, 21st June 2020  The kind of work I was discussing here has, in all seriousness,  just saved my life.  A condition that would have been considered terminal five years ago has been distinctly identified through genomics and is now known to respond to a specific remedy.  This is a real and major development – and, as far as anyone can tell, heredity has nothing to do with it.

Note 1. S Jones, 1993, The language of the genes, London: Flamingo, ch 12.
Note 2. See R Lerner, Final solutions, Pennsylvania State University Press 1992.