Tag: research methods

The Natcen report on disability benefits is disappointing

The DWP declined to publish the Natcen report on The uses of health and disability benefits.  It’s now been released to the Work and Pensions Committee, and is available here.  The report is presented as a qualitative study; wisely, the authors have avoided numbers.  However, the form of the report is disappointing, and I cannot suppose that it is an adequate reflection of the work that was done.

Qualitative social research works, or needs to work, on two guiding principles.  The first is that what people tell you is evidence.  That is often derided by people in love with quants, but it is fundamental to the nature of evidence.  Courts of law judge evidence by looking for corroboration – probabilities and statistics aren’t enough.  As a broad proposition, evidence is corroborated when two or three witnesses  say the same things, confirming what the others have said.  This report doesn’t do that. It consists very largely of the researchers’ summaries of what people told them.  In a 79-page report, there are 15 ‘case illustrations’, and only 27 direct quotations from respondents.  That means, simply put, that while there are plenty of judgments, there is hardly any evidence given for those judgments.

The second principle is voice: what people tell you, and the way they tell it, matter.  Direct quotations, right or wrong, have a purpose and a moral authority.  Researchers have an ethical duty to report what people are telling them.  The way the respondents express themselves is fundamental to any adequate qualitative social research.

It may well be that this format was demanded by the DWP.  (I’ve been asked by other commissioners, in the past, to dump direct quotations and to just say what I think instead – and I’ve refused to do so.)  It’s very likely that the researchers intend to provide the evidence for this report, and to reflect the voice of the respondents,  in a separate publication.  However, they will need the DWP’s permission for this.  Their report, as it stands, does not do what it needed to do.

The Scottish Census of Health and Wellbeing: inept and possibly unethical

There’s some controversy in Scotland about a census of ‘health and well-being’ that’s being asked of schoolchildren of different ages.   This questionnaire seems to have been put together by a committee, all of whom wanted their particular issues to be included and addressed.  It’s  one of the worst designed questionnaires I’ve seen in years.

The area that’s attracted most concern relates to questions about children’s experience of sex. I found a copy of the census questionnaire from a local authority website, and was taken aback not just by the most controversial questions, but by the whole exercise.  I have no particular expertise relating to children’s health, but I taught research methods for more than 25 years, and this is not the way to do things.

First question: why is this a census?  Censuses are intended to give a comprehensive, precise count of issues.  There are well-known problems in doing this, because systematic non-response leads to systematic biases in the results.  Several local authorities have opted out, and many pupils will. The count will be meaningless.  What matters – the same as any other quantitative questionnaire – will be the relationship between answers: for example, whether there is a relationship between educational experience and negative body images.  We do not need a census to do this. It can be done at least as well, and probably better, by a series of smaller-scale social surveys.

Second question: in what circumstances does it make sense  to put together a questionnaire that is this long?   There are 61 questions, but because there are sub-questions,  pupils are actually being asked to answer, by my count,  126 distinct questions.  The rubric claims that the questionnaire should be completed in 20 to 40 minutes. To do this in 40 minutes a pupil could have to answer one question every 20 seconds.  Even if the questions weren’t sometimes difficult, this is wildly unrealistic, and it raises questions as to how valid the exercise can be: the exhausted respondents will skip, fabricate answers to finish or simply give up.

Third question: has the questionnaire been piloted and validated? There are indications it might not have been: the length of the questionnaire, the complexity of the language used, and the validity of responses to deeply personal questions.  Pupils are supposed to know what ‘intimate touching’ means (q 49) and whether an experience amounts to ‘penetrative vaginal sex’ (qs 50 and 51).

Fourth question: what measures have been taken to protect vulnerable respondents?   Some children will be distressed by the questions.  Some of that distress is predictable – for example, from those who have been subjected to sexual abuse.  Every school administering a questionnaire should have someone with specialised competence standing by to offer support in the event of distress – and that cannot be the teacher tasked with supervising children using computers, because it would not be possible for that teacher to break off for a distressed pupil before the questionnaire is complete. The FAQs issued by the Scottish Government say this:

What happens if a child or young person needs help, or wants to discuss something, after taking part in the Census?  At the beginning and end of the questionnaire, children and young people will be informed that if any of the Census questions have made them think of any problems, or has raised any issues they are having, then they are advised to speak to someone in relation to the information they have provided in the Census.  For example, if pupils are having problems with other pupils (e.g. feeling that they are being bullied), they are advised to talk about this with their parents / carers / teacher / support worker, etc.

That is not good enough.  This project should not have passed ethical review.

So, you may reasonably ask, what should the government have done instead?  That’s easy enough to answer:  a series of much smaller questionnaires, based on proper samples, administered by people with a competence in the field, and supported by people capable of responding to any distress.  It should include a proportion of open, qualitative questions. Good social research starts with listening.

Research is not a private matter

I was in Italy this week, taking a view of research ethics for a seminar organised by Pro-Res , a project to develop guidance on ethics for the EU.  The seminar was under Chatham House rules, which means I can’t comment on other contributions, but I can share my own, which I ran under the title “Research is not a private matter”.  The slides are here.