There’s some controversy in Scotland about a census of ‘health and well-being’ that’s being asked of schoolchildren of different ages. This questionnaire seems to have been put together by a committee, all of whom wanted their particular issues to be included and addressed. It’s one of the worst designed questionnaires I’ve seen in years.
The area that’s attracted most concern relates to questions about children’s experience of sex. I found a copy of the census questionnaire from a local authority website, and was taken aback not just by the most controversial questions, but by the whole exercise. I have no particular expertise relating to children’s health, but I taught research methods for more than 25 years, and this is not the way to do things.
First question: why is this a census? Censuses are intended to give a comprehensive, precise count of issues. There are well-known problems in doing this, because systematic non-response leads to systematic biases in the results. Several local authorities have opted out, and many pupils will. The count will be meaningless. What matters – the same as any other quantitative questionnaire – will be the relationship between answers: for example, whether there is a relationship between educational experience and negative body images. We do not need a census to do this. It can be done at least as well, and probably better, by a series of smaller-scale social surveys.
Second question: in what circumstances does it make sense to put together a questionnaire that is this long? There are 61 questions, but because there are sub-questions, pupils are actually being asked to answer, by my count, 126 distinct questions. The rubric claims that the questionnaire should be completed in 20 to 40 minutes. To do this in 40 minutes a pupil could have to answer one question every 20 seconds. Even if the questions weren’t sometimes difficult, this is wildly unrealistic, and it raises questions as to how valid the exercise can be: the exhausted respondents will skip, fabricate answers to finish or simply give up.
Third question: has the questionnaire been piloted and validated? There are indications it might not have been: the length of the questionnaire, the complexity of the language used, and the validity of responses to deeply personal questions. Pupils are supposed to know what ‘intimate touching’ means (q 49) and whether an experience amounts to ‘penetrative vaginal sex’ (qs 50 and 51).
Fourth question: what measures have been taken to protect vulnerable respondents? Some children will be distressed by the questions. Some of that distress is predictable – for example, from those who have been subjected to sexual abuse. Every school administering a questionnaire should have someone with specialised competence standing by to offer support in the event of distress – and that cannot be the teacher tasked with supervising children using computers, because it would not be possible for that teacher to break off for a distressed pupil before the questionnaire is complete. The FAQs issued by the Scottish Government say this:
What happens if a child or young person needs help, or wants to discuss something, after taking part in the Census? At the beginning and end of the questionnaire, children and young people will be informed that if any of the Census questions have made them think of any problems, or has raised any issues they are having, then they are advised to speak to someone in relation to the information they have provided in the Census. For example, if pupils are having problems with other pupils (e.g. feeling that they are being bullied), they are advised to talk about this with their parents / carers / teacher / support worker, etc.
That is not good enough. This project should not have passed ethical review.
So, you may reasonably ask, what should the government have done instead? That’s easy enough to answer: a series of much smaller questionnaires, based on proper samples, administered by people with a competence in the field, and supported by people capable of responding to any distress. It should include a proportion of open, qualitative questions. Good social research starts with listening.