Some of the complexities of social protection, laid bare

The World Bank’s  Sourcebook on the foundations of social protection delivery systems is a substantial document, but it’s a bit of a curate’s egg: while some parts of it are excellent, others should be avoided.  ‘Social protection’ mainly refers to benefits; it seems to take in social work – part of the same process only in some countries – but doesn’t apparently extend to medical care, and social care for older people is largely dismissed in a page (half on page 254, half on 264-5).  The text is based around a seven-stage process of application and service delivery, shown in the graphic.

I wasn’t convinced at the outset that this  was an ideal way to explain how the process of administering benefits and services was translated into practice; that’s partly because they’ve opted not to use or even refer to some well-established literature about claiming, and partly because the labels they use  don’t quite capture what they intend to refer to.  ‘Outreach’ is about how the intended population is identified, and becomes aware of services prior to claiming; ‘registration’ is mainly about basic documentation; ‘onboarding’ is induction.  These stages aren’t necessarily sequential; many services apply eligibility criteria as a part of acquiring information about the population, and needs assessments are sometimes done to winnow out initial enrolment.  (I think the point is made for me in chapters 7 and 8, which have to track backwards to get a view about information, contact, referral and verification.)

Despite those reservations, I warmed to the model as the book went on, because it does at least give shape and structure to discussion of the issues.  It may be particularly useful for some of the advocates of Basic Income to consider: any viable Basic Income scheme still has to negotiate issues relating to documentation, identity, addresses, banking, how updates and corrections are made,  and such like.  This is the first document I’ve seen in an age which engages with that kind of issue.

There are, however, some problems with the way that the questions are explored, and arguably they reflect the agenda that the authors are implicitly following, much of which assumes that shiny new IT contracts and commissioning are the way to go.  I suspect that some readers will have strong reservations about the criteria the authors of this report set for schemes for disability assessment, which need to be ‘valid, reliable, transparent and standardized’ (p 107) rather than being, if it’s not too wild a leap of the imagination, personal,  dignified, expert or sensitive to complexities. The report promotes  a sizeable range of approaches using digital tech, but  the detailed coverage is fairly casual about many of the familiar problems that relate to reliance on such approaches – the obstacles the technology presents to claimants, the difficulty of determining whether their personal circumstances fit the boxes that people are offered, the role of the officials administering the system (most are not ‘caseworkers’ – a bureaucratic division of labour is more common), and the role of  intermediaries.   It doesn’t consider, with the main exception of enforcing conditionality, the possibility of using existing institutions such as schools and hospitals as the base for service delivery.

There are eccentricities in the way that benefits are described – the bland acceptance of proxy means testing, for example, the idea that a tapered minimum income is a ‘universal’ policy, and the treatment of grievances as a ‘confidential’ issue, when systematic reporting and review of complaints is essential to public management and scrutiny.   Taking the UK as an exemplar for the recording of fraud and error is a bit rich, when the accounts have had to be qualified for years because of it.  The entry that grated most, however, was about a ‘predictive tool’ for child protection, in  Box 4.10.   It claims that they have an instrument that can predict future out-of-home placements “accurately” and “to a high degree”.

Specifically, for children with a predicted score of 1 (predicted low risk), 1 in 100 were later placed out-of-home within two years of the call. For children with a predicted score of 20 (predicted highest risk), 1 in 2 were later removed from the home within 2 years of the call.

Calling this ‘accurate’ is overstating the case somewhat: if 1 child in 2 is going to be removed from the home, the other 1 child in 2 isn’t.    I tried to dig for more information, but couldn’t find it – the source the report cites for this study isn’t public.  From a previous published paper on the same project, it seems that the account given here is a misinterpretation anyway.  The purpose of the scheme was not to predict whether the child ought to be removed to a place of safety, but to stop the people who are answering the telephone hotline from dismissing calls about child abuse that might otherwise seem innocuous.  If similar low-level calls were being made repeatedly, they might all be dismissed in the same way.  This looks more like a problem in logging and managing referrals than it is a problem with casework judgments. But for what it’s worth, predictive tools based on profiling referrals have major limitations when it comes to managing benefit claims, too.  The problem is that there are too many variations and complexities for generalisations about claimants to work at the level of the individual.

Grading students’ work

Reversing previous decisions about the grades awarded to students whose education has been interrupted makes some kind of sense, but only some.  The problem is that we expect grades to mean three somewhat different things, all at once.

In the first place, grades are given for achievement – reflecting the knowledge, skills and competencies that students have demonstrated.  This is problematic in the current situation, because even if the grades are given fairly, they will reflect the position of students who will have done several months less work or development than previous cohorts have done.

Second, grades are supposed to represent potential – not what students have actually  achieved, but what they might be capable of doing with further development.  If present achievement gave us a clear guide to the future, that might work – but it doesn’t, and there’s always been the suspicion that it says more about the preparedness of the school and the resources that school students are offered than it does of the abilities of the pupil.  This was a problem before the pandemic, and it will be a problem long after it.  The truth is that we only have very unreliable predictors of what students might be capable of – A levels, in particular, often make over-fine distinctions between very narrow bands across the grades, and are a weak guide to university performance.

Third, the grades represent opportunities, and impose limits on those opportunities.  If a student wants to study medicine, for example, the opening will depend more on high academic performance than it does on personal experience, sensitivity, commitment or interpersonal skills (the sort of thing that we used explore in interviews for social work places).

The emphasis on opportunity is the argument that has carried the day.   In Scotland, the decision will make it possible for more than 3000 students to go into a university course they wouldn’t otherwise have been admitted to.  Many people will look at that and say: why not?   But there is an objection: increased opportunities within the current system might just mean that people have more opportunity to fail.  The French system opens doors to everyone with the Baccalaureat, but it fails half the students after the first year.  In the UK, the institutions with the most liberal admissions policies are also likely to be the lower status institutions, and they may lose  up to 20% of their students as the course goes on.   It doesn’t follow that we’re wasting those students’ time, but  far too many university courses work on the principle that students must ‘sink or swim’.  I’d be more confident in the process if at the outset there were more engagement with students and more emphasis on developing the skills they’ll need to qualify.

Mandatory Reconsideration is “a disproportionate interference with the right of access to court”.

Eighteen months ago, I made a case on this blog that the process of Mandatory Reconsideration demanded by the DWP was unlawful, that it was designed to prevent claimants from access to justice,  and that it stood clearly in breach of the principles enunciated by the Supreme Court in the Unison case.  Now the  High Court has heard a case about MR as it affects Employment and Support Allowance.  Justice Swift demurred from the case I argued for in one important respect:  that even if MR was an “impediment or hindrance” to access to justice, it did not actually deny people the right of access altogether.  Nevertheless, the judge decided that the process was “a disproportionate interference with the right of access to court”, and found in favour of the claimant.

Decisions of the High Court are not necessarily decisive, and it is likely that this judgment will be assumed to apply only to ESA claims.  But the grounds for the judgment are matters of general principle, and they apply  across all benefits to which the process has been applied.   That prompts some questions.   First, what does it take to get rid of regulations that are transparently  unlawful?   Second, why did the process of independent scrutiny, undertaken by the experts of the Social Security Advisory Committee, not raise concerns when these regulations were being introduced?  And third, what on earth was the Scottish government thinking of when it decided to mirror “a disproportionate interference with the right of access to court” in the design of the Scottish social security system?

 

 

Academic freedom: the problems with a contentious report aren’t mainly about statistics

A report published by Policy Exchange seeks to defend right-wing academics against the suppression of their academic freedoms.  Their cause is open to question, and I’m not sure that I should be bothering with a report that has been described as ‘methodologically abysmal‘, but I’m intrigued that there’s so little understanding of basic research methods on both sides of the argument.  On one hand, we have this somewhat inept explanation in the report itself:

The sample consists of 820 respondents (484 currently employed and 336 retired; average age of current academics is 49 and of those retired is 70). Given the approximately 217,000 academic staff working in British universities in 2018-19, our sample is proportionately many times larger than a conventional opinion survey (typically a sample of 1,500 across a national population of 60m). As such our data has a good claim to being representative of the wider academic population even though, as with all opinion surveys, there is a margin of error in the results.

A survey isn’t made more representative simply by being larger.  There are potential biases in the inclusion of a hefty proportion of retired academics and the assumption that non-responses (from page 51, 24% to 39% of the totals) don’t skew the results .

On the other hand, we have the combative response of Jonathan Portes, who comments that this would fail any basic undergraduate course on statistics.  Well, he’s right that their argument is based in bad statistics.  The reporting of the methodology and the questions isn’t systematic or complete. The size of a sample does not make it representative, and making it bigger does not make it more representative, it only magnifies the bias.  But of course this sort of thing  wouldn’t actually fail a project, because undergraduate projects are judged by what they do, not just by how sound they are.   I’m also troubled by Portes’s dismissal of ‘dubious anecdotes‘, the common complaint of those who believe in the inherent superiority of numbers.  What is the difference between ‘anecdotes’ and responses that can be counted?  Why is richer, fuller evidential material less credible than ticked boxes?  Qualitative research studies do the same kind of thing that is done in the courts: they look for evidence, and they look for corroboration of that evidence. The ‘anecdotes’ in most research studies, including this report, are the bits that really matter.  Additional note, 13th August:  Jonathan Portes has written to me to clarify that he was intending to challenge accounts that he thought were ‘fabricated’, rather than the validity of using anecdotes.

In the course of my career, I’ve taught research methods for about twenty years.  I’ve often found that neophyte students come to the subject with preconceptions about what research evidence ought to look like: ideally there should be numbers, and clear categories of response, and statistics, and statements about representativeness. That seems to be the attitude that has prevailed here.  The basic questions we need to ask, however, are not about statistics.  They are, rather, a question of what makes for evidence, and what we should make of the evidence when we have it.  The Policy Exchange report tells us openly that it was looking for corroboration of problems experienced in a small a number of widely reported incidents – that’s the background to their report, in Part 1.  Their sample consisted of academics and retired academics registered as respondents on Yougov.  There may have been some statistical biases in that process, and it’s possible that the retired academics may have answered differently to others; we do not have enough information to tell.

Their respondents pointed to a range of issues.  The questions they ought to have asked about their data, then, was not ‘is the sample big enough?’, or even ‘how representative is this sample?’   but ‘what does the evidence tell us about the issue we are looking at?’  The first thing you can get from a survey like this is a sense of whether there’s an issue at all.   The second is whether there is corroboration – whether different people, in different places, have had related experiences.  There’s some limited evidence to back that up -there are contributions from a handful of right-wing academics,  but the  report also indicates that there is a small but identifiable element of political discrimination across the spectrum.  (I’ve encountered that myself: I have been rejected more than once for jobs because the external assessor at interview objected to something I’d written about poverty.)  Interestingly there is little in the survey relating to more extreme examples, and ‘no platforming’ hardly appears as a problem.  The third is whether we can discern patterns of behaviour.  That’s more difficult to judge, and it’s where information about extents might have been helpful; the main pattern the report claims to identify is a ‘chilling effect’, that people who are fearful of consequences tend to alter their behaviour to avoid the potential harm.  That’s plausible but not conclusive.

The two main weaknesses in this report, in my view, are not about statistics at all.  The first rests in the bias of the design.  The questions asked people tendentiously about right-wing causes such as multiculturalism, diversity and family values.  An illustrative question:

If a staff member in your institution did research showing that
greater ethnic diversity leads to increased societal tension and
poorer social outcomes, would you support or oppose efforts by
students/the administration to let the staff member know that they should find work elsewhere? [Support, oppose, neither support or oppose, don’t know]

I suppose my immediate reaction would be that anyone who claims to ‘show’ a clear causal link between complex and unstable categories of behaviour, rather than ‘argue’ for an interpretation, hasn’t quite grasped the nature of social science.  (The same criticism would apply to someone claiming to prove the opposite.)  But the questions that people ask often reveal something about the position of the team that’s asking, and this is the point at which, if I’d been asked, I’d probably have stopped filling in the questionnaire.  (I wasn’t asked.  I was removed some years ago from the Yougov panel after I objected to the classification of racial groups I was being asked to respond to.  I got a formal letter from Peter Kellner telling me my participation was no longer required.)

The report’s other main weakness lies in its political recommendations, centred on the appointment of a national Director for Academic Freedom.  I couldn’t see any clear relationship between the proposals for reform and the evidence presented.

 

 

 

Universal Credit isn’t working: the Lords Economic Affairs committee’s judgment on UC

The report on Universal Credit by the House of Lords Economic Affairs Committee will not come as a great surprise to anyone who’s been following the evidence.  (Except, perhaps, me: I submitted an evidence paper  myself in March, and in the wake of moving house, and major illness six weeks after that, I’d quite forgotten that I’d done it until I saw it quoted on page 12.)  The chair of the committee, Lord Forsyth, has suggested that the report approves of UC in its ‘concept’, but that there are problems with its design and its administration.  Given the questions that the committee raises about real-time information, digital access and fluctuating incomes, I think the criticisms go beyond that.  The problems they recognise also include a familiar litany of failures:  among them, the 5 week wait, unpredictable benefit streams(“impractical” and “fundamentally unfair”), conditionality, the two-child limit and inadequate levels of benefit.  Despite all that, the system is “not broken irredeemably”.

The initial request for evidence asked whether UC had achieved its objectives.  As the statement of objectives has frequently shifted for political expediency, it’s hard to say even what the current tests are, but the committee proposes a range of distinct principles: dignity, adequacy, providing a predictable income, flexibility, fairness, achievable ends and comprehensibility.  They suggest that there could be a two-week payment up front, a three month fixed level of payment, and that child support could be taken out of the system altogether – those all seem to me to be good ideas.   But the fundamental problems will remain:  a tapered benefit, an obsession with labour market issues when most claimants aren’t going to be in the labour market, and a reliance on information that can’t be supplied or managed.

The government’s misleading figures about poverty reflect a wider problem

The Prime Minister has been upbraided by the Office for Statistical Regulation (part of the UK Statistics Authority) for his assertion that child poverty is falling, when on all the tests used by the government the opposite is true.  I’m not greatly enamoured of those tests.  I’ve considered the case for the standard test of ‘relative’ poverty, 60% of the median income, in other work – it’s not bad, but we need to accept that it’s a pointer, not an authoritative measure. The claim that the figures for 2010/11 represent a test of ‘absolute poverty’ is particularly suspect.  Having said that, however, there’s no real excuse for blustering that poverty has been getting better, when your own figures say that’s not so.

This is part of a wider problem, and one we’ve seen increasingly in the course of the last few years.  The UK Statistics Authority was formed in the hope that it would be possible to maintain confidence in the integrity of official statistics.  In the course of the last ten years, however, we’ve seen a growing contempt for statistical evidence,  shown in the treatment of figures about crime, social security claims, incapacity, the management of coronavirus and more.  It’s done whenever departments publish figures that are not official, when the press is steered to have a go at popular targets like migration or benefit fraud, and when ministers just make stuff up. There is a cost to undermining public trust: it’s not just that some figures can’t be believed, but that everything becomes open to doubt.

The UNDP considers the case for a temporary Basic Income

In previous writing, I’ve expressed reservations  about the idea of a Universal Basic Income, but I’ve also noted that the key objections – the opportunity costs, and the distributive implications – may not apply in the same way in the current crisis.   The United Nations Development Programme has taken forward the idea of a temporary basic income, which might make provision for half the world’s population in 132 developing countries.  They examine three main options: a top-up income (which would be complex, difficult to administer and liable to be inconsistent); a BI which is different in different countries; or a uniform BI paid at a standard rate across the developing world.

None of the proposals would have the long-term effects that proponents and opponents of Basic Income claim would result from such a scheme – people should not be expected to to change their lives, give up work, enter education, start businesses or start a family just because a change in their immediate income is made for six months.  (I’ve doubts as to whether we’d see those effects fully realised in decades – after old age pensions were introduced in the UK, it took the best part of 60 years, two world wars and a health service before we could see the full effects on labour market participation.)    The primary case for offering something like a basic income is much simpler: to maintain enough demand for an economy to function, and to make sure that people in developing economies are included despite the crisis.

There are, however, major obstacles in the way.  Most proposals for Basic Income are so much concerned with the principles that they don’t get into the practical detail of how such a benefit can be delivered – for example, ID, banking, physical addresses or the lack of them,  responsibility for dependents and communications.  That is all challenging enough for a developed economy, and in a less developed one it is easy to see how the ship could hit the rocks.

Yet more failings of Universal Credit

“To govern”, Polly Toynbee writes, “is also to deliver”.  For anyone looking for further evidence of governmental incapacity and incompetence, Universal Credit is the gift that goes on giving.  A new report from the National Audit Office  focuses in the main on the problems facing new claimants, but on the way it points to a series of other issues.   The most immediate problems are

  • The effect of the 5-week delay before first payment.  57% of claimants seek advance payments, which means that their benefit is subsequently reduced to repay the advance; a further 22% delay claiming and incur debts as a result.  So, taken together, 80% of claimants face financial difficulties because the benefit is not designed to provide help when it is needed.
  • ‘Fraud and error’ – a figure which lumps together a load of different problems – is running at 10.5% of payments, almost a record for benefit payments.  Most of this, the NAO reports, is down to fraud by claimants;  we’re not told what type of fraud, but if so, UC is even more untypical of other benefits than it seemed to be at first.
  • The cost of implementing the benefit is increasing: the current estimate is £4.6 billion.

It’s worth reflecting on the last of these.  The Full Business Case for UC had claimed that UC would yield £24.5 bn in people choosing to work more, £10.5 bn in distributional improvements, whatever they are, and £9.1 billion in reduced fraud and error.   The ‘distributional benefits’ are unclear: UC has imposed a terrible cost on the people it has failed to serve, with most claimants suffering financial hardship and (despite some moderation) a ridiculously large number having benefits stopped in the name of discipline – nearly 90% of all  sanctions in 2019 were imposed as a penalty for missing appointments.  The last figure is obviously wrong, and in the wrong direction.  For the first, if there was ever any basis for the DWP’s claim that 200,000 more people would move into work – there probably isn’t – £4.6 billion would cost £23,000 for every one of these.  For the same price, the government could have created rather more than 200,000 useful jobs.

The administration of Universal Credit is judged to be irrational and unlawful

In an extraordinary judgment, the Court of Appeal have declared that an element of the assessment of Universal Credit is so irrational as to be unlawful.  Gareth Morgan has explained, at some length, how Universal Credit manages to take information about regular, predictable incomes and rent, and convert that into an irregular and unpredictable stream of income. The point of contention in this case is a small part of that, but the problem the court was looking at is pretty straightforward.  Banks do not operate on bank holidays and weekends, and the Universal Credit assessment ignores that.  That has meant, for people unlucky enough to have an assessment marked down for an awkward date, that regular income will be  counted as too high, or too low, and the amount they receive will fluctuate wildly.  The court was not impressed by the DWP’s arguments that changing the assessment in this case would be expensive or inconvenient.

“The Secretary of State for Work and Pensions’ refusal to put in place a solution to this very specific problem is so irrational that I have concluded that … no reasonable SSWP would have struck the balance in that way. “

How, we might reasonably ask, did the DWP get in this mess?  There have been three recurring problems.  First, Universal Credit was designed by people in a mental bunker, determined not to share, consult or engage people who understood how benefits work – including their own front-line staff.

Second, the system has been built around the capacity and convenience of the ICT, rather than the tasks that needed to be done.  The technology has never been capable of doing the sort of things that are needed to run a system properly – the failure of the verification system  is an illustration.  The process we now have in place is cumbersome, complex, slow, very expensive and difficult to change.  We did things faster, and cheaper, forty years ago.

Then there is the failure of the safeguard, which depends on the scrutiny of regulations by the independent Social Security Advisory Committee.  I’ve been critical in the past of the SSAC’s approval of policies and regulations that I cannot believe would stand the test of judicial review, such as mandatory reconsideration or the sanctions regime.  (Both policies violate the centuries-old principle of audi alteram partem, or natural justice: no legal penalty should be imposed without first giving the person sanctioned the opportunity to be heard.)  The problem is, I think, that the SSAC tries to reach decisions through consensus, and an insistence on compromise  is fundamentally inconsistent with the role of independent expert advisers in identifying specific issues that fall within their area of expertise.  Illegality should not be treated as being open to negotiation.  The bulk of the SSAC’s work is confidential, but in the light of this judgment they need now to review whether they had identified the fatal legal flaws in the regulations,  and if not why they failed to do so.

I’m not keen on Gareth Morgan’s proposed fix for the problem of dates, which is designed to tweak the assessment for the people affected while minimising disruption to the information that the DWP gathers.  By all means, let’s disrupt this. Real-time assessments are beyond our capacity to deliver; whole month assessments, delivered at the end of the same month, lead to radically unstable income streams; individual variations in payment arrangements lead to complexity and confusion.   I lean towards the French system, which is to use, as far as possible, retrospective information about means, and the same predictable, common dates for everyone: it makes a huge difference to the predictability and stability of income.

My private thanks to the NHS

I didn’t join the final ‘clap for the NHS’.  On that day, I had gone for blood tests in the early afternoon.  The medical practice is 45 minutes away from the nearest hospital and labs, so analysis took a little time, but I was called back to go directly to to Victoria Hospital, in Kirkcaldy, within two hours of seeing the GP.

The care I received was exemplary.  It had first to be confirmed that I did not have Covid-19.   After that, everything worked as it was supposed to.  The GP identified the problem and took the necessary steps to deal with it, at great speed.  There was a coordinated response from a range of different consultants and specialities,  supported by specialist services in Edinburgh and Southampton.  The nursing staff in particular were thoroughly professional, warm, polite and good-humoured.  What I owe them all is immeasurable.