Category: Public administration

Review: B Greve (ed) Handbook of Social Policy Evaluation, Elgar 2017

This is a review of a book I’ve been sent by the publisher, Edward Elgar. I’ve explained the terms on which I review books here.

Bent Greve, the editor, explains that this book is intended to cover “methodologies, cases and how and when to make evaluation in social policy, and its possible impacts”.  Part 1 offers six chapters on the techniques, most of the chapters in part 2 discuss issues in evidence based policy making, mainly offer discussions of evidence on sixteen policy areas (though two of the seven chapters are really case studies rather than discussions of that sort), and part 3 has ten further chapters of case studies.  That really means that half this book is about evaluation and evidence, and half is discussion of specific policies.  The more interesting case studies, to my mind,  were on medicines, long term care and interventions for children, but none is exceptional.  A couple of the other case studies are discussions of policy, not centrally about evidence or evaluation at all.  This does not add up to either a book on evaluation, or a handbook for people who want to do one.

The real place to start, unfortunately, is the price.  This book, 520 pages plus index, costs £195 – that’s not a misprint.  (NB: Since I posted this review, the publishers have told me there is a cheaper e-version at £36 on Google Play.)   The price might be defended for a library purchase if this was an irreplaceable, essential read, but it’s hard for any collection of papers to meet that standard.  Like most collections, it has a mix of good and not so good, and there’s a lot of competition out there.  On evaluation, Parsons’s short book on Demystifying Evaluation (Policy Press, 2017) is a relatively  accessible and practical guide.  I still like Michael Scrivens’ Evaluation Thesaurus, even if it was published in 1991; his evaluation checklist is available free on line.  On evidence and policy, there have been several recent books; I’d recommend Justin Parkhurst’s The politics of evidence (Routledge, 2016) an insightful and well-written book that explains a lot of what’s wrong with the main approach in this one.  Parkhurst’s book is available in a free Open Access version.

There’s other stuff out there for free, too.  The World Bank offers a virtual library of methods and case studies in evaluation, along with guides such as P Gertler et al,  Impact evaluation in practice (World Bank, 2011).  There are problems with the  World Bank’s approach; it’s technocratic and liable to be uncritical.  The objections are

  • theoretical – evaluations, especially RCTs,  often look at the wrong things in the wrong way;
  • political: aims, methods and outcomes cannot be understood in a political vacuum;
  • statistical: the variables are interdependent, the methods heavily sensitive to the statistical assumptions  and results are unlikely to be replicable; and
  • evidential: the available data don’t support the main methods.  Garbage in, Garbage out.

Some of the writers for the Handbook refer to some of these points, but neither the criticisms nor the practicalities are dealt with thoroughly.   A reader has to dig for the objections, because they are referred to in the context of the paper where they occur, not systematically – you have to get through a hundred pages, well into the chapter on systematic reviews, before you even catch the sense that the “evidence hierarchy” introduced on page 6 is contentious.  There may be room for a state-of- the-art review of evaluation and policy, but this isn’t it.

New research: listening to social security officers

For the last three months, I’ve been working with Helen Flanagan of PCS to listen to the voices of social security officers about the social security system.  PCS is the civil service union representing people working for the DWP, HMRC and others.  Helen’s explanation of the work can be seen here, but it will be mid-May before we’re able to publish the findings in detail.  Yesterday’s meeting made liberal use of the things said by DWP officers, in three contexts: a pamphlet issued by PCS on the Future of Social Security in Scotland, a film by Jennifer Jones, and initial findings from our report – we still have statements from fifty or more people to process.

This is from Helen’s summary.

  • Over 200 staff have participated so far, in their own time, from a range of DWP workplaces in Scotland.
  • Discussions were not limited to the devolved powers, we wanted input on policy, structure, working conditions, the experience of service uses, and what goes wrong with service delivery.
  • What we have with DWP is a system in chaos: the constant change and reorganisation of work, with inadequate staffing levels, poor training and failing off-the-shelf IT systems.
  • What we have also found is a real desire from staff in DWP to be able to serve people better, and for a system that treats those applying for help with humanity.

And, although I can’t really do justice to the range of issues, or the sincerity of people’s voices, in a short soundbite, let me pick out a couple of things that people have told us:

Really senior leaders are saying that we’ve never gone through the scale of these changes, but then on the other hand, things are just pushed through without planning. They expect staff to be ‘calm amongst the madness’, but then staff are then  hit about the head by management if they don’t adapt to the changes properly.

Segmentation plays a big part in bad service. Cases bounce about. It’s really difficult to embed with it or own it yourself. You can’t take ownership of anything.

The current robotic system fails to take much notice of the human factor and the fact we are dealing with real people

Morale is appalling. The deskilling that goes on in this place is appalling. I … was no more skilled than anyone else when I came in. All the changes that have come in over that time, I’ve had skills and training, but they’ve picked away at all that specialisation and skill, so you’re demoralised and run down. You have to put you in menial tasks and deal with small stuff rather than deal with proper work.

Further note, 6th October:  The report has now been published on the PCS website.

P.S: This was my 800th posting on the blog.

 

Benefit administration: some lessons from the United States

The UK may have some claim to distinction in its efforts to transform the benefits system, but we are not alone.  In the United States, revisions to health care have been taken as a stimulus to States to introduce new computer systems that will solve all the problems, typically covering Medicaid (health care for people on low incomes), food stamps and other benefits such as help with maternity.  Kentucky was early off the blocks: they hired Deloitte Consulting to introduce a new system.  50,000 people suffered ‘massive disruption.’  The Kentucky Courier-Journal reported:

“State workers, bewildered by the complicated new Benefind system, find themselves obstructed from helping many clients by errors, glitches and programming flaws.”

When Deloitte moved on to their next big contract with Rhode Island, covering 30,000 people a month, they were determined not to repeat the same mistakes.  A spokeswoman explained to the Providence Journal in Rhode Island:

“The design, development, testing and implementation of this new system is unique to Rhode Island and the people we serve. Kentucky used a different approach on all of these things.”

Instead, they found new ones to make. The Governor of Rhode Island complained that

“We paid them a lot of money, we didn’t get what we paid for. And they represented to us that it was in much better shape than in fact it was: defective functionality, incomplete interfaces, engines that still aren’t working.”

The formal  Assessment of the Unified Health Infrastructure Project  (it’s quite short, and worth reading) reports that

  • the IT system was not functioning properly
  • despite reassurances from the contractor, the system was not ready to go live when it did
  • too much was taken on trust because of the contractor’s experience in industry
  • staff had not been properly trained
  • “Basic user functionality and important interfaces … have significant defects or have been deferred, requiring extensive manual workaround processes”
  • benefits were not processed when they should be, and not withdrawn when entitlement had ceased; there were “errors in eligibility determination, benefits issuance, and
    provider payments”.

To those engaged in Universal Credit, it all looks eerily familiar.

Nothing daunted, Deloitte is now engaged in an even bigger project, providing a new system for income testing and assessment for the state of Georgia, expected to cover three million people.  A spokeswoman for Georgia’s Department of Human Services has reassured Georgia Health News that “We had the benefit of watching other states.”   As do we.

John Slater explains the thinking behind the project management of Universal Credit

John Slater has been responsible for a series of Freedom of Information requests about the Universal Credit fiasco.  Yesterday he sent me a copy of the project management plan  introduced by Howard Shiplee, who was responsible for the development of Universal Credit from May 2013 until his departure, following illness, in September 2014.  Shiplee had previously been responsible for building construction for the 2012 Olympic Games.

I was puzzled by the plan, and wrote back to John:

I’m baffled – I can see no relationship between the steps to be taken and the design of a social security system. It looks more like a plan for building a McDonalds outlet, where all the groundwork’s laid and you know exactly what you want to do, so it’s all about delegating tasks. … I think you’re a project manager, John – – can you explain it to me?

I found John’s response so marvellously clear and helpful that I asked him if I could share it on the blog.  Here it is.

“Hi Paul,

You are right my background is programme and project management (my first degree was IT so I understand that aspect as well). You aren’t far off with your McDonalds analogy.

The plan is a classic case of an organisation focusing on the IT side of a major change programme. UC is one of the biggest change programme ever undertaken and nothing I’ve ever seen produced by the DWP reflects this.

The 100 day plan is a classic example of people that have been on a training course (e.g. Prince2 or Management Successful Programmes) but have never done the job for real. If you look down the left hand side of the ‘plan’ you’ll see the following headings:

  1. Key dates & decisions
  2. BT – Business (I suspect BT means business transformation)
  3. BT – Service Design & Build (I suspect BT means business transformation)
  4. BT Interfaces (I suspect BT means business transformation)
  5. Pathfinder Day 2
  6. Programme Approach
  7. HR
  8. Finance
  9. Assurance
  10. Security
  11. Comms (Communications)
  12. Stakeholder
  13. Supplier

With the exception of point 1 these are typically referred to a work streams. The idea is that each of the workstreams goes along their merry way cooperating with each other to deliver the programme. The reality of this approach with any complex programme is that it always goes horribly wrong.

If you look at points 2 to 5 then it is utterly focused on the IT. The plan looks like something to produce a software product of some sort. There is no mention of culture change, process engineering (this should be done before any software is produced) and the biggest issue of all people! This covers the claimants, DWP employees, Council Employees, Welfare Advisors and so on. They are just expected to magically learn and make it work. The trouble is human beings don’t work that way.

Part of the issue is that the DWP employees working on UC at the time hadn’t ever done anything like this before so didn’t have a clue. The put people in roles (e.g. programme manager, programme office manager etc) but they hadn’t done it before and had just been sent on a training course.

I’ve been doing this stuff for 30 years and I would have struggled to get UC up and running (and I’m very good at this aspect of complex programmes). Bringing in someone like Howard Shiplee was always going to fail. I’ve run programmes involving a lot of construction and it’s a different world and a totally different mindset. I suspect if you looked at the approach used for construction during the London Olympic build it wouldn’t look dissimilar to this plan. With construction the focus is generally on design and then build (known as D&B). The key factor is the supply chain and can the main contractor get the materials and people on site on time and in the right order. If you look at the plan again I don’t think it’s unreasonable to see the left hand side of the dark vertical as ‘design’ and the right hand side as ‘build’. This is what Howard Shiplee understood and it was so deeply ingrained I doubt he could have done anything else.

In respect of the pathfinder system released at Wigan it was a cobbled together lobotomised version of the IT that would ultimately be required for the complete UC. At this stage of the programme IDS knew the IT was fundamentally flawed, hence the talk of large sums being written off at the time. He also knew that they had to start over again but couldn’t admit that as it would be politically disastrous. Therefore, they rolled out the lobotomised version that only covered a small subset of people claiming JSA and claimed success. While this version was being rolled out painfully slowly the DWP was working desperately to produce a brain new IT system that ultimately will be the UC IT System.

Personally I think the new IT system will also fail. The methodology (Agile) as it’s been used by the DWP means that too much has been done in isolation. The system is going to be extremely complex and as bugs appear I’m not convinced the DWP will be able to find out the cause and then develop a solution that doesn’t result and another problem.

Kind Regards

John”

 

The no detriment principle rears its head again

In several pieces and presentations last year I pointed to the implications of the ‘no detriment’ principle agreed by the Smith Commission, which means that the Scottish Government will bear the costs of changing policies that otherwise would be suffered by the UK government.  In January I gave an example of this: HMRC has charged the Scottish Government for the cost of not doing anything about stamp duty.  Now I read this, in a report published by Audit Scotland today:

56. DWP has adjusted its systems and guidance to accommodate the introduction of the SRIT (Scottish Rate of Income Tax). It estimates this will cost £1.7 million. DWP charged  the Scottish Government £870,000 for this work in 2015/16, and expects further costs in 2016/17.

Yes: that’s a bill for changing its guidance.  Presumably based on average cost rather than marginal cost, because there’s no marginal cost in adding detail to routine updates of guidance.   The principle  is important, because when it comes to big changes – such as introducing new disability benefits – the DWP is going to follow through with billing for changes throughout the system, and the costs are likely to be prohibitive.  How would the British government respond if they get a bundle of equivalent bills from the EU – the cost of removing references to 28 countries from leaflets, or changing address books?  They really need to start rethinking their approach to transferring responsibilities.

Why we should have some sympathy for Lord Howard

It’s been reported that Lord Howard, once a hardline Home Secretary who became briefly the leader of the Conservative Party, has had a little local difficulty with the law.  When his car was detected speeding, Lord Howard was unable to say whether he or his wife was driving.  It’s a route they drive frequently, the notice of committing an offence comes some time after the offence has been committed, and either of them could have been driving.   Lord Howard has been heavily penalised for not making the declaration, suffering a penal fine and extra points.  The two drivers had the same number of points;  they could have agreed a story between themselves; they could have lied.  They chose not to.  Instead, they gave the honest answer: we don’t know.

Regular readers of this blog will be ahead of me.  I’ve argued for years, sometimes to the bafflement of MPs and MSPs,  that people claiming social security can’t sensibly answer the questions that the authorities want to ask them.  People with disabilities can’t say whether they’re disabled or not (most of them get it wrong).  People who are forming a relatonship with someone else often can’t say when that person becomes a partner.  People who start work in our new, ‘flexible’ labour market may not  know whether they’ve got a job or not, or even if they’re going to be paid.   When they fail to answer the black-and-white question, or when they plump for the wrong answer, they’re penalised for it.  I trust that Lord Howard will now have the insight to champion their cause.

The slow death of local government

The Times has reported the prospect of “all out war between the Sturgeon administration and Scotland’s local authorities.” (Councils to be stripped of powers in local government revolution, 11th November 2016, p 19).  It may be a little difficult to think of any field where councils have many powers left to be stripped.  Local authorities are a shadow of their former existence: after 1945 they have lost powers relating to social security, health, public utilities, and more recently they have  lost most of their functions relating to housing or social work.  Education has been boxed in by a national curriculum.  Planning and licensing  have been strangled by the imposition of a quasi-judicial framework, requiring councillors to behave as if they were judges rather than representatives. Residual powers relating to police and fire have been centralised.    So current proposals to take out still more functions, or to shift responsibilities for bin collection to ‘towns’, would leave little more than a shell behind.

Some years ago, I asked a group of councillors for their views.  Many were deeply resentful of the Scottish parliament, which in their view claimed a democratic legitimacy which they thought they had in equal measure.  The current system of local government is already remarkably centralised by European standards – Highland Council, to take the most obvious example, is responsible for a geographical area the size of a small country.  I don’t think there is any way to resolve the conflict within the current framework, but there is at least a way to put reform in a positive light.  There’s a powerful case for real decentralisation – locating services at a level where communities can get much greater control of the affairs that matter to them.  To do that, there needs to be a system of local government that is based on existing communities – towns, villages, and traditionally recognised areas – as a centre for schools, housing and public amenities.  That would call for reform both at central and at local government levels.

The Tax Credits fiasco

I’ve spent much of the afternoon listening to the proceedings of the Work and Pensions Committee concerning Concentrix. HMRC had identified 1.5 million Tax Credit cases where they had concerns; they engaged Concentrix, a private firm, to process cases.  Concentrix was reported to have sent out a million letters fishing for information, challenging for example whether they were not living with an undeclared partner; the firm’s representative told the  Committee that they had sent out 324,000, though over a shorter period.  People who did not reply to the letters had their benefits stopped; 90-95% of those who asked for reconsideration had the decision overturned (that was Concentrix’s estimate – HMRC gave a lower figure, of 73%).  HMRC had terminated Concentrix’s contract, but they seemed much more concerned about the collapse of the phone line in August than about the huge number of wrong decisions that their policy had generated. Frank Field MP, the Chair of the Committee, told the Independent:

The Committee was astonished by the extraordinary evidence we heard. From Concentrix we saw a company desperately out of their depth and unable to deliver on the contract awarded to them by HMRC. From senior HMRC officials we saw a palpable disregard for the human implications of this gross failure of public service. From the tax credit claimants we saw dignity in the face of appalling and traumatic experiences.

 

 

More problems with EBT cards

EBT stands for ‘Electronic Benefit Transfer’.  The system has been used in several US States and the general approach has a fairly consistent record of messing up the administration.  It’s not astonishing, then, to read a report about an audit in Pennsylvania,  which found that 2324 dead people received benefits last year.  The audit is 114 pages long, much of it in repetitive appendices,  and it may not be your leisure reading of choice, but there’s a brief press report here.  The breach arguably isn’t that bad numerically, given that there are nearly two million cardholders – but this is only one possible category of misuse.  What this audit is really about is trying to devise procedures to plug the gaps that the discrepancy has revealed.

This kind of problem is fairly predictable – I’ve criticised the approach several times on this blog.  Financial institutions and banks generally know where the vulnerable points are in their security, and over the years they’ve developed familiar processes to reduce the problems.  Alternatives to money, such as EBT cards, come with none of those protections, and that’s why reports like this have to be written to develop them.

Apparently we’re all insisting that water joins the retail revolution.

The general arguments for competition – arguments about quality, innovation and cost control – don’t apply very clearly to the supply of domestic water.  The provision of fixed water supplies is one of the strongest cases of a “‘natural monopoly”; the quality of the product has to be uniform; and there are strong public goods argument to say that the provision should not be subject to the choices of providers or consumers.  No house in the UK is considered fit for habitation if it does not have an internal fixed water supply providing drinkable water.   So it’s already questionable whether the arguments for ‘competition’ being forwarded by the regulator, Ofwat, offer any clear advantage to citizens.

That makes the recent statements of Ofwat’s Chief Executive, Cathryn Ross, somewhat puzzling.  She claims:

“We are living in an age of retail revolution, but water customers are being left behind. Customers tell us they think they should have the freedom to choose and don’t understand why water is the only retail market in which there isn’t some form of competition.”

Who, I wonder, are these customers?  How many of us think of water as a ‘retail market’? Are we baffled by the lack of competition?  Are there really deputations of villagers  standing there with pitchforks and firebrands, demanding that their services be privatised?