Social Housing Green Paper: The Potential Pitfalls of Housing Association League Tables
The much-awaited Social Housing Green Paper was recently released and one of the main proposals is to introduce league tables to rank the performance of social landlords. In this blog, cross-posted from the UK Collaborative Centre for Housing Evidence (CaCHE) website, Dr Chris Foye draws on the CaCHE research project, Wellbeing as a tool for evaluating housing outcomes and housing policy impacts, to discuss some of the key issues around this proposal.
The Government’s logic for introducing a social landlord league table is clearly stated by James Brokenshire. In the foreword to the Green Paper, he states, “we also want to empower residents, to give them the tools they need to hold their landlords to account. To achieve this, we need to make it easier for residents to see how their landlord is performing compared to others.” This logic closely resembles that used to rationalise the introduction of performance metrics in other ‘quasi-markets’ like health and education. It views performance metrics as a means of providing ‘voice’ for tenants in the absence of choice, and allows central government to monitor and compare the bang for the public buck produced by different social landlords.
Two key questions arise at this point: ”how should performance be defined?” and “how should performance indicators be used?”. On the first question, the Green Paper suggests that “key performance indicators should be focused on issues of key importance to residents, such as: keeping properties in good repair; maintaining the safety of buildings; effective handling of complaints; respectful and helpful engagement with residents; and, responsible neighbourhood management, including tackling anti-social behaviour.” Clearly, this does not provide definitions, but it indicates the priority areas in which the Government believe performance should be scrutinised. The paper seems particularly interested in measuring residents’ satisfaction, both overall and specifically in relation to repairs. On the second question, the Green Paper argues that key performance indicators should be provided to the Regulator every year for open publication, although it is not clear whether the league table scores would be published in full, detailing the score of every social landlord on every metric, or whether a more opaque model would be adopted which, for example, only shows which social landlords have performed in the bottom-quartile of particular metrics. Significantly, the Green Paper also suggests the possibility of linking Government funding for new development to measured performance on the key performance indicators.
In the CaCHE research project, we address both of the above questions and, although the final report is still in progress, there are several points worth making;
Metrics prioritise things which can be quantified
Using metrics as a basis for allocating resources encourages the prioritisation of those things which can be quantified over those things which cannot. It is much easier to measure the state of repair of a house (through, for example, the Decent Homes Standard) than it is to measure the sense of security or social networks experienced by residents. How, for example, is it possible to put a number on the anxiety of the following lady who has been told that, along with her husband and elderly mother, she only has another 6-years guaranteed on her lease:
“I’m already a pensioner; so is my husband. Of course, Mum is disabled, and she’s in the pension age as well. So if you add our years to six years, you can see at what point we are going to be when this one is coming at an end. And when one is at an advanced age, it’s psychologically, emotionally… It’s the feeling knowing that, okay, like now, we’ve already… done five months, okay? And five months, that has come off – six times twelve: 72. …For me, I’m looking at it as having a bank balance, and you’re not able to credit anything, but you’re just minusing and in the end it comes to a zero, when you have nowhere.” (Taken from Fitzpatrick and Watts, 2016)
Metrics can create perverse incentives
The use of performance metrics as a basis for reward or punishment incentivises organisations and individuals to ‘game’ the system, leading to perverse behaviour (e.g. Bevan and Hood, 2006; Muller, 2018). If a low or declining rate of major crimes in a district becomes the metric according to which police officers are promoted, then some officers will respond by simply not recording crimes or downgrading incidents; if successful outcomes are used as a basis for measuring surgeon’s performance then surgeons will refuse to carry out riskier operations on patients with more complex problems.
There is a real risk that rewarding or punishing social landlords on the basis of performance indicators will encourage similar perverse behaviour. Distributing resources on the basis of customer satisfaction, for example, would incentivise housing organisations to employ a range of methodological tricks such as changing the order of the questions or surveying residents via phone rather than in-person (see here for discussion of biases). Even if the measurement is rigorously and uniformly implemented – which would be expensive – customer satisfaction metrics (and complaint metrics) could still incentivise housing organisations to unfairly prioritise certain types of residents. According to the top-down theory of life satisfaction, those with sunnier dispositions are more likely to report higher customer satisfaction, not because they get superior customer service, but because they are more satisfied in general. For example, the graph below shows that housing and life satisfaction fluctuated around the mean almost hand-in-hand, despite sustained increases in average rooms per person (the solid grey line) and the proportion of homes meeting the Decent Homes Standard. Satisfaction judgements inevitably spill over into each other, which makes it very difficult to isolate satisfaction with life from satisfaction with housing, repairs, or customer service.
UK average living space, housing satisfaction and life satisfaction from 1996-2008 (Source: BHPS)
Furthermore, according to the empirical evidence on adaptation (e.g. Foye, 2018) those individuals who have experienced bad housing in the past are likely to have lower expectations (and higher levels of customer satisfaction) in the present. This adaptation theory is also supported by the dotted grey-line in the graph above showing that as average rooms per person increased, so did the average living space of those self-reporting a space shortage. In sum, if a particularly cynical social landlord sought only to maximise their customer satisfaction metrics, they would be wise to target tenants who are easy going, happy, and used to very poor housing conditions and customer service. Is this the kind of behaviour we want to incentivise?
There is also a broader risk that, by encouraging social landlords to focus on maximising a pre-defined set of metrics, a league table would degrade and anonymise the work of staff and experience of tenants, whilst discouraging innovation. Tenants become a number, and staff are encouraged to set aside their own intuition, practical-knowledge, imagination and conscience, and focus on optimising the metrics.
“Uniform metrics vs Diverse objectives”
One of the fundamental arguments in our research is that the choice of metrics for evaluating social landlord performance (or housing outcomes more generally) is inevitably an ethical choice that relies on value-based judgements; and because different people and organisations have good reason to value different goods (e.g. some emphasising independence; others social connectedness), the best way of accounting for this diversity of perspectives and purposes is to start from the bottom-up.
To this end, it is encouraging to see the Green Paper propose that the Regulator should engage with relevant bodies, such as landlords and resident groups if/when preparing performance indicators. This sits well with the conclusions of Muller that “measurements are more likely to be meaningful when they are developed from the bottom up, with input from teachers, nurses and the cop on the beat”. A bottom-up deliberative approach would make use of the practical, often tacit knowledge that both practitioners and residents have of what works and what matters.
However, if a set of metrics were developed from the bottom-up, so that they respected the reasonable diversity of goals and values held by social landlords and residents, and accounted for the variety of institutional contexts in which they operate, then it is difficult to see how they could be translated into a standard league-table. For league tables to function properly, they typically require organisations to be ranked on the same metrics. League tables make the most sense if we’re all trying to do the same thing. For social landlords to function properly, however, they arguably need to have different objectives because they have different specialisms (see this PWC report), operate in different local markets, and serve residents (both present and prospective) who quite reasonably value different things from housing.
Any league-table that focuses primarily on comparability and fails adequately to recognise that housing organisations legitimately pursue diverse objectives risks homogenising the sector, as social landlords feel compelled to target a uniform set of metrics. Yet any system that seeks to capture diversity comprehensively risks producing information that is of limited utility to its intended audiences. This is the dilemma at the heart of all performance assessment systems of this type. It remains to be seen how the Government tackles it in the coming months.
Dr Chris Foye is a Knowledge Exchange Associate for the UK Collaborative Centre for Housing Evidence.