Posted by: ayrshirehealth | August 24, 2012

Measurement: for Improvement or for Judgement? by @micmac650

Data for …….

I love data. I get a kick out of analysing, finding new patterns and the joy of discovery. But unless it’s just a hobby, then you have to do something with the results, and that’s where it gets difficult. Using “data for improvement, not for judgement” is a mantra in quality improvement circles.

Having been brought up on Westerns and World War 2 movies, with a world divided into white hats and black hats, I have a tendency to splitting, and latched on to this phrase with relish:

  • improvement = good
  • judgement = bad

Recently, at the European Observatory Summer School (strongly recommended) I found myself splitting again, but this time between data for:

  • understanding (good) and
  • data for advocacy (bad).

It’s worth considering these splits a little further.

Judgement vs Improvement

In the NHS data are commonly used to judge us, usually by government –

The aim is to hold us to account for our performance, by comparison to our peers or to a standard.  So far so good – so why do I feel negatively about this?

Well, maybe it’s just me, but I don’t like someone else, uninvited, sitting in judgement on me and my organisation – it feels like a parent-child interaction rather than an adult-adult interaction.  There is a certainty, a reductionist simplicity associated with judgement (“you are performing badly”), which does not sit well in the complex healthcare arena.  I get irritated by the way it diverts the organisation’s attention from other matters, causing a distortion of priorities.

Conversely, if you’ve achieved the standard, or are the best amongst your peers, seeking further improvement can be an uphill struggle (have you tried sitting in A&E with your kid? Four hours doesn’t feel like excellence).

Collaborative approach

The improvement approach seems so much more collaborative. Let’s define the problem together; let’s agree on a practical measure; now what can we do to improve? It immediately draws us in to solving the problem together. It’s not my problem, it’s our problem. It’s not your measure, it’s our measure. It’s focussed on solutions.

It sets no limits to ambition. Achieved 98% of patients seen within 4 hours? OK, let’s aim for 3 hours.

And yet… Patients used to wait 12 hours to be seen in A&E. Patients used to wait 12 months to be seen in clinic. Genuine improvement has been driven by the judgemental approach.

I’m not so sure that such dramatic improvements in patient care can be demonstrated by the collaborative improvement approach, despite the overblown claims.

Advocacy vs Understanding

I was talking to a colleague about a research study he was leading, and asked him what result he was hoping for.  His response, although tongue in cheek, has stayed with me: I’m not hoping for any specific result, I just seek the truth.  This is the approach I admire, and to which I aspire: using data to understand. Yet how often do we see people using data with an agenda (and not just politicians)? Presenting data selectively, displaying data in a way that maximises their position rather than maximising understanding, foreshortening the y-axis, selecting the years that make their case; so many statistical tricks, so little respect for the truth, so little respect for the audience.

So when I heard a lecturer at the European Observatory Summer School discussing how to maximise the political impact of healthcare system data, I immediately felt uncomfortable. Choosing to emphasise one measurement requires a judgement, a belief that you are advocating for the right reasons – it’s a political act, not a scientific one. But this is the world where science meets application. Health departments can’t wait for the right answer, they have to allocate budgets now. And whether or not I am intending to influence, the very act of choosing what to measure is a priority setting exercise.


Splitting is, of course, a distortion of reality. So I was heartened to hear from the Chair and Chief Executive of Healthcare Improvement Scotland how they planned to balance their unusual role, as both an improvement organisation and a regulator.

Their organisation’s understanding of how to combine the various approaches to measurement was impressive, and does I think reflect an increasingly mature approach to data within NHS Scotland, in significant contrast to the Deliverology approach of old.

Not “OR” but “AND”

Data are essentially neutral – they carry no opinion, and need to be interpreted in context to generate information, and drive action. In the end, we need both the yin and yang: improvement and judgement; understanding and advocacy. Not “or” but “and”.

The leadership challenge is to get the balance right.  When you find your team’s performance is different from other organisations, seek to understand why, rather than simply berating them.

– Use judgemental data to drive understanding  – Use better understanding of data to drive improvement – Use improvement data to advocate with other teams.

But most of all, make sure you use the data!

Mark MacGregor is an Associate Medical Director in NHS Ayrshire and Arran and a consultant nephrologist.

Next weeks blog will feature our first ‘guest’ blogger – @nhshem (Elaine Mead) who is Chief Excutive of NHS Highland


  1. […] Measurement: for Improvement or for Judgement? by Mark MacGregor on the Ayrshire Health blog. Share this:TwitterFacebookLinkedInPinterestMoreEmailRedditPrintDiggStumbleUponTumblrLike this:LikeBe the first to like this. This entry was posted in charitable trusts, customer service, data, health, health services, public sector, working practices and tagged National Health Service by weeklyblogclub. Bookmark the permalink. […]

  2. […] week, considering how effectiveness and results of the National Health Service are measured in Measurement: for Improvement or for Judgement? This post is worth reading by those who are managers in  other public services since the issues […]

  3. […] Measurement for Improvement or for Judgement? On ayrshirehealth August 24, 2012. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: