The Failure of Facial Recognition Tools – Lessons for Healthcare

The headlines yesterday reported that police facial recognition tools were ‘staggeringly inaccurate’. The idea is that cameras scan faces in a crowd – for instance, at a football match – and match faces to those on a police database, thereby helping the police to identify known offenders. On the face of it, the numbers do sound extraordinary, since over 91% of those identified as being a known suspect turn out to be innocent members of the public – but should we be ‘staggered’ to find out that the technology performs so badly? The maths suggests the opposite – it was almost certain that the numbers would be this poor.

The maths is not difficult. Let’s assume that the cameras are going to be used at a football match attended by 50 000 fans, and that every fan goes through the recognition software. Let’s also assume, for a moment, that this is a particularly crime-ridden football match and 10% of those attending are on the police database – so there would be 5 000 offenders in the crowd.

I’ve not been able to find out how good the technology is on a case-by-case basis, but let’s assume it is accurate 90% of the time – because that sounds like quite a good test and makes my maths easier in the process! That means that it will successfully identify 9 out of 10 of the criminals, and also 9 out of 10 of the innocent bystanders.

So for 5 000 criminals it will identify 9 out of 10, which is 4 500 and miss 500 – not too bad.

For 45 000 innocent bystanders it will correctly identify 9 out of 10, which is 40 500, but it will mistakenly identify 4 500 as potential criminals.

Put these together and we have 4 500 criminals and 4 500 innocent bystanders that the police think are criminals – a 50% success rate, or a 50% failure rate, depending on which way you look at it.

Now look what happens if we have a more realistic number of criminals in the crowd – surely even the worst football clubs don’t have 5 000 criminals at every match! What if only 1 in 1000 fans are criminals? Maybe I’m naive, but the idea that 50 known criminals are in every football match still seems like quite a high number to me. We can do the same maths.

Out of the 50 criminals, the police will identify 45 and miss 5.

Out of the 49 550 innocent bystanders, the police will correctly identify 90%, which is 44 595, but will incorrectly identify 10%, which would be 4 955.

4955 is a similar number to the first example, but now a far higher portion of the total, since there are only 45 identified criminals – a 99% failure rate. All of a sudden, the 92% failure rate by the police seems more understandable!

The big problem is that when you are looking for something rare, the chance of a false positive result (the innocent bystander being thought to be a criminal in this case) starts to massively outweigh the number of true positives unless your test is fabulously good. In fact, if only 1% of the crowd is a criminal, facial recognition software would have to get it right not 90% of the time, but over 99.9% of the time just to get a success rate of 50%. If the police have any mathematicians on board, they should know this.

So what does this mean for health? In health we often do tests where we are trying to find something important when the chances are much more likely that nothing serious is going on. The obvious example is screening – when we test patients who have no symptoms in order to make sure that if there is a problem then we find it early enough to deal with it – such as mammograms in the breast screening programme, PSA testing for prostate cancer, or a treadmill test to look for heart disease. Other examples are when doctors and nurses arrange tests routinely – such as an ECG and a Chest x-ray for most people who attend A&E with any hint of chest pain – or ‘just in case’ tests to make sure we are not missing anything. These tests may all be of value, of course, but the lesson we must learn is that when the chance of finding a true positive result is low, the problem of false positives can become a huge problem.

What really matters then is to consider the possible consequences of a false positive – it might be something fairly minor, like having to be recalled for a repeat blood test, or it could involve unnecessary procedures like a biopsy, or an invasive angiogram to check someone’s heart is ok. Then there is the anxiety that can be caused, and the waste of health resources spent separating false positives from true positives.

These are difficult, complex issues, but if we can learn something from the problem the police are facing, we might get better, as both healthcare professionals and patients,  at asking these sorts of questions before we arrange any test:

  • Do we need to do this test?
  • How likely is the condition that we are testing for?
  • How likely is it that the test will result in a false positive?
  • What are the consequences of a false positive?
  • What might happen if I don’t do the test?

Challenging stuff!

DNACPR – have we got the language right?

Decisions around cardiopulmonary resuscitation can be fraught with difficulty at the best of times; you are talking about what to do when someone’s heart stops, which is never an easy subject. I have been thinking for a while that the language doctors use is an added barrier to a good understanding of the issues, and I have had an Opinion piece published in the ~British Medical Journal which is free to read and can be found here. I hope it might be of interest.

You have to subscribe to the journal to be able to leave a comment, but do feel free to leave a comment here if you have anything to add to this important discussion.

The Real Driver Behind the Pfizer Fines

The record £89.4m fines imposed by the Competition and Marketing Authority on Pfizer and Flynn Pharma just before Christmas were a real good news story for the NHS. While I wrote about the story, my own efforts 4 years ago drew a blank in terms of actually making things happen, and it was Dr Stephen Pike, a GP in Shoreham-by-Sea, West Sussex, who has been the real driver behind the CMA investigation that has ultimately led to this landmark judgment. Dr Pike made a formal complaint, on behalf of Coastal West Sussex CCG to the Office of Fair Trading (the forerunner of the CMA) at the end of October 2012, and contacted me in February 2013 to let me know the good news that a formal investigation was to take place. In the 4 years that have followed, Dr Pike has been assisting the CMA in their investigation, and so has needed to keep his own name out of the news until the final verdict had been reached. Now the CMA has reached its verdict, the pivotal role Dr Pike has played in this investigation can be acknowledged, and the credit given where it is due.

That a pharmaceutical giant has been called to account for its actions in this way is a fantastic achievement. What is more, Stephen plans to seek a compensatory payment in addition to the imposed fine, from both Pfizer and Flynn Pharma for the losses incurred by CCGs as a result of the excessive pricing of phenytoin capsules for the past four years, to be re-invested by CCGs back into patient care. This is where the real impact of this judgement could be felt – since the compensation could be significantly more than the fines, and will have to be paid back to the CCGs and not go to the Exchequer.

We Need to Talk About Conflicts of Interest

When I penned my previous post on the possible role of antibiotics in the treatment of back pain, I was unaware of one vital piece of information which, for me, changes everything: The doctors behind the research had a significant conflict of interest, which they had not declared when they submitted the article for publication.

This was first brought to my attention by Ben Riley, who keeps the Ferret Fancier blog, and the issue has also been covered by Margaret McCartney in the British Medical Journal. To summarise, three of the four authors are part of an organisation called MAST Medical, which states:

The latest research shows that back pain from Modic changes can be successfully treated with a prolonged course of antibiotic treatment.

To ensure that treatment is successful patients should consult a MAST certified doctor and/or therapist.

It is no surprise that to become ‘MAST certified’ requires attendance on a course run by MAST Medical, and payment of an inevitable fee. Apparently the authors did not think that this was a conflict of interest, because the website was launched three months after the article was published – a defence of their position which stretches the concept of conflicts of interest beyond breaking point.

The current system with regards to competing interests relies on self-declaration – this is a problem, since the editor of a journal cannot police every article that is submitted to them, or challenge authors concerning conflicts about which they are unaware.

Scientific method should make self-declaration a reliable system. A true scientist is more concerned about elucidating the truth than promoting their own interests; they will always be keen to see if their results can be reproduced by other researchers before declaring them to be proven; a true scientist may still have a conflict of interest – but they will positively want to declare it as they know that it could bias their interpretation.

Unfortunately, not everyone in the medical world behaves as a true scientist. This group, far from being concerned that they could be biased, have chosen to defend what is an obvious financial interest in the results of their trial. A cynic might guess that they delayed the launch of their website precisely so that they could get away without declaring it. As a medic I am very concerned that they are promoting antibiotics for the treatment of back pain before their results have been replicated elsewhere.

Sometimes declarations may fail to be made for very obvious reasons of personal gain – we only need to remember Andrew Wakefield and the MMR scandal to realise just how serious this can be – while at other times it may be due to laziness or thoughtlessness on behalf of the authors. An example of this more innocuous, but nevertheless important, neglect to make a proper declaration occurred when I wrote to the British Medical Journal, as part of a diverse group of people involved in healthcare, to raise concerns about the prospect of screening for dementia.

Our letter was initially published as a rapid response, where it triggered a reply from an eminent group of doctors who declared that they had no competing interests. The letter was subsequently published as an Observation article, and a further reply came from many of the same authors, several of whom decided on this occasion to declare interests, including financial support from pharmaceutical companies and appointments related to the field of dementia. These conflicts are not wrong in themselves, and do not invalidate their comments, but neither are they unimportant, nor did they arise in the intervening three-month period between the letters. What changed? A prick of the conscience? A word from the editor? Or perhaps the fact that the BMJ tightened its wording about conflicts of interests in January 2013 – between the two submissions.

Even a robust policy on this matter is insufficient, however. The BMJ has very clear guidance on what should be declared, but there is very little the editors can do when proper declarations are not made – with retraction of the article being their most draconian punishment. Retraction may have repercussions in the scientific community, but it is rarely reported on in mainstream media. The furore and excitement on the front pages of our national newspapers that surrounded the news that back pain could be treated with antibiotics is hardly likely to be repeated were the paper to be retracted by the journal at a later date, and so patients with back pain – and even their GPs – may never get to hear the full story.

The BMJ, along with many other leading journals, has encouraged the use of a uniform reporting system for competing interests, but this is by no means universal, and still lacks teeth. There needs to be a national debate on this important issue, and serious consequences for failing to declare significant competing interests. I don’t know what these should be – a ban on publication for a period of time, rather like a football player being suspended? A published apology? Even fines?

If this seems heavy-handed then we should remind ourselves of the consequences of misleading reasearch. Andrew Wakefield published his discredited research in The Lancet in 1998. It took 6 years before his financial conflicts of interests were unearthed by Brian Deer of The Sunday Times, but it was only in 2010, when he was struck off the medical register by the GMC, that the journal finally retracted the article. How much damage was done in the field of public health in the meantime – and is still being felt today as the outbreak of measles in Swansea is a stark reminder?

The danger with Wakefield is that we write him off as a dishonest maverick and fail to learn the lessons of a system that is broken and in need of a major rethink. We don’t need to wait for the next scandal before we talk about conflicts of interest – we need to do it now.

Books for the Blues

The Arts Council-funded project to promote self-help books for mental health conditions was widely reported in the media last week. In true Daily Mail style, their coverage was accompanied by an image of a bikini-clad woman, apparently reading one of the ‘books on prescription’ whilst sitting on an idyllic beach with her toes resting in the gently lapping water – not exactly the experience of most patients in the throes of an episode of depression! The inappropriate visuals notwithstanding, the scheme was generally well reported, and there is much to be said in its favour – including formal backing from the RCGP. The challenge will be to make it really work to the benefit of patients, and to avoid any risk that it will be used as an excuse to cut an existing service.

From May 2013 libraries will be stocked with a list of 30 self help books on a range of topics from depression and anxiety to obsessive compulsive disorder, along with a sister list of ‘mood-boosting’ novels and non-fiction titles. GPs are to be encouraged to issue a ‘Reading Prescription’ to patients suffering from mild-moderate mental health problems who will then receive library membership as well as access to the full reading list. At first glance it seems odd to use the term prescription – these books will be freely available to anyone who cares to visit the library after all – but perhaps there is sufficient power in the doctor-patient relationship that a recommendation from your GP could carry significant weight. I have a small handful of self-help books which I encourage my patients to dip into, and I am always encouraged by how frequently they take up the suggestion and the positive results that follow.

There is certainly evidence in favour of the role of self-help books in the management of depression in primary care, including a recently published randomised controlled trial. What this trial and my own experience with self-help books have in common, however, is guided support. In the trial this consisted of 3-4 face-to-face appointments totalling 2 hours of contact, while my own practice would involve a small number of typical 10 minute GP appointments – either way, the self-help books were not used in isolation. What seems especially important in my own practice is to be able to recommend a book which I have read personally – that way I can choose the right book for the right patient, and explain exactly why I think it will suit them. It is the importance of this personal touch that left me concerned by one quote, from the Daily Telegraph, which reported Debbie Hicks, director of research at the Reading Agency as saying: ‘All 30 books could also be picked up by library members who wanted to avoid the perceived “stigma” of visiting their GP for mental health issues.’

If the scheme is to succeed, then GPs will need to have faith in the books we are prescribing. That the list has been carefully compiled and has Royal College approval is certainly a good start, and will bring confidence that these titles are a cut above the many thousands of mediocre self-help books out there, but something more needs to be done to really breathe life into the project. At the very least GPs should be sent more information about the books on offer – a brief summary and a personalised review so that we can see beyond the book title. There is plenty of time until we are due to kick off in May, so perhaps someone, somewhere is already penning these helpful hints – let’s hope so.

One thing is certain – the options available to patients with mental health problems who present to their GP are too limited. As Pulse reported recently, waiting times for talking therapies are often too long to be a viable treatment, and medications are sometimes all that is left to fill the void. We should welcome any new alternatives that can bring benefit to patients – and, just maybe, people will be more prepared to see their GP if they think they might be given a book rather than a pill.

This article was originally published in Pulse Magazine (access restricted to healthcare professionals)