Wednesday, 22 November 2017

Diabetic Retinopathy and Adaptive Comparative Judgement Grant Success

Very shortly after I started at Liverpool University, I was invited to a meeting with a doctor from the Eye and Vision Science department of the University who was also a surgeon in the hospital. He explained his passionate desire to do something to prevent blindness in China by implementing a proper diabetic retinopathy screening programme. No such programme currently exists, and there is much ignorance about the condition where there are no symptoms (only a retinal scan can reveal problems) and blindness is sudden and irreversible. The scale of the problem is staggering: there are 110 million people with diabetes in China.

Discussions had got as far as thinking that a MOOC might be the thing to do to train people to diagnose the condition by grading retinal scans. I said this probably wouldn't work, and that the real issue was finding an effective way to deal with the complexities of scale of the problem. The challenge of diabetic retinopathy grading is a straight-forward cognitive problem. There are numerous initiatives (including in Liverpool) to use machine learning to do it - but these attempts have limited success. The sensitivity and specificity  of the diagnosis is critical (i.e. ensuring that false positive and false negative results are minimised) - and the machine learning does not always perform well - although it can improve if it is effectively connected to human learning.

The problem of grading is one of assessment on the one hand, and hierarchy on the other. Experts do grading, and experts have to be trained. The scale at which educational assessment now operates has led to a search for new models of assessment and creative uses of technology. Adaptive Comparative Judgement is one of the most interesting. It enlists a large group of assessors to make simple, low-stakes judgements about which of a pair of artefacts (student work) is better. It produces a ranking from which grades can be established. I asked whether grading by an expert could instead be ranking by a group. I suggested that if this was the case, then the complexity of scale of China could be managed by a crowd-based approach using Adaptive Comparative Judgement. Fortunately for me, this idea completely transformed the discussion - particularly in the vision of the doctor leading the project.

An EU bid followed in 2015 which was unsuccessful, but served to stimulate interest across a consortium, and made the connection between the ACJ, Blockchain and xAPI. This year, I joined a group in Liverpool going for a "long-shot" bid to the EPSRC for £1m to develop a training programme based on ACJ, coupled with machine learning and the development of a new low-cost scanning device. The EPSRC had 150 submission to work through and could only fund a handful of projects. It was a long shot.

Well, it looks like it wasn't such a long-shot after all! I suppose what this is making me think is that thinking remains the most important thing in universities. Universities need thinkers, not people who are going to tow a corporate line. The disaster of managerialism and marketisation have done their best to turn many universities (I think particularly of my former institution, Bolton) into fiefdoms where thinkers are sacrificed like heretics of the "corporate religion".

A powerful and simple idea can go a long way. 

No comments: