Polythetic Assessment: What it is and why we need it

There are fundamental differences in the ways we categorise and group things, including learners. When we design assignments and exams, the point of the exercise is to create a common set of variables against which the performance of each student can be assessed. Effectively the students form a "cluster" around the measurement of these variables.

This approach to clustering around a common set of variables is called "monothetic" clustering.

In ordinary life however, we often don't cluster things monothetically.  When we select our favourite music for one of those Facebook/Twitter challenges which everyone is doing now, the criteria for selection into the cluster of "my favourites" does not depend on common variables. However, intuitively we know that there is some kind of deep pattern which unites all of these things, without necessarily being able to put our finger on exactly what it is. In a sense, the point about these Facebook games is to provide some evidence for what these deep patterns might be, and in the process, who we are in making the judgement.

This approach to clustering is called "polythetic". Its a term used today by the data scientists, but it has a longer pedigree in phenomenology. It was first used by Edmund Husserl to describe "family resemblances" between phenomena which could only be revealed step-by-step over time. Later Husserl's follower, Alfred Schutz, used the term to describe the experience of music in a striking paper called "Making Music Together" (Social Research, 1951)

The problem with monothetic judgement or clustering is that it forces things into a rigid framework. But most things, including student work, can be good or bad in many different ways. For those who argue against Learning Outcomes, the wish is to find ways of making defensible judgements about student work whilst embracing the diversity of the goodness or badness of that work.

More deeply, the difference between monothetic and polythetic judgement is that the former is 2-dimensional and the latter is 3-dimensional. To understand this means to dig-in to the logic of patterns, and understand that no pattern can exist which doesn't feature some aspect of "nothing" within it which segments one part of the pattern from another, and it can be shown mathematically that "nothing" entails looking at the world in 3 dimensions. To move to a polythetic approach to assessment is a bit like introducing perspective and a "vanishing point" into art. Indeed, the idea of "vanishing" is extremely important to understand polythetic assessment. It is assessment about deep patterns in things, not surface features.

I've made a video introducing "polythetic assessment" which describes the importance of measuring "nothing" when we look at phenomena. This work draws on physics, mathematics and biology - but it has direct applicability to educational assessment in ways which I will illustrate in future posts.

If we want more personalised, self-directed, creative learning, then we will need to find new ways to assess. Polythetic assessment presents a starting point from which we can look at the learning process afresh.

The Downes, Siemens and Lamb debate: Two Internets and Two-Cultures

It feels as if there are two internets at the moment, and these two internets are at the heart of a battle between the educational technology thought-leaders (see https://www.linkedin.com/pulse/hippies-now-shop-dispensaries-revolution-has-been-george-siemens/). George Siemens and Brian Lamb are institutionally-focused on the need for what might be called “Internet-1”. Internet 1 is the hair-shirt internet of online education, but perhaps a necessary place as teachers struggle to get their stuff online. “Zoomworld” can be a rather dull place where people are glued to screens like coronavirus masks, gazing at pixelated images of each other, piecing together broken audio, each knowing that the other is feeling the discomfort that all are feeling – but nobody can speak of it. 

 Stephen Downes supports what might be called “Internet-2” - a convivial and sometimes transgressive internet. It’s the internet of Houseparty, TikTok, Youtube, memes of Trump and toilet paper, musicians playing in their bathrooms, or creating coronavirus lyrics to old songs. Sometimes driven by corporations who want to steal our time and sell us things, Internet-2 appeals to a psyche disturbed by the dramatic transformation of the environment. But for the next few months at least, it will be the locus of social creativity.

If you were an 18-year old student looking to go to a University which looks likely to be online until at least Christmas, which internet would you prefer? Students might still swallow Internet-1, pay the money, and turn up in Zoomworld in September. They may worry that if they don’t go in September ‘20, the demand for places in ’21 might make it much harder to get in. They may still believe that their much-vaunted expensive certificates will benefit their careers when all this is over, despite the signs that the world may never be the same.

But lockdown will give them plenty of food for thought about the differences between their online experiences. It may lead them to consider whether they might learn more from the creativity of TikTok or Houseparty, making music videos or publishing art on Instagram, than in the hair-shirt of Internet-1. University leaders and teachers would do well to consider this question too with some urgency: their future may depend on it.

One interesting difference between the two internets is that internet-2 is full of music. In The Glass Bead Game, Hermann Hesse described the eponymous Game’s provenance in an enlightened educational world in the 25th century as: “a kind of highly-developed secret language drawing upon several sciences and arts, but especially music and mathematics, and capable of expressing and establishing interrelationships between the content and conclusions of nearly all scholarly disciplines”. With our 21st century STEM-obsessed eyes, we may look at this and say “mathematics – of course!” But music? Why? It may be to do with technology.

Some of the oldest technologies that exist in our museums are musical instruments. And Internet-2 is full of music: time-based art-forms, games, movies and jokes all conveyed through technologies which would have been considered miraculous a generation ago. Some will scoff. Entertaining and diverting it may be. But educational? No, for that we insist on the hair-shirt of Internet-1. But music tells us something about technology which is missing from Internet-1. Musicians approach the technology of their instruments as a means of amplifying feelings. The musician’s knowledge, their body and their instrument becomes one. Where does this happen in Zoomworld? Behind the predictable monotonous cry of “Can you hear me?”, feelings – which can be more apparent face-to-face – are too easily ignored.

Zoomworld need not be like this. But in order to give it soul, the bonds of campus-based custom and practice need to be loosened and the full capacity of the internet (1 and 2) must be embraced as a path towards higher learning.

Two Cultures?

The resistance to taking Internet-2 seriously as higher learning partly lies in an old debate about the separation between the arts and the sciences. C.P. Snow’s “Two Cultures” criticised the separation of studies of artefacts of culture, and the phenomena of nature. As culture, students learnt about Beethoven or Pink Floyd whilst overlooking the vibrating molecules and communicating cells which underpin the whole thing. The latter belonged in scientist’s laboratory, where conversely questions of aesthetics were ignored. This overlooking of the aesthetics as a facet of nature is relatively recent. The inquiry into music, for example, has been fundamental to scientific development since antiquity. From Pythagoras to Keppler, to Newton (who professed no love for music but still divided the light spectrum into a musical scale), Goethe or Helmholtz, the soul of scientific inquiry was aesthetic. We must repeat Snow’s question: what happened to our universities that they simultaneously squeezed-out aesthetics from science, and science from the arts?

In some ways, lockdown provides a torch for reinspecting this. In lockdown, our lives seem much flatter: the screen for the departmental meeting is the same screen as the lecture. It feels is as if the richness of social experience has been forced into the narrow bandwidth of a transistor radio. Where’s the structure? Where’s the climax? What’s the point? Click here to continue.

Our institutions of higher learning were not designed to be filtered into an online restricted bandwidth, divorced from the campus upon which they established their history, reputation, and their more recent capital investments. As institutions with established structures and practices, the campus restricted the freedom admit the full gamut of online experiences and activities as a threat to institutional stability.

Embracing the richness of technology, the richness of Internet-2, requires a fundamental organisational shift. Pedagogy, curriculum, technology, management and the divisions of knowledge must all be challenged and transformed in the way that Snow wished.

Why is internet 1 so stiff? It cannot be that we don’t have the technology. It must be that something holds back the exploitation and discussion of the full gamut of technologies in the establishment of meaningful encounters with students. Some institutional leaders might be tempted to blame a lack of “digital fluency” among staff. But in reality, the hair-shirt is of the institution’s own making, resulting from the way it organises technology, sets expectations of staff, and its desire to keep the disruptive world of Internet-2 at bay.

Yet in Zoomworld, one has only to drop the formal demands of curriculum and protocol and ask students what they really think is happening to the world, and more importantly how they feel, to open out an inquiry into the aesthetics of current experience as both nature and culture.

Nourishing the Soul of the Person

An online university can be a great thing. But it requires the reconception of a University. John Henry Newman was faced with a similar challenge in the 19th century in establishing a new kind of University. His problem was to bridge the division between religion and science, and in doing so he recognised the need to revisit the fundamentals of what the human intellect was, and the conditions for its growth. Of the intellect he wrote that it:

“energizes as well as his eye or ear, and perceives in sights and sounds something beyond them. It seizes and unites what the senses present to it; it grasps and forms what need not have been seen or heard except in its constituent parts. It discerns in lines and colours, or in tones, what is beautiful and what is not. It gives them a meaning, and invests them with an idea.”

As many commentators have observed over the last few decades, the university has moved far away from this encompassing inquiry into being human in the name of “knowledge economies” and “markets”. The urgency to remedy this is upon us. And we do have the tools to reorganise ourselves.

What we learn, we learn about each other...

We've learnt nothing, it seems. Decades of well-intentioned public education policy has resulted in ecological catastrophe, global inequality, and a political environment across many major countries in the world which resembles the political situation in the 1930s. The latter has been driven by global networks of communication which have been amplified by remarkable telecommunications technology. Now we see our apparently miraculous advances in healthcare and increasing longevity threatened by the systemic effects of economic thinking which creates scarcity, and technologies of global travel which exacerbate contagion.

It's not even that we have learnt nothing. We seem to have forgotten a lot. Our universities,  in an important way, are institutions of memory. They have succumbed to a political toxin where knowledge is for sale, research is currency, and the richness of possibility and speculation has been distilled to curricula and certificates to bought by the young out of fear that their lives will be a misery without them. Our institutions of higher learning have become institutions of forgetting. In my own field of educational technology, I looked at the work going on 10 years ago in (then publicly-funded) JISC, and compared to what is being talked about now. There is no comparison in the quality of the thinking: the 2008 financial crisis was the moment when we forgot ourselves, and let the door open to what is happening now. 

The job of universities is not to sell courses or publish papers in 5-star journals. The job of universities is create contexts for conversations about the future, the present and the past. It is an inter-generational process. We now find ourselves overwhelmed by environmental complexity which many feared would happen: it turned out the people who feared this most were ignored. The complexity and variety of the problems we face requires equal variety in the scientific thinking to address them. But to gain equal variety in our scientific thinking, there needs to be sufficient variety in the ways we organise the conversations in education. If the variety in the ways conversations are organised in universities is impaired, then the variety of the scientific thinking will not be up to the job of the scientific challenges we face. 

The lockdown presents some interesting challenges for education. Universities will be desperate to maintain their business. They will seek to utilise technologies they have ignored for a long time in order to reproduce the practices of the institution online. This will not be a good experience!

But there is nothing about a "context for conversation" that says that 200 people should log in to zoom and listen to a lecture at a particular time. What it really means is that there is a set of conditions for creating meaningful communications between teachers and learners. Individual and small group communication, inquiry-based learning and personalised projects provide the best way to do this. There's really nothing new in this - and it can all be done with the minimum of technology. Short pre-prepared videos to set the scene and text chat will suffice. But it is a pedagogical shift, not a technical shift.

Deep down, it is a shift away from what we traditionally consider to be "teaching". Why prepare huge amounts of content when content is everywhere, and can be referenced with a URL? Why force learners into a one-size-fits-all pathway for the convenience of assessment, when each individual can be supported and encouraged to explore their own interests or be directed down individual paths that mean something to them? Why put so much effort into parading the teacher as a distant talking encyclopedia when the technology allows the teacher to make themselves available for inspection and inquiry so that each student has the opportunity to establish a more personal relationship?

It comes down to a fundamental principle of education: what we learn, we learn about each other. It's not our communication tools which are the problem. It's what our institutions have done to them. Essentially they became tools in a political game.

So now is our opportunity to learn something. And what we learn, we will learn about each other.

What does it mean for a communication to be meaningful?

While ruminating on the monotony and sheer lack of variety of Zoom, I have been reflecting on how many academic friendships I have struck up through email. Just text - when it is done with care and thought - can create profoundly meaningful connections between people.

Academic communities have always operated like this. In the middle ages, monks would correspond with one another, citing references and sometimes sending books. In a time when scholars would only come to know each other through what was written, the potential for intellectual development rested on the written word.

By comparison we have an embarrassment of riches in terms of means of communicating at a distance. And it almost seems be the case that because of this our communication is less meaningful. It's hard to generalise - but it's easy to blame the technology and not look deeper. Yet it does throw the spotlight on what we think happens when we communicate through any medium.

In any communication process, utterances or speech acts must be made. They may be spoken, or they may be written, or they may be constructed in some more sophisticated multimedia form, but they must be made. In order to be made, utterances must be selected. Indeed, not only the utterances must be selected, but also the medium upon which they are communicated. The first question of communication is then, What is the mechanism of selection of utterances, and how is the selection mechanism constructed?

In thinking about the construction of the selection mechanism, one thing is obvious: no utterance is made in a vacuum. Rather, utterances are made to communicate with some other person. If we know nothing of the other person, it is very hard to formulate an utterance. We often see learners who are unsure of what is expected of them to be tongue-tied and unable even to ask questions. Having no insight into the person you are talking to is a conversation killer!

Conversely, being able to make an utterance entails having some understanding - a model - of the person you are talking to. More profoundly, that means that all utterances are made in anticipation of what the other person might say in response to them. Intellectual communication by email or text can be so powerful because the deep ideas which are explored in such communication underpin the understandings of academics of each other. Meaningful communication occurs when mutual expectations and anticipations are aligned.

I've been teaching some students recently using Big Blue Button, and what has been so revealing is that it is not the ability to broadcast my face and voice or Powerpoints has been educationally powerful. The most powerful things in the interactions have occurred through the text chat, but that only occurs when myself and individual students start to understand each other (they are working on projects around educational technology). Its in the text chat where the utterances are selected carefully in order to align expectations with individual understanding.

There was a key moment in these interactions with the students when I asked them "How many of you know your lecturers?" Hardly any of them did. This led to a more profound conversation about their expectations of university and the lack of personal connection with those teaching them. What's striking is that the face-to-face big lectures, etc, are incredibly impersonal. The online environment, if we get it right, can be more meaningful - but not if we try to emulate what the face-to-face environment does!

There's a bigger question here about why education has become so impersonal. It's not that Zoom is crap. It's that what we are trying to do in Zoom or Teams at the moment is mirroring something that was always crap. Here, I think we have to look to the essence of what the education system is geared-up to do: assess. It's assessment which is the thing which ensures that the institutional certificate is scarce (and therefore maintains its value, and the excuse to charge lots of money for it). Assessment underpins the business model. And the way we assess - by breaking things down into categories and forcing people to engage in rather unnatural activities to "jump through the hoops" - has an impact on everything else we do in education. As yet, in education, we don't have a better idea. But we need one.

So we need to look beyond technology in addressing why our Zoomworld feels deficient. We need to look at the constraints which are preventing us from having more authentic discussions. And the principal one is assessment!