Rethinking 'Impact'

At the #CETIS12 session on visualisation, one the of principal reasons for visualisation was suggested as being “as a way of showing the impact of the JISC interventions in the sector”. The pretty visualisation pictures were deemed “better than a report saying ‘this has been used by x, y and z’”. It probably is better than doing that, but at the same time, we might reasonably worry that visualising the impact of interventions is no more honest or productive than writing a report about it; indeed, it might be worse because it is more visceral. I joked that we should visualise the impact of research in visualisation (didn't go down too well :-/)

I have argued that visualisation and analytics is about decision and control, and fundamentally about attenuating an increasingly rich information set which by its sheer diversity threatens meaningful collective decision-making, and which necessitates some technique of making sense of it. But I have said all that elsewhere... (see http://dailyimprovisation.blogspot.com/2012/01/knowledge-and-performance-art-of.html)

What the conference discussion set me thinking about was the notion of ‘impact’ – particularly as I’m particularly interested in social innovation more generally at the moment, and the ways in which society might organise itself to reward effective social innovation.

The notion that ‘impact’ can be evidenced by external communication networks puts undue weight on the externalisable aspects of innovations. But the internal aspects of projects are usually the most significant: the major impact of any funded project is usually on those receiving the funding. Very often  the confidence and experience gained can lead to growth in individual social capital and increasing ability to make a difference in the organisations in which they engage (often coupled with internal promotion, etc). That sort of impact is a much 'slower burner' than the immediate communication networks suggest.

Not that I think there is a licence for not engaging with the community, or trying to find new ways of getting people to adopt your stuff – those are important reality checks, and avoid the danger of individuals disappearing off in impractical dream-worlds of interventions. But at the same, what goes on in peoples heads is of great significance, and acknowledging this will help us to understand the deeper nature of social innovation. The problem is we don't know how to do it. We only know how to visualise communication networks. But what we want to know is "what do projects mean to people?"

This relates to thinking about informational transfer, the growth of meanings and ‘entropy’ which I blogged about http://dailyimprovisation.blogspot.com/2012/02/cybernetics-of-meaning.html. That sounds very ambitious, but the potential for its practicability is enticing me. Although this is difficult stuff, and very mathematical (I'm having to brush-up my maths quite a lot here - thank you Khan Academy!) I’m struck by the three equations of anticipatory systems by Daniel Dubois (see http://dailyimprovisation.blogspot.com/2011/10/artificial-qualitative-data-from-quants.html). These equations  are ‘recursive’ (i.e. each state depends on the previous one); incursive (each state depends on the previous one and the current state); and hyper-incursive (each state depends on projected 'future' states).  Each of these equations is recursive and produces a ‘logistic map’.  The principal feature of the logistic map is that over time, the entropy increases.

The intervention situation of social innovation is such that meanings proliferate, and the selection of meanings within a society is usually complex. Moreover, individuals struggle to negotiate different meanings as they move through different social contexts: classically, school children struggle to negotiate the meaning constructed within the school with the meaning constructed online.

Coming back to 'impact', whether an effective intervention (i.e. one with impact) is a technology or an explanation (conceptual development) it results in some sort of communication and the effect of that communication is to change the situation with regard to the selection of meanings within a social group.  Whilst for some conceptual development, this can be seen as the growth of a social network, but what is important here is to be clear as to what we are seeing.  What is produced is, in effect, a reduction in entropy, and our next question must be “what is causing it?”

In order to consider that question, we need to look deeper at the individuals engaged directly in projects and practice with people around them. Here it is instructive to look at a couple of hypothetical project scenarios:

1. The project with 'fixed ideas' (i.e. it has a theory which doesn't change in the light of practice)
2. The project with no ideas and flexible methods

The first scenario is obviously unrealistic (although aspects of it do exist!), whilst the other two are more plausible. For people with fixed theoretical ideas, there is high certainty (low uncertainty) over the way they react to certain situations. This may be characterised as low entropy - we know what they will say! This is in comparison to the project with no ideas and flexible methods – there there is high entropy with some uncertainty over the selection of possible meanings.

Suppose both project 1 and project 2 engage with a real-life social domain X. There now follows a process of communication-making between each project and the domain X, within which the entropy of each entity interferes with the social domain. What happens?

The people in Project 1 are unlikely to be changed if their theory is so weighted to maintaining the status-quo (although of course, if the theory is brilliant, this is not a problem!), but they will have an effect on the social domain they interact with. This need not be a bad thing, as the social domain reorganises itself in the light of the interventions of project 1 (although [particularly if the theory is less than brilliant!] I'm sure project 1 will drive them nuts!). Project 2 is likely to be changed by the domain: its entropy will be changed in the light of the interaction (probably lowered). The self-organisation of the social domain may show some knock-on effects of this lowering of entropy. But in terms of the observable social effects, it may be project 1 which has the maximum 'impact'. Yet, the change in entropy within project 1 is non-existant. In project 2, there is change in both the project team and in the social domain.

The point of this is that when we look at communication as changes in entropy, patterns emerge in different types of projects. There are fewer patterns than there are different types of project, and yet these patterns reveal meaningful project interactions which we can all recognise. Moreover, I think examining these patterns of entropy can allow for reasonable forecasting as to the likely future impact of projects. Patterns of change of entropy in the project team will no doubt lead to new and transformed projects, and new patterns of interaction. The lack of patterns of change in the project team of Project 1 will make it likely little will change in the future, and indeed social reaction to previous interventions will make future ones less likely to succeed. (I believe we have seen that sort of thing happen a lot recently)

This all needs more work to flesh out in full... Measuring impact in an accurate way is fundamental to social innovation. We really need to know what works, why it works, reward those who make it work, and spot the 'fixed thinking' projects that might look great on the surface (even visualised social impact!), but deep down are going nowhere.