Wednesday, 4 January 2017

The Game of Shannon Information

In Information Theory, the transmission of information, or mutual information is defined as the number of bits that need to be transmitted in order for a receiver to be able to predict the message sent by a sender. It's a bit like a game of Battleships: each player must guess the configuration (message) of the other's battleships by trying to place their bombs in positions where the other's battleships will be destroyed. Each move conveys information. The player who succeeds with the least number of bits (i.e. number of turns) wins. Like all games, there are processes of reflexivity which accompany moves in the game; there are "metagames" - games about the game - as players speculate on the different strategies of each other.

Thinking about Shannon information in this way leads me to think about the extent to which the mathematical theory of communication is really a description of a game. This is particularly interesting when we consider "meaning". Some moves in Battleships can be more meaningful than others. The most meaningful moves are those which which reveal where the opponent's battleships are. This can be partly revealed by where opponents don't place their bombs, and indeed, Player B's understanding of player A's game concerns where they think player A's ships are. The purpose of the game is to reveal the constraints of the other player.

In ordinary language, we understand and share many of the basic constraints of communication - particularly the grammar of a language. These constraints, in this sense, manifest themselves in the displayed redundancy of the language - the fact that there are more e's in an English sentence than z's, or that we say "the" more often in sentences than "hippopotamus". The measure of Shannon entropy, defined as the average "surprisingness" of signals in a message, contains within it this notion of redundancy, without which nothing would be surprising. Its virtue is that it is easily measurable since it simply involves counting words or letters. Yet Shannon's formula glosses over the fact that this shared constraint of grammar was, once upon a time, learnt by us. How did this happen?

Here the broader notion of constraint helps us by making a connection back to the game of Battleships. In the game of language acquisition, the objective is similar: to discover the constraints bearing on those who talk a particular language, just as Battleships aims to discover the constraints bearing on the player who seeks to destroy my ships without revealing their own. The language acquisition game is different not just in that it doesn't appear adversarial, but also that there are many kinds of 'move' in the game, and crucially, a single move (or utterance) might be described in many ways simultaneously: with sound, a facial expression, a gesture, and so on. In other words, the Shannon notion of redundancy as "extraneous bits of information" is quite apparent. Such redundancy of expression reveals constraints on the person making the utterance. At other times, such redundancy can serve to constrain the other person to encourage them to do something particular (saying "do this" whilst pointing, using a commanding tone of voice, etc).

At this point, we come back to Shannon's theory and his idea of Information and redundancy. The game of "constraint discovery" can account for Information transmission in a way which doesn't make such a big deal about "surprisingness". Suprisingness itself is not very useful in child language acquisition: after all, a truly surprising event might scare a child so as to leave them traumatised! Shannon's notion of Redundancy is more interesting, since it is closely associated with the apprehension of regularity and the related notion of analogy. Redundancy represents the constraints within which communication occurs. Shannon's purpose in Information theory is to consider the conditions within which messages may be successfully transmitted. The information 'gained' on successful transmission is effectively the change in expectation (represented by shifting probabilities) by a receiver such that the sender's messages might be predicted.

However, communication is an ongoing process. It is a continual and evolving game. We discover the constraints of language through listening and talking to many people, and through identifying analogies between the many forms of expression we discover, whilst at the same time, learning to communicate our own constraints, and seeing the ways in which we can constrain others. Eventually, we might grow up to communicate theories about information and how we communicate, seeking sometimes to constrain the theories of others, or (more productively) to reveal the constraints on our own theorising so that we invite others to point out constraints that we can't see.

Isn't that a game too?

No comments: