Information in motion
How we make meaning over time, space and multiple perspectives.
This is the fourth and last in a series of articles about information (here is the previous article). I am collecting all my notes from years of teaching into these informal articles, updating them as new information comes in and as my views change.
Human scale
This series started by looking at information in, and as, physical phenomena, then in differences and patterns, and then how it infects and influences human and computational systems as an active entity. Now I change perspective to explore the active and subjective nature of people in perceiving and processing information. To do so, I return to a frequent reference, James J. Gibson and his ecological approach to perception.
This means bringing things to human scale. “The world of physical reality does not consist of meaningful things,” says Gibson. “The world of ecological reality, as I have been trying to describe it, does.” We can find raw data in the interactions of particles or wavelengths of energy, but to turn them into “meaningful things,” we need to process and interpret them in some way, individually.
Part of this has to do with language, and Gibson defines his terminology very specifically. Dealing primarily with visual perception, he formulates principles based on the geometry of solid angles, for example, but always comes back to the perspective of the perceiving animal. I mentioned previously, for instance, that he separates an environment into medium, surfaces and substances. When we speak of “planes”, we refer to abstract, platonic forms without colour or depth; a surface, by contrast, is generally opaque, substantial and coloured.
Thus he is not interested in atoms or microscopic organisms, nor astronomical phenomena, but in organisms composed of a minimum number of cells such that they are mobile and utilise multiple senses (or perceptual systems, as he calls them[1]). This rules out plants. The animal, then, “is a perceiver of the environment and a behaver in the environment.”
Change over time
Gibson’s definition of an environment is not limited to space – no medium, surface or substance is static, any more than an animal is. But we have a “geometrical habit,” as he says, of separating space from time.
I discussed in a previous article information as difference, and this generally involves some kind of change over time. This contrasts with a view of information as lifeless and static – something that just sits in a brain or hard drive. Psychologists who speak of “stimuli,” according to Gibson, refer to things that carry no information about their source. And a stimulus is also usually regarded as temporary – if it persists, a sensory receptor tapers off and we become sensorily adapted to it: we don’t notice it anymore. Gibson rejects the language of senses and and receptors, choosing instead perceptual systems and information available to us to pick up from patterns in the environment: “Only stimulation that comes in a structured array and that changes over time specifies its external source.”
A structured array of environmental information flows in time. We perceive information about an environment in difference and change: specifically in events – mechanical, chemical, changes in surface, lighting, and so on. And paradoxically, in perceiving what changes, we also perceive what persists – in Gibson’s words, “What is invariant does not emerge unequivocally except with a flux.”
Exactly what we perceive, in this view, are the changes that occur in the surfaces of things attached to and detached from the environment. The surface, more than medium or substance in an environment, “is where the action is,” according to Gibson. This is where we see the edges and reflections of fixed and moving objects, where mechanical collisions, chemical reactions and state changes generally take place, and where vibrations propagate outward into the medium. These Gibson calls ecological events. (I would include the spread of a virus, through contact or dispersal into the medium, as an ecological event – see here.)
For animals, the medium has an intrinsic polarity of up and down due to gravity. Radiant light generally comes from above. An enclosed medium can be “filled” with light, sound and chemicals, some of which may emit odours. The medium is generally transparent, as opposed to surfaces which are generally solid. It thereby affords seeing things, and it affords movement.
Position & movement
Let’s take a brief detour back to physics, to bring together space and time in relation to information and observation. Hermann Minkowski was Albert Einstein’s teacher in Zurich, and just before he died in 1908 he added to Einstein’s theory of special relativity with his own theory of spacetime.
One of my own students, Maria Euler, trained in physics and introduced us to this theory, and henceforth I encouraged students to think in terms of spacetime and not just space because the latter is never static.
The diagram at the top of this article illustrates Minkowski spacetime. (My students know that I love a good diagram.) I think it’s fairly self-explanatory, except that the three dimensions of space are flattened into a hypersurface – if you plot two dimensions on a graph, you get a curve; add a third and you get a curved surface. This represents everything an observer (or for our purposes, perceiver) can perceive at a given point in time. Time is shown moving upward, so imagine that hypersurface moving upwards and changing its shape over time depending on what is observable. The observer is at the centre, so this view is specific to an individual observer. This is where relativity comes in: the distances between things are shown in both space and time – as spacetime intervals – relative to each observer. So you can imagine lots of these cones in different places and times – one for each observer.
Now returning to Gibson, he says, “Any point in the medium is a possible point of observation for any observer who can look, listen, or sniff.” Therefore, from any given point, one can pick up information about the world that surrounds it. This is where he merges with Minkowski, as when he says:
If it is assumed that no two observers can be at the same place at the same time, then no two observers ever have the same surroundings. Hence, the environment of each observer is “private,” that is, unique. This seems to be a philosophical puzzle, but it is a false puzzle. Let us resolve it. One may consider the layout of surrounding surfaces with reference to a stationary point of observation, a center where an individual is standing motionless, as if the environment were a set of frozen concentric spheres. Or one may consider the layout of surrounding surfaces with reference to a moving point of observation along a path that any individual can travel.
So perception implies movement, through both space and time, “because all observers are animals and all animals are mobile,” Gibson reminds us. An animal is not only a perceiver, but a behaver in the environment. Movement is then defined with reference to a comparatively rigid environment, and we find our place, our direction and our way through perceiving a succession of information.
Because we move from one point of perception to another, these points are connected by lines of locomotion; not geometric points and lines, but points of view, and paths of movement – that’s an important distinction, since we’re not talking about abstract spaces but actual places situated in time inhabited by perceiving humans. Through movement, information changes, and each point of perception is unique, and potentially equivalent with all other points. Because no two individuals can be at exactly the same place at the same time, but any individual can potentially stand in all places, and all individuals can stand in the same place at different times, as long as the environment is relatively persistent, all its inhabitants theoretically have an equal opportunity to explore it.
Subjectivity
But this is indeed all still at a theoretical level; let’s begin to get more specific. Animal locomotion, according to Gibson, “is not usually aimless but is guided or controlled – by light if the animal can see, by sound if the animal can hear, and by odor if the animal can smell.” In other words, the information we pick up as we’re moving through time and space guides and controls our movement. See the previous article in this series for an in-depth discussion linking this to a computational perspective.
But even without moving, we are constantly moving. When I teach moving image to postgraduate students, one of the first things I get them to do is to watch each other’s eyes as they watch a film: even when watching a single screen, our eyes are constantly moving, in two ways:
First, our eyes constantly dart back and forth. These tiny movements, called saccades, take place several times each second. Neuroscientist Jeffrey Zacks characterises this as a kind of subconscious questioning: shifting our gaze from one part of the world to another “gives us answers to our visual questions.”
Second, we blink – up to 50 times a minute, again unthinkingly. What does this have to do with the visual perception of information? Well, between saccades three or four times a second, each move taking about 1/12 of a second, and each blink taking several tenths of a second, we are functionally blind for an astonishing one-third of our waking lives, according to Zacks (not even accounting for head movements, of which there are many as well, and selective attention, detailed in the last article). This means we miss a lot, and constant movement helps to fill in the gaps.
Duality
Even without any movement, we automatically see from multiple perspectives, because we have two eyes – a perceptual system, as Gibson calls it. Alternately close each eye while you’re reading a single word of this text. Nick Sousanis details this in his excellent book Unflattening, pointing out that neither eye gives the “correct” view, and – in relation to movement above – seeing is like walking (since we also have two feet), as a constant negotiation, a balance between two different sources. (Zacks notes that we also have two hemispheres to our brains, which each interpret information in different ways.) Sousanis shows how viewing the sun from different points on Earth enabled us to calculate the circumference of the planet, and viewing other stars from two different points in our movement around the sun let us calculate their distance from us.
One exercise I do with students is to get them to draw the same object or scene from two different perspectives — this adds a kind of exponential amount of information I think, turning a static scene into a story.
Shifting perspective
Just as the world is never exactly the same at every time and place, no two individuals perceive things the same way, even if they observe a comparatively static environment from exactly the same spot. And even the same person changes over time. Perception of any available information, therefore, says something about the observer as well as the environment.
To illustrate this, when I run workshops on observation, I start by getting students to simply draw what they observe. This makes a direct connection to movement in multiple ways. First, the connection between eye, brain and hand bypasses logical thinking; this has been discussed eloquently by Lynda Barry, Betty Edwards, Nick Sousanis and others. Second, in doing so it highlights Gibson’s emphasis on surfaces and the solid angles we view them from, and away from the conceptual nature of things as we define them in language; according to the illustrator Shaun Tan:
“You discover how confounding the world is when you try to draw it,” Tan says. “You look at a car, and you try to see its car-ness, and you’re like an immigrant to your own world. You don’t have to travel to encounter weirdness. You wake up to it.”
Third, it brings framing to the fore: in choosing what to include and exclude, it shows how even the most seemingly objective scientists or journalists have to make selections. I discussed selective attention in the most recent article in this series.
Fourth, it’s difficult – especially when drawing things in motion, like people. For an extreme amount of difficulty, I sometimes have students draw from the screen, while watching a film; even with a relatively static film like Chris Marker’s La Jetée, this is a huge challenge – try it for yourself.
On one hand, drawing often freezes a scene, artificially stopping someone in mid-stride for example. Some artists, like Louis Netter, develop a kind of photographic memory for this, remembering a mental snapshot of someone they see only fleetingly, then drawing them afterwards; others, like Jayoon Choi, are good at quickly capturing people’s poses and personalities as they’re observed. On the other hand, artists from the Ice Age up through Duchamp have been skilled in depicting motion in a static medium.
Mental movement
What happens when we perceive movement itself? Zacks cites experimental evidence that watching someone else move, even in a film, provokes a complementary reaction in the part of our brain that regulates our own movement – we mentally rehearse the same action. The urge to move is usually suppressed, but he notes that many children haven’t yet been conditioned to do so, and many adults will subtly move in reaction to perceived motion – flinching when something comes at them, looking or turning their body in the direction someone else is oriented. Knowing this, you can exploit it – lecturers for example can both influence and be influenced by their audience, responding for example to gaze and body position.
This holds for facial expressions too, and we even imitate someone else’s accent after talking with them for a while. This is social information; to operate effectively in human society we need to understand people’s intentions – and sometimes influence them. Again, the last article discussed this. Zacks points out, “A lot of the information you need just to walk the aisles of the grocery store or carry on a conversation with your lunch partner is not physically present in front of you.”
To inform
Besides observing and influencing actions, the other main way we can change our own or someone else’s view is through dialogue, in some form of language. According to Sousanis, “The means by which we order experience and give structure to our thoughts – our languages – are the stuff we breathe in and a sea we swim in.” Recall Gibson’s notion of medium; here it’s applied to the social environment. If Marshall McLuhan was right that the medium is the message, if language is a medium, what message does it carry, in itself, apart from all the individual messages it encodes?
How about the communications media we use today – primarily digital, compared to McLuhan’s time. When information is transmitted instantaneously, copied endlessly, and held everywhere at once but in no one’s memory, it inhabits a timeless space which distinguishes it from insight, says the philosopher Byung-Chul Han. There is thus a problem with information as a concept that characterises our age.
What if we compare this to Gibson’s medium – air? Recall here Walter Van de Velde’s characterisation of information as a living thing that breathes in our attention, discussed last time. Let’s therefore elevate information to the status of a participant in a dialogue. Just as information can prompt action in people, as detailed in the last article, from a philosophical perspective, we create reality every second through our very perception, through the Gibsonian process of separating entities from one another in the world.
We can categorise actions in linguistic terms, but language conversely influences how we perceive and act: “The medium we think in defines what we can see,” says Sousanis. To make sense of the relation between language and action, I’ve been using a conversational model that comes from cybernetics – the science of observation. It’s detailed here, but in general, it separates (linguistic) descriptions from actions between two participants, with two-way feedback mechanisms in between. This helps to show how we can use information to stimulate action, and conversely how we describe or rationalise actions.
When it comes to language, it especially takes its form in stories – narrative is a fundamental way we understand the world, interpret causes, effects and intentions, and predict future actions. Zacks describes what he calls event models, which are like internal, cognitive interpretations of Gibson’s ecological events that we perceive effortlessly.
These work across both language and actions – as Zacks details, they apply to written or oral stories as well as films and live action we perceive. But they work in very different ways. Movies provide raw data about changes in surface, and Gibson devotes the last part in his book to how we perceive these rather impoverished (framed and flattened) views of the world. “On the other hand,” Zacks points out, “books almost always tell you about objects’ and people’s names and categories.” Show, don’t tell, goes the advice to writers, but using language, notes Zacks, “it is almost impossible to mention an object, person, or animal without categorizing it.”
Watching a movie or reading a story is thus a form of seeing from someone else’s perspective – more directly in a movie, in the ways it is filmed, or mediated by descriptions in a story, and more broadly in the languages that each medium uses, whether the film conventions of framed shots, montage, lighting etc; or the story’s use of words – themselves “invented” (or evolved) by other people. According to Sousanis, just as binocular vision produces depth, multiple perspectives produce a different kind of depth.
Here we arrive back at a more traditional view of information, as articulated by Claude Shannon (and discussed in the first of these articles). The philosopher Federico Campagna puts it nicely: “As a passage of information, communication implies a traceable movement between nodes that can be mapped linguistically.” [3] Gibson makes a subtle distinction:
We are told that to perceive something is to categorize it, to distinguish it from the other types of things that it might have been. The essence of perceiving is discriminating. Things differ among themselves, along dimensions of difference. But this leaves out of account the simple fact that the substance, place, object, person, or whatever has to last long enough to be distinguished from other substances, places, objects, or persons. The detecting of the invariant features of a persisting thing should not be confused with the detecting of the invariant features that make different things similar. Invariants over time and invariants over entities are not grasped in the same way.
He suggests that the perceptual system simply extracts the invariants from the flow of information. To perceive distinct things in the world, he says, the perceptual system must abstract the invariants. “Abstraction is invariance detection across objects. But the invariant is only a similarity, not a persistence.” Having ideas is a fact, he believes, but it is not a prerequisite of perceiving. “Perhaps,” he wonders, “it is a kind of extended perceiving.”
In fact this question was addressed centuries ago by the philosopher Avicenna. His “floating man” thought experiment challenges us to imagine we have no limbs or senses, just somehow float in a medium. What do have in that case? Only our own existence – we exist but do not experience. As described by Campagna, this is the opposite of Descartes’ well-known maxim – not “I think, therefore I am,” but I simply exist first, and this precedes my thinking, and perceiving.
Campagna therefore demonstrates philosophically what Gibson shows scientifically – that language is limited in capturing all that we perceive in the world, and logic doesn’t suffice to extract meaning out of all the raw information we perceive. He shares with Gibson a view that we each construct our own reality every moment, without conscious thought. But going further than Gibson, he sees more than meets the eye.
To summarise so far, we already see from different perspectives – via our two eyes, through movements by our eyes or through the world, or through another’s eyes. We perceive environmental information without thinking, and we interpret it through language. But how do we see what’s not there? What’s the “more” there than meets the eye? What’s between the lines? What might be rather than what is? As Gibson says, “a concept of information is required that admits of the possibility of illusion.”
Beyond information
Cue Marcel Duchamp. He was also heavily informed and influenced by science, but then twisted its, and our, perspective through his work. He applied the term “antiretinal” to his art because he opposed a simplification of vision, a mere classification of facts – what you see is what you get.
Duchamp wanted us to see what’s in front of us in a new way, to find a hidden dimension in vision. As Penelope Haralambidou eloquently states, he was interested in “appreciating, through the intervention of the eye, what lies beyond the screen of memory and prejudice and being filled with wonder by the much more meaningful shadowy side, illuminated by the seduction of a revealed truth. If only we take the trouble to look.”
What is “beyond”? Where is this hidden dimension? String theory, Sousanis points out, finds hidden dimensions curled up tightly within our everyday reality. Campagna sees it behind each entity in the directly perceivable world, “as if under a veil,” a part of each thing that resides elsewhere, or points to something else, somewhere else. This is the realm of the ineffable – that which cannot be described in words. Campagna’s philosophy shares with physics a view of the world as a kind of shadow cast by higher dimensions of reality. Just as a 2D plot of points becomes a 3D surface in Minkowski spacetime, can we peel off the square of flatland to see the reality we take for granted from outside?
Yes we can, but we need to employ some mental tools – whether the thought experiment of the scientist or philosopher, the imagination of the artist, or simply memory: Gibson notes that we are aware of surfaces that we’ve seen before, and we can imagine and depict those that may never exist.
We can see the depth that comes from taking a different perspective, or multiple perspectives, as discussed above. Turn the world upside down, look in the folds, change your scale, defocus your eyes, change your mind, expand your thinking. What Herbert Marcuse called “the critical dimension” also counts as an added dimension, and following Karen Barad, I like to think of criticality not in terms of critique (that’s too easy) but as in “going critical” in physics – reaching critical mass.
In The Architectural Parallax, Slavoj Žižek sees contingent spaces that emerge as possibilities through the transition between viewpoints — can we imagine a virtual architecture in the gap between two subjective perspectives?
Another dimension comes from a doubled way of seeing. Just as two eyes, or two views, weave together a new hypersurface, Campagna points to the philosopher Ernst Junger, who described a kind of “stereoscopic gaze” in which we see two things at once – for example a face in the moon or shapes in the clouds, without renouncing the Gibsonian empirical evidence of solid angles and the reflection of light. This is a view that accepts stories as well as science.
We can expand this multiplicity of perspectives to Gibson’s focus on the animal more generally, and broader still to other nonhuman entities. What does a rock see? What does it want? Or a tree, a galaxy, a mushroom? Some things we can use as tools to see further – climbing the tree, for example. Other things we look through or attach to our bodies, and this capacity, Gibson says, “suggests that the boundary between the animal and the environment is not fixed at the surface of the skin but can shift. More generally it suggests that the absolute duality of ‘objective’ and ‘subjective’ is false.” Sousanis notes that being with a dog, with its heightened sense of smell, “one can access other dimensions of experience.” Similarly, a person in a car or on a bicycle sees the world differently, travelling at a different speed and having new affordances and capabilities. Is a human driver alone responsible for what happens while driving? What does a car want, and why does much of the human world seem to be built more for cars than humans?
Information in trouble
As we arrive at the end of this series on information, we have to accept that the very notion of information itself relies on its own shadow: the absence of information. In previous articles, I discussed a lack of information where there is a lack of variation, but that was a difference in quantity, not quality. Here I am referring to the perception of something and the creation of meaning where no perceivable information exists. Call it apophenia if you like, but we have an innate drive to make sense of things, even where sensory information is absent.
This is the key distinction between the Gibsonian perspective and a philosophical perspective. Gibson is not interested in the sensory deprivation of Avicenna; he describes the perceivable environment of human animals. He began to describe the limited information we get from flattened images. But digital media explode this notion – we have more information than ever but we perceive it in even more deprived, disembodied ways (yes, even with technologies like VR). More information, but paradoxically, even though it can be transmitted with less loss, thanks to Shannon, the sheer quantity of information concentrated into the constrained space of the screen simply creates even more noise.
This is where Han distinguishes insight from information. But here is where he, in turn, departs from fellow philosopher Campagna: in how they see the role of time in the construction of meaning. For Han, information is timeless and insight takes time; for Campagna, insight is to be found exactly in the single point in space or the instance of time, for this is where we can see the timeless. For both philosophers, then, we can say that the information is the noise. Information is the problem.
Look, then, into the light.
Illuminations
Did you know that people (and all living things) emit light? Researchers found this was related not to heat but to time, specifically bodily rhythms. Ultrasenstive camera equipment is needed to detect the very weak photons emitted, but let’s speculate that this light can by subtly detected by other living beings, by means we are not yet aware of. Were this the case, it would call into question Gibson’s distinction between radiant light and reflected light. To him, we cannot see light directly, only its reflections off of surfaces –we see information in light. But what if we see light itself as information?
In this sense we perceive light subconsciously, before we distinguish surfaces and differentiate things in the world. It is thus the precondition to interpreting visual information. From a philosophical perspective, too, we perceive the gestalt or ineffable nature of things before we categorise them. Campagna calls this “presential knowledge” or knowledge by presence, “in which the knower, the known and the process of knowing become one and the same thing.” In this sense it corresponds to Gibson’s notion of direct perception — as Campagna describes it as “that ‘light’ which enables all other forms of perception, by making their objects ‘visible’ to our cognition. Self-knowledge coincides with knowledge of the ineffable lying at the heart of the world, since both of them are the ‘luminous’ precondition by which any other type of understanding is possible.”
But unlike the things that are illuminated, which we can grasp through descriptive language, this light itself represents pure ineffability. “As such,” writes Campagna, “it is at once the foundation of descriptive language (which would be impossible in condition of ‘darkness’), while also escaping its grasp.” We can make sense of the objects of descriptive language through sensory perception and the rationality of language, “yet such levels disclose to us only a specific layer of reality — and not that which is innermost.” This innermost and indescribable nature of things we can liken to illumination—as if all things give off a certain glow just as living things do. In the metaphorical sense, we refer to knowledge as enlightenment or illumination: “Do you see?”
Gibson recognises the presence of illusion, and says that a test of whether something is real or not is movement: “Whatever goes out of sight as you move your head and comes into sight as you move back is a persisting surface. Whatever comes into sight when you move your head is a preexisting surface. That is to say, it exists.” The method of illumination I am discussing however, metaphorical it may be, is just one of the useful fictions we employ to make sense of the world. All our classifications imposed and constrained by language are fictions too—they make communication possible, but not necessarily understanding. They merely represent things, and cannot completely describe things as they are —nor the part of things that may exist in dimensions inaccessible to us, like the wavelengths of light we cannot detect.
Gibson says that we can detect the difference between representation and reality upon closer inspection: “The most decisive test for reality is whether you can discover new features and details by the act of scrutiny.” But to philosophers like Campagna, “reality” exists beyond what is directly perceivable. To see additional dimensions requires a shift of perspective which looks equally inward as outward: the link between dimensions is the individual.
In Gibson’s words, “To daydream, dream, or imagine wishfully (or fearfully) is to be aware of surfaces or events that do not exist or occur and that are outside the limits of possibility.”
Notes
- Regarding the senses as systems, Juhani Palasmaa believes that touch, not vision, is our main sense, and looking at something, we imagine touching it.
- Campagna, F. (2021) Prophetic Culture. (Bloomsbury — in press)
