Images, Causality, Disassociation, Interactivity and Videogames

I’ve got an eight month old. Watching a baby come to terms with the world can teach you a lot of things. For instance, and as a kind of hors d’oevre, consider the word “shush.” To an adult human being, its an imperative verb which indicates that you should be quiet. To a baby it resembles the sound of blood rushing in the womb and is, therefore, supposed to be calming. As a baby learns that sounds can have arbitrary meaning, the “shush” as simulation becomes the “shush” as symbol – the baby comes to appreciate that we can mean things with sounds we make.

My baby spends a lot of time feeling the texture of things. In particular, he’s interested in pictures in books, over which he carefully draws his pointer finger, alternating between the finger tip and scraping with the fingernail. Its not too hard to see that he is curious about the difference between images of things and things themselves. In particular, he seems to have cottoned to the fact that things themselves feel a certain way when you touch them whereas mere images feel like paper or laminate or cardboard, and are more or less undifferentiated qua image with respect to feeling.

When I dwell on this interest, it strikes me how marvelous images really are: they represent a profound collapse of the ordinary causal relationship between light entering our eyes and the objects with which that light has interacted. Wood grain looks like wood grain because it has the physical structure of wood grain. Its dark, striated areas appear as such because the material is ridged, casting some parts into shadow with respect to the source of illumination. A photograph of wood grain inherits the visible properties of the object while it separates them from an immediate cause. The visual aspects of a photograph can be easily manipulated (particularly in the modern era) without changing the way the photograph feels, but most modifications to actual wood grain meant to accomplish a visual change will also result in changes in the physical structure of the object. Our brains, of course, evolved in a context where this relationship between the way we perceive things and the underlying structure of the things themselves, is often strong. This is why when we see a piece of wood we expect it to feel like a piece of wood. It’s probably why my child is so interested in touching pictures in his books: because the breakdown between the visual perception of the thing and any obvious physically relevant structure is novel.

Part of the power of images is related to this detachment from material cause. Things themselves only ever depict (in our senses) that which is literally possible. Images can depict whatever they are designed to depict, whether its causally plausible or not. A normal person has the visual form of a person on account of the fact that they are made up of bones, muscles, fat, etc, that they have a certain mass and weight. When a human bends their knees and leaps into the air, the height of their leap is, ultimately, a property of all these material causes. Superman,  however much he might resemble a person, can leap tall buildings with a single bound, because the resemblance is, in a sense, entirely incidental. A comic book merely depicts physics and thus may take liberties, while a 100 meter dash is physics. Images can exploit the fact that that which is depictable is much more various than that which is possible.

To take a lurching step towards the point before my baby wakes up from his nap: technology in general has this property of obscuring the relationship between cause and effect. Technology can even be understood primarily in terms of the careful manipulation of cause and effect to accomplish what might otherwise be an unlikely outcome. From this point of view a computer is almost literally a cause/effect obfuscator. It presents to us, the user, a two-dimensional interface on which almost any cause and effect relationship can obtain at all. A real xylophone has the property that larger blocks vibrate at lower frequencies, and so a necessary material relationship between music and the structure of the xylophone appears. We can easily imagine a simulation of a xylophone where the relationship between apparent block size and the sound each block makes when struck is the opposite or totally random. Take the piano as an example somewhere in between: its keys are all the same size: the strings which produce the sounds are hidden behind the curtain, so to speak. We can’t as easily infer from the piano that sound is deeply related to vibration, which is related to mass and energy. Computers are the apotheosis of the movement between the xylophone and the piano: their inner workings are, at the human scale, so subtle, that no amount of inspection with the senses can reveal how cause and effect are tangled up inside them.

Armed with these insights, we can put on our game designer’s hat and begin to build up a new way of thinking about what precisely we are doing when design digital interactive systems. I’d like to make two points: the first is that we often feel alienated from experiences when there is a disconnect between the apparent causal structure of those experiences and their actual evolution in time. A good example of this is those old physical racing toys you sometimes still see: a steering wheel controls (by virtue of a connected lever) a plastic car while the image of a road, with obstacles, printed on a loop of paper, is scrolled through a viewing window. The player is expected to avoid collision with these obstacles by virtue of their own understanding of the implied relationship between the objects: cars crash when they strike things like trees or other cars. We quickly grow tired of these sorts of games, not just because we are expected to enforce the rules ourselves (which is also true of games like Chess) but because the causal relationships they do embody are trivial compared to (and distant from) the causal relationships they appear to embody.

The point is that, if we want to engage players, we should provide simulations of causal relationships which are meaningful and we should avoid both acausal elements (like pure randomness) and discrepancies between depictions and causality.  If the presentation of our game suggests, by reference to physical processes with which we are all familiar, that a particular causal relationship is in force in our simulation, then we ought to make that relationship present or we should eliminate the appearance of that relationship from the presentation.

Canny readers will probably recognize that this goes against philosophies like “juice it or lose it,” which seem to suggest that the experience of play is actually enhanced by the intense elaboration of the appearance of our game elements. A more nuanced position can be developed, however: we can and ought to feel free to elaborate on the image our interactive system presents precisely in those ways which underline the causal relationships which our system embodies. When a ball strikes a wall, its probably good to indicate that with sound, dust particles, a shaking screen. On the other hand, if we do elaborate 3d modelling of rocks falling down a mountain, but they don’t interact with our player’s avatar, then we’ve introduced the appearance of a relationship that our system fails to deliver on.

None of this is to say that such appearances might not lead to more saleable products or that they might not provide pleasure to players. That leads me to my second, moral, point. We, as game designers, ought to respect our players by giving them interactive systems which communicate clearly about the relationships they embody for exactly the same reasons that we ought to communicate honestly in real life or in any other art form.

This isn’t to say that our simulations have to correspond to reality or be as realistic as possible. On the contrary, if we wish to explore systems which deviate from reality with our players, we must take even greater care to harmonize the representation of those systems with their underlying structure. We might dazzle players for awhile with elaborate audiovisuals, but unless those operate in concert with the causal structure of our games, we’ll almost certainly have wasted their time (or, at the very least, missed an opportunity to provide real interactive value.)