Camille Utterback exposits "embodied interaction with symbolic spaces" – the body and language of digital art.
All forms of "interactive text" demand a physical body with which to interact. When we use the now-common interface that consists of a mouse and keyboard as input devices, and the computer screen as display mechanism, it is easy to forget the body whose eyes perceive the screen, and whose hands and fingers manipulate the mouse and keyboard. In her book How We Became Posthuman, N. Katherine Hayles (1999) has eloquently explored how "information lost its body." Hayles investigates the theoretical, historical, and literary maneuvers through which modern society has dissociated information from a body or medium. The consequent elevation of abstraction over embodiment is mirrored by a corresponding lack of computer interfaces that meaningfully engage our bodies with the information and codes represented in our machines. The degree to which our physical interactions with machines is impoverished is illustrated by a common saying in the university department where I teach. The saying describes the "human-computer interface" from the computer's point of view -- "when the computer stares back at you, it sees you as one eye and one finger." This saying is originally attributed to Joy Mountford.
With much of my artwork -- in both traditional and digital media -- I have attempted to draw attention to the connections between human bodies and the symbolic systems our bodies engage with. The digital medium interests me because it is a perfect site to explore the interface between physical bodies and various representational systems, be they language, the linear perspective used in three-dimensional rendering, or the various forms of computer code itself. In my digital works my strategy for this exploration has been to develop interfaces that honor and engage more of the body than just "one eye and one finger." Interfaces, by providing the connective tissue between our bodies and the codes represented in our machines, necessarily engage them both. How and to what extent new interfaces may engage the body, however, is up for grabs.
Practical interfaces are about maintaining the user's sense of control. In this scenario representations on screen must respond to the user in a logical and predictable way. Artists can explore other possibilities. In this essay I discuss interactive works by myself and others that incorporate poetic rather than practical interfaces to text or spoken language. In these text-based pieces, the characters function as legible signs, but also take on their own behaviors and responses to the user. These behaviors do not fit the normal obedient role of digital text that is cut, pasted, and clicked. These characters draw attention to themselves through their "misbehavior" as they become active objects, or overstep their bounds in other ingenious ways. Conversely, the pieces that provide unusual interfaces to spoken language question the line between bodies and language by physically putting the user in an unusual "position" with regards to the words. In each case the relationship between the symbolic and the physical is simultaneously thrown into relief and muddled.
In David Small and Tom White's (1997-98) Stream of Consciousness: An Interactive Poetic Garden installation, text escapes from the flat screen and spills out into the viewer's physical space (figure 18.1).
18.1. David Small and Tom White's (1997-98) Stream of Consciousness: An Interactive Poetic Garden. (Webb Chappell)
The installation consists of a garden with rock slabs, plants and water flowing from one level of a multi-tiered fountain to the next. Words and phrases projected down onto the water appear at the top pool of the fountain and swirl and flow with the water as it cascades through the garden, disappearing with the water as it drains out of the bottom pool. The "physically" modeled behavior of the words produces the convincing illusion that the text is floating on and carried along by the water. The text appears to have entered the viewer's world where forces like gravity and fluid dynamics affect its course. While the text still carries its symbolic weight as words, it also becomes the physical objects of leaves or detritus carried along by the water's flow.
The illusion that these characters are somehow objects as well as signs is furthered by one's ability to physically alter the course of the letters as they cascade down the fountain. As viewers press or push their hands on a small pressure-sensitive pad, The pressure-sensitive pad used in this installation has undergone changes through the course of exhibiting the piece. Originally the interface was a liquid-filled bag with a video camera inside that tracked pressure as changes in light. At the installation of the piece in the Ars Electronica Center in Linz, Austria, the liquid bag was replaced by a commercially available pad, which is more responsive and robust. glowing blue areas appear in corresponding areas of the fountain. The size and shape of the glowing area depends on the size and pressure of your touch. By positioning your "hand" in the path of the letters you can block their flow down the fountain, causing them to swirl in new patterns, and eventually change their meaning as they morph into new words.
The words that Small and White choose to "float" down their fountain hint at the conundrum of the simultaneously virtual, physical, and signifying text in this piece. In one sequence of text the characters are symbols from the periodic table of elements -- "Ni," "Ca," etc. These symbols morph into the word for their corresponding element when you stop them midstream. The boundary between words "standing for" elements that make up the physical world and "standing in" for those elements as a physical object in the fountain is blurred as you push and pull them around in the water, manipulating them with your fingers instead of your mind. The tension between the intangibility of the projected text and its behavior as a tangible object (which you can "touch" via the interface) parallels the tension between the text's position as a signifier for a real object and the real object it represents.
Pressing and sliding one's fingers over the pressure-sensitive pad in this installation provides a more sensual experience than interacting with a mouse, but the interaction is still confined to a relatively narrow channel. The interpenetration between the real and the symbolic in this piece is in fact quite lopsided. While the text seems to have escaped into the physical realm of the fountain almost completely, "you," via the pressure-sensitive pad, are present in the abstract world of these symbols only in the form of a blue glow that changes its position and size.
In Text Rain, by myself and Romy Achituv (1999), text again takes on the behaviors of objects that respond to forces in the real world, and also to the physical gestures of viewers (figure 18.2).
18.2 Text Rain by Camille Utterback and Romy Achituv (1999).
In Text Rain however, the interface of video camera and tracking software allows a viewer's entire body to engage with the text. In the Text Rain installation viewers see a mirrored black-and-white video of themselves on a large projection screen. Colored letters in the projection fall down on them from above, like rain or snow. The characters can be caught, lifted, and then let fall again. If a person accumulates enough letters along their outstretched arms, or any other dark object, they can sometimes "catch" an entire word, or even a phrase. The letters are not random, but lines of a poem by Evan Zimroth (1993) about bodies and language. Falling text is excerpted from "Talk, You" by Evan Zimroth, published in Dead, Dinner, or Naked poems by Evan Zimroth, TriQuarterly Books, Northwestern University Press, 1993. Used with permission.
Similarly to the text in the Poetic Garden, the text here continues to serve its symbolic function as an decipherable code, but also as an "object" viewers can engage with as if it were a real physical entity. In Stream of Consciousness, physical interventions cause the text to morph and mutate; in Text Rain the physical act of catching letters is necessary in order to read the text at all. The act of reading takes on a physical dimension.
Using a video camera as an input device allows the letters in Text Rain to respond to a wide variety of human gestures and motions. There is no "wrong" way to interact with this piece. Because most of one's body is visible in the virtual space of the screen as well as in the physical space in front of the screen, a pleasurable confusion results between the screen space and the real space. Because no complicated apparatus is involved to become "immersed" you can easily feel present in both the physical and virtual space simultaneously, or seamlessly shift back and forth between the two.
It is also significant that this interface allows participants to engage not only using their whole bodies, but also with other people's bodies in the installation space. People often cooperate to catch letters -- holding hands or stretching coats and scarves between them. The video interface allows people to be physically engaged with the text, but also to engage with each other while interacting with the text.
In another piece of mine, Drawing from Life (2001), the text's behavior is even more tightly coupled to the viewer -- it becomes them (figure 18.3).
18.3. Drawing from Life by Camille Utterback (2001).
Upon entering the installation space, participants encounter a live video projection of themselves, but their images are completely transformed into the letters "A," "T," "G," and "C" -- the letters representing the four proteins of DNA. The letters are color-coded based on the color associated with each protein from computer-analyzed gels scientists use when decoding the genome. The color saturation of any particular letter is based on the brightness of the color in the incoming video, so some amount of detail about each person is visible. It is remarkable how recognizable individuals are even in this abstracted form.
By abstracting live imagery of a viewer's body into the letters that compose DNA, the installation raises questions about our embodiment and the code that is both part of, and helps produce our "selves." As in the previously described installations, here again the boundary between flesh and abstraction is questioned by the content of the interactive text as well as by its behavior.
When viewers recognize that the projected imagery is a translation of themselves, they "test" the correspondence by moving parts of their body -- tilting their head, waving their arms, etc. Once the connection is made, viewers "play" with manipulating their transformed symbolic "self" using their physical body. The ease with which one controls one's video image is comparable to the experience of controlling one's image in a mirror. By distancing this connection through the abstracting of the live image into letters, viewers become more aware of the discrepancy between the abstraction and their bodies. The letters also continually flicker and change between the characters, as if they had a life of their own. Viewers recognize that this abstraction is simultaneously "them" and "not them." The image of DNA characters "stands in" for them, and on some level "stands for" them in the way that genetic code does, and in the way that the letters "stand for" the proteins themselves.
The text in Jeffrey Shaw's (1989) The Legible City installation does not respond to user's actions per se, but instead puts the user in the position of acting within the space of the text (figure 18.4).
18.4. The Legible City by Jeffrey Shaw (1989).
Stream of Consciousness puts the text into the viewer's physical space. Text Rain inserts the viewer's image into a flat abstract space along with the text. The Legible City inserts the user's point of view via computer-generated linear perspective into a dimensional space made entirely of text. In this installation the user sits on a real bicycle and pedals his or her way through a virtual landscape of text. The individual characters of the text in this installation become the architecture. Each letter is monumentally rendered in three dimensions, and takes the place of a building along the city streets of this space. The letters march off into the horizon, defining streets and avenues that correspond to real city maps of New York, Amsterdam, or Karlsruhe, Germany. In two of the cityscapes, the size of each letter in the text actually corresponds to the size of the building it represents. The text from which each city is "created" is also text about that city. The texts vary from contemporary quotes and writing to descriptions of historical events. The Drawing from Life installation conflates code with body. In a similar move, The Legible City equates descriptions of cities with the city itself.
In The Legible City, in order to read the texts stretched along the city streets (similarly to Text Rain) one must use one's body. A button on the bicycle interface allows the user to instantaneously switch views between the cities, but to move anywhere within the environment requires real physical exertion. Unlike so many virtual worlds, here distance matters. One's legs are the means of transport, but also an essential part of the equation if one wishes to read. The tension between this symbolic city -- both "rendered" by text and virtual -- and the physical exertion required to move through it, is the tension between the material and the abstract that has informed all of the pieces discussed so far.
In Shaw's The Legible City, one has the illusion that one's point of view is changing based on the changing imagery on the projection screen. As is often the case with computer interfaces, the viewer sits still while the display simulates motion. Two of my works require that one actually physically change one's point of view to interact with the piece. In the first of these pieces, Vicissitudes, a six-foot ladder and a chalk outline on the floor provide the interface for two audio tracks (figure 18.5).
18.5. Vicissitudes by Camille Utterback (1998). When a user is completely at the top of the ladder or within the chalk outline, only one soundtrack is audible, but the opposite visual -- ladder or outline -- is clearly in view. Between either extreme a mix of "up" and "down" is heard. Light levels of spotlights focused on the ladder and chalk outline correspond to the volume levels of the respective soundtracks, providing additional user feedback, and creating a dramatic ambiance.
One soundtrack consists of interviews of people describing times in their life when they felt "up" or "on top of the world," the other track of times when these same people felt "down" or "low." Climbing the rungs of the ladder raises the volume of the "up" soundtrack, while lying down in the chalk outline raises the volume of the "down" soundtrack. Many of our linguistic constructs rely on physical metaphor, though they have become transparent to us due to their common usage. Through its interface, this piece explores the embodiedness of language itself.
In a recent installation created with Adam Chapman -- See/Saw -- we use a see-saw as the interface to two screens (figure 18.6). One screen is positioned behind each seat of the see-saw across from each other in the installation space. As the users operate the see-saw, changing their position from up to down, their motion causes changes in words on the screens behind them. The words or phrases on the opposing screens will always exist in a state of tension or balance corresponding to the dynamic of the people on the see-saw. This installation plays with physical point of view, and the physical relationship of balance and tension between the two users as it relates to language.
18.6. See/Saw by Camille Utterback and Adam Chapman (2001).
Of the pieces described so far in this essay, Vicissitudes and See/Saw involve the largest-scale motions on the part of the user. While these interfaces engage the viewer's body on a large scale with a symbolic space of language, the language or symbolic space cannot exert a physical force on the viewer. In Orit Kruglanski's (2000) piece As Much as You Love Me, the interface allows words to manipulate the user physically as well as emotionally. This interactive poem uses a specially designed force-feedback mouse. As the user uses the mouse to collect icons on the screen she hears the spoken lines of a poem. Each line of the poem is what Kruglanski refers to as a nonapology and starts with the line, "don't forgive me for..." The more nonapologies the user collects and hears, the stronger the magnetic force on the mouse becomes, making it eventually almost impossible for the user to move the mouse. When the last line of the poem is collected ("forgive me this: I can't remember loving you") the force-feedback turns off in a dramatic reversal that mirrors the poem. In Kruglanski's piece the symbolic or emotional weight of words is brought to bear on one's physical freedom of motion. Via the interface of the force-feedback mouse, the symbolic content is viscerally enacted by the interface onto the user's body.
In all the pieces discussed in this essay, the symbolic world of text or language attains presence in the physical world and engagement with viewer's bodies via unusual forms of interface. Creative interfaces that connect our bodies to this digital media allow the line to be transgressed it in new ways. In the Interactive Poetic Garden and Text Rain, text becomes objects with seemingly physical properties that the user can engage with using their body. In Drawing from Life, characters take the shape of the users' bodies, mirroring their motions and gestures. In The Legible City, text forms the architecture one traverses on a physical bicycle. In Vicissitudes and See/Saw, the body on one side of the interface must "move." In As Much As You Love Me, the interface allows the symbolic to reach into the physical world and constrain the user's motions. The flip side of the text's transgression into the physical in these pieces is the manner in which the user's body enters the symbolic space of the texts -- as a blue glow, a photographic image, or a point of view. What is at stake in these artistic investigations, more than the creative possibilities for "interactive text," is the position and status of our bodies as they are increasingly represented on screens, or in the virtual space of our machines. As we create new interfaces between our bodies and our symbolic systems we are in a unusual position to rethink and re-embody this relationship.
Responses
References
Back, M., R. Gold, A. Balsamo, M. Chow, M. Gorbet, S. Harrison, D. MacDonald, and S. Minneman (2001). "Designing Innovative Reading Experiences for a Museum Exhibition." IEEE Computer 34, no. 1 (2001): 80-87.
Hayles, N. Katherine (1999). How We Became Posthuman. Chicago: The University of Chicago Press.
Kac, Eduardo (1997). Time Capsule. http://www.ekac.org/timec.html.
---. (1997). A-Positive. http://www.ekac.org/apositive.html.
Kruglanski, Orit (2000). As Much as You Love Me. http://www.ok-centrum.at/ausstellungen/cyberarts00/kruglanski.html.
Shaw, Jeffrey (1989). The Legible City. http://www.xspace.at/english/shaw.html.
Small, David and Tom White (1997-98). Stream of Consciousness: An Interactive Poetic Garden. http://acg.media.mit.edu/projects/stream/.
Stelarc (1997). Parasite. http://www.stelarc.va.com.au/parasite/index.htm.
Utterback, Camille and Romy Achituv (1999). Text Rain. http://www.camilleutterback.com/textrain.html.
---. and Adam Chapman (2001). See/Saw. http://www.camilleutterback.com/seesaw.html.
---. (2001). Drawing from Life. http://www.camilleutterback.com/drawingfromlife.html.
Xerox PARC (2000). The Tilty Tables. http://www.theredshift-xfr.com/tilty_tables.html.
Zimroth, Evan (1993). Dead, Dinner, or Naked. Evanston, IL: TriQuarterly Books.