Simon Penny recalls that the origins of the human-computer interface, politicized by a military heritage, are now explored by artist-enigineers who chaperone fragmentation and dissent.
Katherine Hayles and Eugene Thacker's responses reinforce and extend my arguments and observations in useful ways, for which I am grateful. Hayles focuses on the relevance of my position on embodied interaction to hypertext in particular. While in my terms, the bodily engagement in hypertext interaction at the conventional desktop terminal is highly constrained (both physically and conceptually), Hayles rightly emphasizes that even in this domain, the embodied dimension of interaction is worthy of more serious consideration.
In hypertext reading, the substance of the interaction is almost entirely literary/symbolic, with embodied interaction playing a minimal facilitating role, getting the desired symbolic codes from the mind to the machine via the fingers and keyboard. Yet still, a physical discipline is involved, in this case a physical discipline of almost complete inaction save subtle and complex finger moves. She asks: "What implications are generated by the reading practices required by electronic environments, compared to print? With print, the reader's physical activity is limited to the relatively trivial action of turning pages." Reading a book demands a physical ritual akin to her example of kneeling in church: body static, usually sitting, head bowed, gaze steady and located on the page. Public reading (in libraries) demands silence. In all, a rich disciplinary regime. In hypertext, the words are no longer "durably inscribed," yet the bodily discipline is largely unchanged, except that mouse moves replace page-turns. Perhaps this is why (some of) the literary community has found it comfortable to embrace hypertext!
Her employment of the term "cognitive entailments" for "a form of bodily cognition" in which assumptions "are absorbed through physical actions rather than conscious reflection" is precisely the kind of understanding that is key to the development of an aesthetics of interactivity, particulary that side of the study concerned with user behavior. Hayles notes, "[T]hese physical enactments teach the interactor." Her use of the term "interactor" and other neologisms like "spectactor" or the instrumental stalwart "user" all point to the fact that this kind of interaction is a new cultural phenomenon which is awaiting adequate language to represent its qualities.
But terminology like "physical enactment" can tend to reinforce Cartesianism. This is a tendency I strenuously attempt to avoid. I would prefer to speak in terms of a new more holistic conception of "embodied mentality" where physical enactments are not thought of as being separate from the cognitive flow of the user. As she says so clearly, "[I]t is time to recover a sense of the body as a site for embodied cognition, cognition constructed not through words but through physical interactions with procedural works."
Procedural works are designed systems that exhibit behavior with which we might be engaged. The design occurs substantially at a meta-level, designing the qualities of behavior which human users can recognize and to which they will respond. This meta-literature is where the new aesthethics operates.
Hayles's discussion of a "complex relationality" echoes my formulation of interactional modes that transcend the instrumental and mechanistic, where ecological or social metaphors are more relevant. All manner of critical, aesthetic and design issues arise when the way a system behaves is driven not directly by the users' activity, but when that information is taken as input into a more complex system which might include complex internal decision-making involving the behavior of other users or other physical or data systems, such as weather patterns or network traffic.
Eugene Thacker focuses upon the core ethico-aesthetic questions around simulation, which I address in my paper. As both he and I emphasized, the fantastic rhetoric surrounding "VR" (indeed, the very term itself) has oversimplified the many ways in which the material- and data-worlds can be arranged to intersect at the point of perception. He rightly asserts, "The virtual environment is layered onto the real (we might be tempted to say 'material') environment. As military technology today makes clear, the VR, videogame interface is the 'real world' of navigation, combat and warfare." Nor need we look as far as the arcane world of soft munitions ("softwar"). This complex of layering is typified by new widgets such as in-car navigation aids combining GPS with databases onboard or remote, the various capabilites of enhanced cellphones and numerous other devices and systems. Thacker's observation that the mode of relation to data "naturalizes a kind of 'seek-and-destroy' strategy" reminds us of the militarized mentality inherent in much computer practice, a direct outcome of the context of the basic research, an historical reality so persuasively documented by Paul Edwards in his book Closed World. As a case example, Thacker pinpoints the military origins of the idea of a single, unitary and consistent "body" that "operates across two different ontological worlds." He rightly observes that this conception is both over-literal and unnecessary when he asks, "is there any reason to assume that 'the body' in a VR environment is the same as 'the body' in the real world?" This kind of critical questioning is immediately generative of critical and experimental artworks.
The places where dataworlds and the material world intersect, referred to in Europe as "mixed reality boundaries" and in the US more often as "augmented reality," is a littoral zone, which is and has been a key site for exploration by artist-engineer types for a quarter of a century at least, with key works by Myron Kreuger, Paul Sermon, Jeffrey Shaw and many others. (I hesitate to employ the simple term "artist," as the territory requires a range of training and understanding which must include deep technological skill and fluency not usually associated with "artists.") He concludes with remarks which again identify a territory for artistic exploration, and happily, there already exists a community of artist-engineers for whom building systems which generate "reflexive behavior in real time" and which "occasion reflexive thought" is central.
His formulation "rethink virtuality beyond representation" encapsulates tasks crucial to an aesthetics of interactivity. "Representation" has rich and varied meanings in the visual arts, in literary and cultural studies, and in computer science and related disciplines, as well as in politics and elsewhere. This cloud of meanings is enormously rich in disjunctions and discontinuities. Its elucidation is in itself a fascinating study, and one which is fundamental in the establishment of a common language for the new aesthetics.