In this essay, Robert P. Fletcher demonstrates how, while putting together digital and print media affordances, augmented print may evoke in readers a sense of the uncanny. Fletcher also explains how works such as Amaranth Borsuk’s Abra (2014), Aaron A. Reed and Jacob Garbe’s Ice-Bound (2016) or Stuart Campbell’s Modern Polaxis (2014) seem to demonstrate the existence of a never-ending return of the “familiar” in electronic literature.
The aim of this little volume is, as far as may be, to translate into verse what the lines and colours of certain chosen pictures sing in themselves; to express not so much what these pictures are to the poet, but rather what poetry they objectively incarnate. Such an attempt demands patient, continuous sight as pure as the gazer can refine it of theory, fancies, or his mere subjective enjoyment. —Michael Field (Katherine Bradley and Edith Cooper), preface to Sight and Song, 1892
After the author is gone and the page is gone, what is left but for the poet to split herself in two, to take shape as a split self with a forked tongue and speak in tongues. But further still, to then displace her self and let the reader give the text its next shape. —Amaranth Borsuk and Kate Durbin, manifesto for Abra, 2014
Introduction
In this essay I want to link the trope of "digital ekphrasis" (as theorized by Cecilia Lindhé and others) to the poetics of augmented-reality texts, which weave together "born digital" and print layers. As seems appropriate for the dual mediation of augmented print, I will first look backwards to the history of ekphrasis in print literature before examining its present and speculating on its future through the work not only of Lindhé, but also of theorists and practitioners of augmented reality (AR). While glancing at the work of a number of critics and artists, I will cite primarily four instances of "augmented" print—my own prototyped remediation of the ekphrastic poetry of the aesthetes Katherine Bradley and Edith Cooper (who published as "Michael Field"), the augmented poetry of Amaranth Borsuk and her collaborators, the narrative game Ice-Bound by Aaron A. Reed and Jacob Garbe, and the AR graphic novel Modern Polaxis, by the Australian comic-book artist Stuart Campbell, who publishes under the name Sutu. I've included these specific, disparate examples of print augmentation—poetry and fiction, professional and DIY, high culture and pop culture, remediation and original texts—to explore what makes it distinctive as a representative medium and to reveal some of the cultural, thematic, and formal common ground. Looking at these amalgamations of print and digital textuality through the lens of digital ekphrasis reveals that electronic literature will most likely always arouse ambivalence, just as the trope of ekphrasis in traditional media has, for better or worse, provoked an uncomfortable sense of the uncanny through its appeals to sensory and mental experience.
Ekphrastic Indifference
First, let me establish the aesthetic and rhetorical theories of ekphrasis that will frame my discussion of AR texts. Writing in 1997 of the centuries of ambivalence associated with what we might now call multimodal "picture poems," W.J.T. Mitchell outlined the "dangerous promiscuity" of ekphrasis, how its "mutual interarticulation"—with words helping to determine the significance of images and vice versa—threatens the stability assigned by audiences to each medium, at the same time that it provokes the wish that each medium's limitations can be overcome. He makes the case that ekphrasis has typically provoked fascination through "three phases, or moments of realization"—indifference, fear, and, lastly (at least for some), hope—and I will use this response process to outline my own consideration of ekphrasis in AR texts.
Citing Lessing's Laocoon and Nelson Goodman's aesthetics, Mitchell labels skepticism towards ekphrasis a "studied indifference," which "grows out of a commonsense perception that ekphrasis is impossible"; from this perspective, there are "inherent, essential properties of the various media" and they have "their proper or appropriate modes of perception" (152). Bradley and Cooper had their own encounter with ekphrastic indifference in the reactions of fellow poet William Butler Yeats and the young art historian (and their close friend at the time) Bernard Berenson, both of whom thought their book of poems about paintings, Sight and Song, was an interesting failure, doomed by the poets' subordination of their poetic vision to the images and by their confusion of "the material of poetry, which is feeling with colour and outline, the materials of painting," in Cooper's transcription of what Berenson told her (Works and Days 225). This complaint is a version of what we would now call medium specificity, and Maria Engberg and Jay David Bolter have beaten me to the punch by already noting the common thread of its appearance in the discourses of art history and augmented reality. In an essay that appeared in Convergence in 2014, they trace the need felt for such clear boundaries between perceptual experiences of the visual and the verbal back through Marshall McLuhan to the modernist art historian Clement Greenberg, who, by no coincidence, was an intellectual heir of Berenson. In arguing for what Engberg calls a "polyaesthetic," she and Bolter push back against the tendency today to "look exclusively for ways in which digital media reconfigure our senses physiologically," making the case that other "individual choices and cultural factors contribute to the reconfiguration" (5). To illustrate this phase of ekphrastic indifference, I searched for an image that might epitomize the feeling that print AR - like ekphrastic poetry - is a perverted use of a medium, or at best a curiosity that elicits bemusement, and I thought immediately of a 2014 story on National Public Radio in the US about a wearable book produced for an MIT class project. This edition of James Tiptree Jr.'s "The Girl Who Was Plugged In" comes with a vest that, "[b]y combining networked sensors and actuators, [ . . .] can change lighting, sound, temperature, chest tightness and even heart rate of the reader to match what the main character in the book is going through" (Hu). Of course, the students included a wonderfully ironic joke in their project by choosing for their prototype text a famous dystopic story (written under a pseudonym by Alice B. Sheldon) in which a woman lives her life through the body of a celebrity avatar. The comments following the story on the NPR website run the gamut, but a scoffing indifference is definitely well represented, and it echoes the kind of criticism often leveled at ekphrasis in literary history, for example: "a total waste of time/energy/resources---if fiction doesnt 'grip' the reader---then its [sic] just weak art --- go back and rewrite. [A]dding this 'enhancer' to it just would be obnoxious-- and pathetic" (Comment by w.p. on "Sensory Fiction: Books That Let You Feel What The Characters Do," February 6, 2014 ). If words are dependent on images and other gimmicks, then they aren't good words, true poetry, or visionary art that rises above material existence.
Ekphrastic Fear
This brings me to Mitchell's next phase, "ekphrastic fear," which he defines as the realization that "the difference between the verbal and visual representation might collapse," and which he spots "in a wide range of literary theorizing, from the Marxist hostility to modernist experiments with literary space, to deconstructionist efforts to overcome ‘formalism' and ‘closure,' to the anxieties of Protestant poetics with the temptations of ‘imagery,' to the romantic tradition's obsession with a poetics of voice, invisibility, and blindness" (154-56). This unease about the image-text boundary has a long history in human culture; it shows up now in a lot of the anxieties of our own age involving digital media—not so-called sensory fiction alone. I think the "virtual" in general, and AR in particular, elicits it because augmented reality threatens the collapse of a number of entrenched binary values: real/virtual and word/image, of course, but also internal/external, spiritual/material, mind/body, masculine/feminine, and perhaps even a new and rather paradoxical one: the stable, still image vs. the fragile, slippery, ghostly animation. In Dreaming by the Book, Elaine Scarry argues that the oscillation between solidity and transparency in ekphrastic composition, as in the description of a wall viewed through gauzy fabric, echoes a subliminal negotiation of reality in the perceptible world:
Locke says that in the ordinary operations of perception, the idea of solidity "hinders our further sinking downwards"; solidity establishes the floor beneath us that, even as we are unmindful of it, makes us cavalier about venturing out. The idea of the solid wall prevents not our further sinking downward but our further sinking inward. It provides a vertical floor for all subsequent imaginings that lets us perform the projective act without vertigo or alarm, and thereby lifts the inhibi-tions on mental vivacity that ordinarily protect us. (12)
I am reminded here of the task video-game designers face of coding rigid-body characteristics into the proper game objects to prevent the player's avatar from sinking through the virtual floor of a game’s fictional world. For Scarry, ekphrastic description of place can establish for the reader a "vertical floor that, by promising to stop our inward fall, permits us to enter capaciously into the projective space without fear and therefore with the lifting of inhibitions" on vivid rendering of the world’s reality in the imagination (14). However, I would link that potential fear of the "inward fall" and the accompanying "vertigo" to the "anxieties about merging with others" that Mitchell attributes to ekphrastic fear (163). Ekphrasis can threaten us with a sense of, not only the world around us, but our very bodies as transparent, permeable, and violatable. The especially vitriolic ridicule of MIT's sensory-fiction experiment is an instance of that techno-determinist anxiety about the collapse of categories (you've heard it before: the ethereal life of the imagination falling before the debased indulgence of the bodily senses facilitated by technology). Philosophers like Brian Rotman also see in such anxieties about the corruption of mental life a fear of the dissolution of a unitary, "serial" (or alphabetic) self into a multiplicity of "parallel" subjectivities (83):
Without question, an irrevocable change is happening to the individual self: the thing thought to be fixed and definitional of human identity is becoming unmoored as the technological upheaval transforming the landscape of Western culture makes itself felt deep within our heads, within our subjectivities, our personas, our psyches. Multiple networks of person-to-person connectivity, forms of disseminated and dispersed intelligence, many-leveled mediation - enmeshed in a proliferation of virtual modes of agency and deferred presence - are installing a new psyche at the cultural intersection of these techno-effects and our ancient enbrained bodies. (81)
Ekphrasis in AR operates precisely by enmeshing the "solid" phenomenal world with a proliferation of phantasmal or virtual objects. Citing Mitchell, Rotman explicitly likens this conflict of "serialism" and "parallelism" he has identified to "the ancient competition between the verbal and the pictorial as the superior means of truth telling" (83). My last characterization of the fate of reading in a digital age - the disappearance of the reader's (and writer's) imagination - offers a second traditional reason for ekphrastic fear - fear of constraint - but it puts a contemporary spin on the anxiety rather different from what has usually been provoked by the word/image binary. True, it has long been said in a logocentric literary criticism that students should read the novel before watching the movie or that the pictures inserted in the illustrated edition force one to visualize the character in a prescribed way and thereby limit one's imagination. But, creators and readers of electronic literature know that the digital computer brings an entirely different meaning of "constraint" into play. The idea that the machine and its rules may have a role to play in the imaginative, "literary" experience adds a new dimension to that ekphrastic fear. From anxieties about the female android in any number of science-fictions, to worry that machine reading may be replacing close reading in literary study, to horror that machine-generated text is replacing authorial voice, the inclusion of the machine as a partner in the production of meaning provokes strong reactions.
Ekphrastic Hope
On the other hand, Mitchell and other theorists of ekphrasis have written about the enabling power of constraints on creativity, sometimes in a language conducive to thinking about augmented reality. Scarry calls ekphrasis "perceptual mimesis under instruction" (6) - which situates the writer and reader of the ekphrastic text in a tutorial relationship - and she argues that the imagining of a scene under the direction of a great writer "closely approximates actual perception," especially when compared to perception while dreaming, which evidently lacks vividness. Scarry devotes much of her book to enumerating and describing the techniques in ekphrastic composition, such as what the cognitive scientist J.J. Gibson called "kinetic occlusion," in which "one surface passes in front of another surface—a picture dropping from a wall . . . or, more simply, [a] hand passing over [a] face" (12-13). This is another description of the ekphrasis effect that dovetails with the experience of augmented reality, such as Borsuk and Bouse's Between Page and Screen, in which the material reality of the ephemeral poetry is reinforced here by my partially-occluded, ghostly presence on the screen (Figure 1).
Figure 1: My ghostly presence in Borsuk and Bouse's Between Page and Screen
However, Scarry calls upon a technology much older than AR to illustrate the striking vividness of kinetic occlusion:
In his description of his childhood room at Combray at the opening of Swann's Way, Proust describes the way the bright images from a magic lantern would play across the walls, overlaying the wall's opaqueness with their own "impalpable iridescence" the figure of Golo, now moving in a jerky trot, now stopping quizzically, his form adaptable to any part of the wall. (12)
Thus, print artists of the past have idealized and at times actually exploited the relationship between the physiological and technological mechanisms of perception to achieve their effects.
It's not surprising to find, then, that the constraints of ekphrasis both old and new have at times inspired writers and readers with utopian visions. For example, the late Victorians Bradley and Cooper - channeling aesthetic philosophers like Vernon Lee and Walter Pater - claimed that the exercise of ekphrasis constitutes an "effort to see things from their own centre" and thus come as close as we can to escaping our subjectivity - even if that is ultimately an impossibility. In the remediation I have done of the poem from Sight and Song entitled "A Portrait," which is a response to Bartolommeo Veneto's painting Bust of a Courtesan, I have taken a poem-and-painting pair noteworthy for its same-sex eroticism and gender politics and created a performance that highlights what Cecilia Lindhé, in her version of "digital ekphrasis," calls "the interaction between visual, verbal, auditive and kinetic elements in digital literature and art" (par.13). When one triggers the aura by scanning Veneto's Bust of a Courtesan with the Aurasma browser - whether on a screen, in a book, or on the wall of the Städel Museum in Frankfurt - a video (created with Garage Band and iMovie) plays that blends audio of a skilled reading of the poem by the poet Anna Evans (recorded for the project) with details from the image and scans of the printed pages from Sight and Song (Figures 2 and 3).
Figure 2: "Bust of a Courtesan" and Augmentation (Sight and Song Title Page)
Figure 3: "Bust of a Courtesan" and Augmentation (Image Detail)
When the video ends, a digital facsimile of the book's title page appears. If one taps it, one is taken to the Project Muse page for an Anna Parejo Vadillo essay in on the multimodality of Sight and Song and its suitability for new-media presentation. I am in the process of developing augmentations and an accompanying website for more of the poems from Michael Field's Sight and Song, and I believe this new-media project is highlighting the challenges to patriarchal representation that Parejo Vadillo and others have seen in this late-Victorian pair of lesbian feminist poets. Jill Ehnenn associates Bradley and Cooper's ekphrasis with their queer aesthetic, which is cognizant of the persistence of perceptual and subjective norms (that "thick wall of experience" with which we're each "ringed round," in Pater's metaphor) even while it exposes their conventionality. According to these poets, the ekphrastic experience - which we could call, in our own vocabulary, an intermedial disruption of a dominant perceptual regime - is a step in "a process by which we eliminate our idiosyncrasies and obtain an impression clearer, less passive, more intimate."
Such enthusiasm constitutes Mitchell's third phase of awareness in the potential for the image-word relationship, "ekphrastic hope": "this is the phase when the impossibility of ekphrasis is overcome in imagination or metaphor, when we discover a ‘sense' in which language can do what so many writers have wanted it to do: ‘to make us see'" (152). I would argue that a contemporary version of this enthusiasm is the excitement generated over multimodal composition. It has led Lindhé to conceptualize a "digital ekphrasis," and in doing so she makes the case that the more comprehensive theory of ekphrasis in rhetoric - including sensory experience beyond the visual - allows us to understand and appreciate the intermedial functions of digital textuality in new ways: "digital literature and art align with this concept of ekphrasis, especially in the way that its rhetorical meaning is about effect, immediacy, aurality, and tactility. The multimodal patterns of performativity in the rhetorical situation stage a space-body-word-image-nexus with relevance for how we could interpret and discuss digital aesthetics." Critics of electronic literature have coined analogous terms to capture the ekphrastic effect in new media, whether it's Engberg's "polyaesthetic," which I cited earlier, Hayles' troping of "intermediation" (first used by Higgins in 1965), or Elika Ortega's "multimateriality."
Intermediation, in particular, has gained currency with new media critics and others, such as art historian Barbara Maria Stafford, for whom the "practice of intermedia communication" via analogy is inextricably linked to "a general theory of artful invention" (8). For Higgins in 1965, "much of the best work seem[ed] to fall between media" (49) because "continuity rather than categorization is the hallmark of our new mentality" (50). When he revisits the concept in 1981, he makes a distinction between "mixed media" that pair image and word without raising questions about the nature of each, and "intermedia," in which "the visual element (painting) is fused conceptually with the words" (52). If it can be argued (doubtful though it is to me) that ekphrasis in general may merely mix or juxtapose image and word, digital ekphrasis in new media, especially so-called "mixed" or augmented reality, takes on that challenge of fusing media conceptually in the experience of the spectator/reader. In 2005, Hayles extends the term to include just such a triangulation of different representational systems (especially the digital) and viewer:
I want to expand its denotations to include interactions between systems of representations, particularly language and code, as well as interactions between modes of representation, particularly analog and digital. Perhaps most importantly, "intermediation" also denotes mediating interfaces connecting humans with the intelligent machines that are our collaborators in making, storing, and trans-mitting informational processes and objects. (33)
What these terms share is an attempt to capture both 1) the effect of "dissociation" between visual and verbal (or other) media, which my colleague Amy K. Anderson has also written about through reference to the New Rhetoric of Chaim Perelman and Lucie Olbrechts-Tyteca, and 2) the paradoxical effort on the part of the viewer to find analogical connections between such representations, which Stafford has argued is crucial to "consciousness as the art of connecting." Indeed, Stafford echoes the aesthetes Bradley and Cooper and the contemporary new-media artists Borsuk and Durban when she reflects on the ethical implications of visual analogy:
How we couple representations in space is the key to understanding selfhood. The activity of linking has an emotional component, fitting our desires to an expanding universe of events in which both self and others are mutually transformed. . . . Yet the fragility of objective and subjective coherence also makes us conscious of how contingent, jumbled, and nonlapidary all material things, including ourselves, in fact are. (141-2, 145)
While these ethical and aesthetic effects of ekphrasis are provoked by various multimedia, I would argue that the function of AR, especially at the present time, is to exploit them through the subjective experience of the uncanny that it provokes.
AR and the Uncanny
In their analysis of the relationship between locative AR and the uncanny, Gentès and Jutant cite both Freud and Todorov when reminding us that "the uncanny mode structures a whole tradition of literature, films, and video games that play on the discrepancy between appearances and actual meaning or impact of things and that question perception and knowledge of the everyday world" (92). I myself have argued that video game avatars, like automatons in ages past, can evoke a Freudian sense of the Unheimlich when their players are immersed in the fantasy world (Fletcher). However, Gentès and Jutant also point out that "everyday tangible objects cannot morph so easily" (95). Despite my skepticism of the strong version of medium specificity, one reason that I see the ekphrasis of AR as especially prone to produce an uncanny semiotic effect right now is that its mixed reality - for example, book and animation - can "undermine our assurance in what this ‘everyday' means, both and at the same time familiar and unfamiliar: Heimlich/Unheimlich" (96).
Curious whether or not AR producers integrate these concepts of ekphrasis and the uncanny into their work, I asked three important artists—Aaron A. Reed, Sutu (Stuart Campbell), and Amaranth Borsuk—about their interests in mixing new media and "everyday tangible objects" in the form of books. Despite some differences, they all express variations on the idea of ekphrastic hope, which is brought about through an uncanny experience not unlike that of Proust's magic lantern. In his collaborations what if I'm the bad guy (from 2011) and Ice-Bound (exhibited at the 2014 ELO conference), writer-programmer Aaron Reed has worked in both GPS-based and tracker-based (that is, locative and print) AR, but he traces them to different aesthetic traditions (sculpture or guerrilla theater, on the one hand, and magical illusions or art books, on the other). Reed sees these two versions of AR as having specific effects, with GPS-based AR more clearly involving the player in an embodied performance, while tracker- or print-based AR focuses less on exploiting embodiment and more on getting the human-computer interface out of the reader's way. Nevertheless, Reed's interest in both forms of AR originates in its ability to trick the brain "into relating digital objects to the analog world, in a different way than anything confined to a screen." When I asked Aaron about how he got interested in AR, he described an early experience with an app that overlaid a ghost on a location and the frisson, even "the physical, visceral sense" it gave him, which extended to the next day when he passed the spot "where that blue ghost was hiding yesterday." Not surprisingly, Ice-Bound exploits the figure of the ghost in its narrative for what Reed hopes are "fleetingly cool" effects that are designedly "ephemeral." In his academic essay on what if I'm the bad guy with his collaborator Phoenix Toews, Aaron emphasizes the changed "perspective" they set out to offer the player of their GPS AR game. In "juxtaposing and overlaying" the imagery of Ice-Bound, Reed found that he himself was thinking differently about the content of digital and print sides of the story; he hopes again that "there will be similar perspectival changes for the players." According to Reed, at least for the time being, while it's a new medium, AR keeps readers or players from "tuning out." I would claim that the ekphrastic process plays a part in that attentiveness.
Critics like Scott McCloud have shown us that even traditional comics are ekphrastic by their very nature, exploiting the gutter, so to speak, between images and words, and drawing attention to the conventionality of the medium. So it's not surprising that comics are appearing in new media like AR. While his book is "not alive in a supernatural sense," the Australian comics artist Sutu does try to "provoke the feeling that invisible elements could occupy your space" in his Modern Polaxis, an AR comic book that employs both the platform's ekphrastic, palimpsestic effects and the tropes of science fiction to help us "learn to taste the tea on both sides" of an uncanny reality. A "paranoid time traveler" according to the promotional video, Polaxis has, in his narrative, either a bad reaction to an experience with Ayuhuasca or an encounter with body-snatching alien plant people, depending on which layer we're reading, print or animated augmentations (Figures 4 and 5).
Figure 4: Print Layer of Modern Polaxis
Figure 5: Augmented Layer of Modern Polaxis
Both the print and AR layers are chock full of the imagery of boundary transgression: the shadows of Plato's cave, the live puppet, the cyborg body violated by a bio-port, drug-induced altered states, and body snatchers, to name a few. In reply to my emailed questions, Sutu wrote of the "magical surprise" the AR gave him when they got it working right, and the feedback he has received from readers since then that describes a similar impression: "People are foremost drawn to the magical effect of it and inquire about the story later. Which sounds a bit gimmicky, but in the aftermath of a sale I've received plenty of emails from happy customers who have enjoyed the story too. So that's a relief." One of the most intriguing ways in which form and content are brought together in the comic partakes of the "kinetic occlusion" that Scarry has linked to imagistic illusion in language. According to the book's theory of time travel, solid objects are keys for the overlapping of space-time, while bodies are transparent, permeable. Indeed, the AR reveals Polaxis's seemingly solid hand to be transparent (Figure 6).
Figure 6: "Kinetic occlusion" reveals the solid to be permeable.
The bipolar story told in Modern Polaxis - is he a time traveler or a paranoid schizophrenic? - works to undermine not just the sense of a solid physical reality in the world, but a whole set of binary values: fiction/non-fiction, post-industrial/developing worlds, ancient/modern, authentic/manufactured, and indigenous/globalized.
Despite not having written an AR text that includes ghosts, alternate realities, or cyborgs, Amaranth Borsuk - co-creator of the celebrated AR book Between Page and Screen and the more recent transmedia Abra - told me that she appreciates the "mediumistic properties" and "hauntological possibilities of AR." She mentioned both the augmented edition of Jekyll and Hyde by Martin Kovacovsky and Marius Hügli and Pry by Samantha Gorman and Danny Cannizarro as works that exploit the spiritualistic effects of ekphrastic texts - the latter having given her "chills" when she read it. Regardless of her different focus in content, it is clear that Borsuk is very much aware of both ekphrasis and the uncanny in AR as vehicles for disrupting the subject-object relationship. Abra is the magical poetic voice or "living book" created through the transmedia collaboration of Borsuk, Kate Durbin, and Ian Hatcher, and it has given rise to costumed live performances, an artist's book, and an iPad app that affords the reader yet another role in the collaboration. In the manifesto they created for it, some of which I have cited as an epigraph, Borsuk and Durbin echo their predecessors of a hundred years earlier, Bradley and Cooper (or "Michael Field" in their own alternate persona), with their awareness of intermedial boundaries and transgression exploited through ekphrasis: "Two poets, rapt throats rising. In separate houses yet hive-minded, they pen invokationtranslations of works of visual art depicting excess where hair and ears stand in metonymically for the act of hearing." As an e-lit scholar as well as a poet, one who has been actively researching ekphrasis - including Lindhé's essay on its digital manifestations - Borsuk professes herself "less anxious than interested in the interconnections between media": "My interest in projects like Between Page and Screen and Abra," she told me, "is in using mediation in ways that complement, rather than supplement the text - as you say, the goal is interdependence, even intermediality (to borrow from Higgins)."
The ghostly metamorphosis in Borsuk's AR texts usually occurs in the language of the poetry itself, and she cites both concrete poetry and lettrisme as traditions that have influenced her with their utopic and skeptical perspectives, respectively, on the potential of representation. The brilliant self-reflexivity of Between Page and Screen - its personification of the ekphrastic process through the letters between P and S - has been discussed by a number of critics, but I'd still like to offer a couple of examples. P and S compete for control over the fraught, unstable reality of their relationship, as the story is played out in the floating space between page and screen, and this struggle over meaning is mirrored in the linguistic transformations programmed into the text. Etymology thus becomes another metaphor for a mediated, collaborative sense of reality, like the romantic coupling and the page/screen binary, because the transformation of words through time violates the borders between languages - threatening the collapse of meaning - but also revitalizes language(s) in that very crossing. One interjected animation, wherein words scroll across the augmented space, encapsulates this playful, Joycean/Derridean sense of difference at issue. I've frozen and simplified the animation below:
between . . . page . . . and . . . screen . . . a . . . book . . . spx . . . in . . . bks . . . dna . . . is . . . rose . . . eros . . . by . . . ano . . . ther . . . nom . . . so . . . po . . . ems . . . grow . . . to . . . face . . . lght . . . zero . . . s . . . bit . . . by . . . byte . . . know . . . all . . . abt . . . amat . . . say . . . bmy . . . luv . . . baby . . . pls . . . o . . . you . . . chic . . . zeus . . . one . . . two . . . any . . . amt . . . wll . . . do . . . amen . . . mmm . . . open . . . dor . . . nto . . . an . . . otr . . . room!
This book does indeed speak in a DNA that is essentially transformative - is rose eros by another name? - as its embodiment only takes place in the collaboration of page and screen and reader.
Unlike Reed, Borsuk does see her print augmentations as a vehicle that draws attention to the embodied subjectivity of the reader, as Lindhé theorizes with digital ekphrasis. When asked about this issue of the reader's embodiment in a 2012 interview with Jen Rajchel, who had felt "something harrowing" when watching herself on the screen performing the text, Borsuk responded positively, saying that she "really wanted the embodied reader to be returned to the equation." And, indeed, as one positions the book before the webcam when "reading" Between Page and Screen, the tactical experience of "holding" language that might dissipate or explode at any moment comes to the fore. However, if you have ever watched the video demonstration of Between Page and Screen on the book's website, you know that Amaranth navigates the book and augmentations flawlessly, without a glitch or a slip. When I perform the text, on the other hand, the augmentations are far less stable, more fragile, and I wondered if my "harrowing" reading procedure - struggling to display and stabilize the augmentations - was more representative of the typical experience and perhaps even significant, mirroring the flux of the epistolary relationship between P and S. Amaranth graciously conceded the possibility:
I think you're absolutely right that the struggle is part of the experience, and I very much like your notion that this reflects the struggle between P&S. It also, perhaps, alerts us to the voyeurism of peeping in on their private letters (we have to work to get a look, as it were, which makes us more aware of our own mediation between them). As one scholar mentioned to me, the book "disciplines" the reader. It is finicky and must be held "just so." In learning the embodied interaction with the book (including experiencing the disorientation of engaging with your own image), hopefully the reader becomes more aware of just how embodied the act of reading is (in spite of the fact that we have been taught to think of books as a space where we disappear into the imaginative world the author has created for us). In fact, all books discipline us in some way. There is nothing "natural" about the way we open and read a print book, though time and education make it feel that way.
This awareness of resistance between media, or dissociation between subject and object, motivates ekphrasis in both print and digital forms, and it constitutes what I would call its ethical impulse, something that should keep augmented reality from being merely a fad.
Conclusion
At the 2015 Electronic Literature Organization conference in Bergen, Norway, the uncanny effects that I have been attributing to current mixed-reality texts were on display in several exhibits. John Murray and Anastasia Salter's From Beyond Hybridity -Ouija Board Project employs an Arduino-enhanced Ouija board as a tactile interface for digitally-augmented text, in effect reversing the expectation of our high-tech world that a magic which can create chills must happen on a screen through a digital interface, and thus also playing with the boundary between "material and the immaterial." Joellyn Rock and Alison Aune's Fish Net Stockings deploys Microsoft's Kinect system and "a richly layered mix of digital video, text, and silhouettes" to engage the user's embodied subjectivity in challenging the traditional mermaid story presented in its virtual world. Finally, Caitlin Fisher and Tony Vieira's Mother|Home|Heaven, originally a geo-located installation, uses the objects of an abandoned house as triggers for iPad-based augmentations telling the ghostly stories of the home's past occupants. For all three of these examples, the magical or supernatural elements of the augmented reality intentionally exploit the aesthetics of the uncanny to produce a revision of narrative reality.
In a 2014 Convergence essay about, in part, David Miller's AR fiction Sherwood Rise, Alexis Weedon, Miller, and their co-authors argue for the value of such mixed-reality storytelling as a tool of "media literacy," focusing on the critical role such a text can play: "In Miller's work, AR is the ‘x-ray specs' revealing what lies beneath or under the surface of a news story" (118). However, the writers also cite Marsha Kinder's discussion of the commercial fad for "transmedia story telling" that has been around since 1990, a fad in which seamless transitions between media "‘construct consumerist subjects who can more readily assimilate and accommodate whatever objects they encounter, including traditional modes of image production like cinema and new technological developments like interactive multimedia'" (114). This established trend toward AR advertising - evident even in the licensing agreement for some AR companies, such as Layar, which favors commercial uses - shows us that it is more fruitful to think of augmented reality as a cultural and aesthetic practice, than as a technology, as Lev Manovich argues in his article on the poetics of augmented space. The platforms for AR may change from smartphones to wearables and beyond - and marketers will no doubt continue to exploit both it and virtual-reality technologies - but AR itself will also persist in an uncanny electronic literature not just "born digital," to use Strickland's phrase, but cached in the world and texts around us. As practiced by the e-lit artists I've discussed, AR has the potential to bring electronic literature to a far wider audience and thus answer the criticism of those like Florian Cramer who see it as a coterie practice. However, just as the picture-poems of the ekphrastic tradition in print have provoked an uncomfortable or even dismissive reaction from some, augmented-reality literature may continue to elicit an ambivalence from its audience, and that may not always be such a bad thing.1My thanks to Amaranth Borsuk, Stuart Campbell, and Aaron A. Reed for their contributions to this essay.
Works Cited
Anderson, Amy K. "Image/Text and Text/Image: Reimagining Multimodal Relationships through Dissociation." Dissertation. University of Kentucky, 2014. Abstract available at https://english.as.uky.edu/dissertation-abstracts.
Bolter, Jay David and Maria Engberg. "Cultural expression in augmented and mixed reality." Convergence: The International Journal of Research into New Media Technologies (2014) 20(1): 3–9.
Borsuk, Amaranth. Interview with the author. 21 July, 2015.
___. Interview with Jen Rajchel. "From artists' books to hacking digital screens." Tri-Co Digital Humanities. 2 July, 2012. http://tdh.brynmawr.edu/2012/07/02/298/. Accessed 20 Aug. 2015.
___. "The Upright Script: Words in Space and on the Page." Journal of Electronic Publishing 14.2 (Fall 2011). http://quod.lib.umich.edu/j/jep/3336451.0014.212?view=text;rgn=main
___ and Brad Bouse. Between Page and Screen. New York: Siglio, 2012. http://www.betweenpageandscreen.com.
___ and Kate Durbin. "Our Mistress of Lit Crescendos the Restless Incumbent In Human Opening Explodes Rococo Glow Into Undazzled Dark While Gem-Flushed Mineral Statuettes Shout Abra Freaking Cadabra." The Collagist 48 (July 2013). http://thecollagist.com/the-collagist/2013/7/9/our-mistress-of-lit-crescendos-the-restless-incumbent-in-hum.html. Accessed 20 Aug. 2015.
Cramer, Florian. "Post-Digital Writing," ebr: electronic book review, 12 Dec. 2012. http://www.electronicbookreview.com/thread/electropoetics/postal
Ehnenn, Jill. "Looking Strategically: Feminist and Queer Aesthetics in Michael Field's Sight and Song." Victorian Poetry 42.3 (Fall 2004): 213-59.
Field, Michael (Katherine Bradley and Edith Cooper). "Preface" to Sight and Song. London: John Lane, 1892.
___. Works and Days: From the Journal of Michael Field. Eds. T. and D.C. Sturge Moore. London: John Murray, 1933.
Fisher, Caitlyn and Tony Vieira. Mother|Home|Heaven. Hybridity and Synaesthesia. ELO 2015: The End(s) of Electronic Literature. Lydgalleriet. Bergen, Norway. 6 Aug. 2015. http://aco.booktype.pro/elo-2015-the-ends-of-electronic-literature/caitlin-fisher-and-tony-vieira-motherhomeheaven/.
Fletcher, Robert P. "Puppets, Automatons and Avatars: Automating the Reader-Player in Electronic Literature and Computer Game." Playing the Past: Nostalgia in Video Games. Eds. Laurie Taylor and Zach Whalen. Vanderbilt UP, 2008. 239-64.
Gentès, Annie and Camille Jutant. "The game mechanics of pervasive applications: visiting the uncanny." New Review of Hypermedia and Multimedia, 18.1-2 (2012): 91-108.
Hayles, N. Katherine. "Intermediation." NLH 38 (2007): 99-125.
Higgins, Dick. Intermedia, 1965, republished with an appendix by Hannah Higgins in Leonardo Electronic Almanac 34.1 (Feb. 2001): 49-54. https://muse.jhu.edu/journals/leonardo/v034/34.1higgins.pdf.
Hu, Elise. "Sensory Fiction: Books That Let You Feel What The Characters Do." All Tech Considered, National Public Radio. 6 Feb. 2014. http://www.npr.org/sections/alltechconsidered/2014/02/06/272044748/sensory-fiction-books-that-let-you-feel-what-the-characters-do. 27 Feb. 2016.
Lindhé, Cecilia. "‘A Visual Sense is Born in the Fingertips': Towards a Digital Ekphrasis" DHQ: Digital Humanities Quarterly: 7.1 (2013). http://www.digitalhumanities.org/dhq/vol/7/1/000161/000161.html
McCloud, Scott. Understanding Comics: The Invisible Art. New York: Harper Collins, 1993.
Manovich, Lev. "The poetics of augmented space." Visual Communication 5.2 (2006): 219-40.
Mitchell, W. J. T. "Ekphrasis and the Other." Picture Theory: Essays on Verbal and Visual Representation. Chicago: U of Chicago P, 1995. 151-182.
Murray, John and Anastasia Salter. From Beyond Hybridity—Ouija Board Project. Hybridity and Synaesthesia. ELO 2015: The End(s) of Electronic Literature. Lydgalleriet. Bergen, Norway. 6 Aug. 2015. http://aco.booktype.pro/elo-2015-the-ends-of-electronic-literature/john-murray-and-anastasia-salter-from-beyond-hybridity-ouija-board-project/
Ortega, Elika. "Hispanic Legacies in Electronic Literature: The Trace of Experimental Writing in Spain and Latin America," Readers of Fiction (Blog). 18 Nov. 2014. Accessed 20 Aug. 2015.
Parejo Vadillo, Anna. "A Note Upon the ‘Liquid Crystal Screen' and Victorian Poetry." Victorian Poetry 41.3 (2003): 531-36.
Pater, Walter. "Conclusion." Studies in the History of the Renaissance. London: Macmillan, 1873.
Reed, Aaron A. Interview with the author. 10 July 2015.
___ and Jacob Garbe. Ice-Bound: A Novel of Reconfiguration (Santa Cruz, CA: Down to the Wire, 2014). http://www.ice-bound.com/. Accessed 20 Aug. 2015.
___ and Phoenix Toews. "Re-visualizing Afghanistan in WHAT IF IM THE BAD GUY: Using Palimpsest to Create an AR Documentary." Leonardo Electronic Almanac 19.1 (2013): 158-69.
Rock, Joellyn and Alison Aune. Fish Net Stockings. Hybridity and Synaesthesia. ELO 2015: The End(s) of Electronic Literature. Lydgalleriet. Bergen, Norway. 6 Aug. 2015. http://aco.booktype.pro/elo-2015-the-ends-of-electronic-literature/joellyn-rock-and-alison-aune-fish-net-stockings/
Strickland, Stephanie. "Born Digital." Poetry Foundation. 13 Feb. 2009. http://www.poetryfoundation.org/article/182942. 27 Feb. 2016.
Stafford, Barbara Maria. Visual Analogy: Consciousness as the Art of Connecting. Cambridge, MA and London: MIT Press, 1999.
Sutu. Interview with the author. 12 July, 2015.
___. Modern Polaxis. NAWLZ, 2014. http://www.sutueatsflies.com/modern-polaxis/. Accessed 20 Aug. 2015.
Scarry, Elaine. Dreaming by the Book. Princeton, NJ: Princeton UP, 1999.
Weedon, Alexis, David Miller, et. al. "Crossing media boundaries: Adaptations and new media forms of the book." Convergence: The International Journal of Research into New Media Technologies 20.1 (2014): 108–124. Endnote