Phantasmal Fictions

This review is a simultaneous publication from the current issue of American Book Review.

The novel begins in a railway station, a locomotive huffs, steam from a piston covers the opening of the chapter, a cloud of smoke hides part of the first paragraph.

- Italo Calvino

Smoke from the pyramidal sawdust pile
Curls up, blue ghosts of trees, tarrying low
Where only chips and stumps are left to show
The solid proof of former domicile.
Meanwhile, the men, with vestiges of pomp,
Race memories of king and caravan,
High-priests, an ostrich, and a juju-man,
Go singing through the footpaths of the swamp.

- Jean Toomer

Smoke drifts over these pages, smoke from outside of sawmills in Jean Toomer's classic Harlem Renaissance era manticore of prose and poetry describing the rural Georgia he explored during a fitful (if temporary) discovery of his multiply ethnicized self. Smoke drifts from the railway station, wafting over to you, the reader, from Italo Calvino's second-person, self-reflexive narration. In both texts, smoke is evoked as a phantasm - in two parallel senses of the term. In literary theory and philosophy, in Gilles Deleuze (The Logic of Sense [1990]) and in the classic work of Jean Laplanche and Jean-Bertrand Pontalis (1961), phantasms are imaginary constructions that structure human behavior and ideas. At the same time, visual studies scholar W. J. T. Mitchell reminds us that phantasmata are deeply akin to the cognitive psychological notion of mental images.

Indeed, the unstable nature of smoke is an apt metaphor for the mental image, clearly apparent, and yet without the material presence of a solid object or even the stability of a memory (since memories point back to events that have occurred in the real world). The metaphor goes further, however. For Toomer, the phantasm of smoke also stands in for a history of cultural theft, a sense of mournful loss encountered by people of African descent in the US. Smoke initiates a political image recalling ages of slavery and nobility in turn. For Calvino, the smoke phantasm blends two parallel stories, a narrative description of a man reading a book in a railway station, and a real life story involving the simultaneous mapping (onto the pages of the book the man is reading) of you, the actual reader, and your own narratives of personal experience. In the books that contain the quotations by Calvino and Toomer, we can observe a number of ways that phantasms can become transformative and transgressive: (1) rich, imagistic, detailed, and sustained storyworlds are produced in dialogue with the reader's interpretation; (2) sociocultural norms and values are critically interrogated; (3) common formal literary and linguistic conventions are skillfully deployed (even experimented with), while maintaining the evocative force of (1). The notion of the phantasmal as mental image and ideological construction is at the heart of the majority of highly regarded literary works, crucially distinguishing the imaginary in literary fiction from that in many other modes of art production. This dual notion is also at the heart of phantasmal media. Yet, because media today are mostly computational systems, the mental imagery they evoke needs to be studied in the context of diverse user/reader epistemologies evoked primarily through data-structural and algorithmic constructions.

In light of this goal, a theory of phantasmal media is intended to cover two gaps within practices that use computation to construct imaginative works of fiction. First, scholars can invoke a transdisciplinary perspective, incorporating cognitive science, computer science, and computational media arts. Second, we can illuminate an underdeveloped potential of computational media arts, the ability of computational systems to address the human condition including social ills, cultural imaginaries, shared values, notions of beauty, and the other hallmarks of many venerable forms of art. This expanded notion of phantasmal media can carry scholarship beyond the boundaries of print media to include computational systems such as interactive narratives, games, electronic literature, digital media artworks, and social media technologies that likewise engage users/readers and developers/authors.

We need to fill in these gaps in our conceptual framework because phantasms are themselves constructed as an outcome of imagination at a number of interrelated levels. These are helpfully (if not exhaustively) described by philosopher Colin McGinn (Mindsight [2004]) as: (1) percept; (2) memory image; (3) imaginative sensing; (4) productive image; (5) daydream/dream; (6) possibility and negation; (7) meaning (cognitive imagination); and (8) creativity/expressivity. Especially relevant is the nexus between imagistic and conceptual thinking, McGinn reminds us that

Sensory imagination employs sensory elements, much as perception does - though, as we have extensively seen, these elements must not be conflated. Cognitive imagination employs conceptual elements, much as thinking does: these elements are not intrinsically modality- specific, and combine to form propositional contents. What is in common is the general faculty that works on these elements - the imagination.

Bearing McGinn's point in mind, the cognitive science perspective on imagination in this essay is grounded in the subfield of the cognitive linguistics enterprise that addresses meaning construction: cognitive semantics. In particular, cognitive semantics theories of conceptual metaphor and blending are useful because they address processes of imaginative cognition. Metaphor theorists propose that the understanding of many basic abstract concepts relies upon metaphorical thinking and analogy, and that metaphorical thinking arises from a basis in embodied human experience of the world. (See George Lakoff and Mark Johnson's Philosophy in the Flesh [1999] and Francisco J. Varela, Evan Thompson, and Eleanor Rosch's The Embodied Mind [1991].) Such scholarship presents metaphor as a series of mappings from one conceptual space to another, and has shown that there are many basic, entrenched metaphors that people use to express everyday concepts. These concepts are often structured by image schemas, "skeletal patterns" that recur in our motor-sensory experiences. Conceptual blending theory builds upon Gilles Fauconnier's mental spaces theory and elaborates insights from metaphor theory to describe the means by which concepts are integrated, guided by "uniform structural and dynamic principles" both unconsciously in everyday thought and in more complex abstract thought such as in literary arts or rhetoric. Conceptual blending theory is not currently a predictive theory that can forecast exactly how humans combine ideas. Rather, it describes constraints on the process of combining concepts - what makes one way of blending concepts more optimal than others - and provides a systematic way to talk about integrating concepts. While conceptual blending theory has been criticized for post-hoc and overly broad explanations (and defended regarding the same), it provides appropriate terminology and structure, in dialogue with other cognitive science results, to better understand phantasmal media experiences according to user needs, interests, values, feelings, and more.

Cognitive linguists (Seana Coulson, Todd Oakley, Raymond W. Gibbs, and others) have proposed that human concepts form "idealized cognitive models" (ICMs) upon which our understandings of objects in the world and abstract concepts are built. Using this terminology, works like Calvino's in which the reader is implicated in the text, or computer games in which the player controls a character within a gameworld, can both be seen as metaphorically mapping ICMs (mental spaces) that humans have of themselves into fictional ICMs, or to use terminology from conceptual blending theory, as selectively projecting aspects from conceptualizations of both real experience and fictional experience into a blended experience. Since blending integrates concepts from quite different, even clashing, conceptual frames, scholars speak of a double-scope experience - and if this experience is narratively structured, it could be considered to be what Mark Turner has termed a double-scope story. Turner describes stories as "complex dynamic integrations of objects, events, and actors." Hence, the projected, or blended, story at one moment differs from the next, even as one person's projected story differs from another's. In "Material-Based Imagination: Embodied Cognition in Animated Images," Kenny Chow and I introduce a notion of material-based imagination that builds upon the theories just discussed along with embodied cognition perspectives of meaning-making:

Instead of concentrating on goal-specific computation, we explore material-based, open-ended imagination through cases of fluid and flexible representations in the form of animated visual images that could be called elastic anchors for imaginative elaboration.

Though that article focuses on interactive and generative animation, the argument might be extended so as to build upon both media theoretic scholarship and cognitive science definitions and results, in ways that are useful for better understanding and producing phantasmal media.

This type of approach is crucial to recall when addressing how phantasmal media might effectively address issues of ideology. Take the example of sociocultural identity found in many literary works ranging from Kamau Brathwaite's Ancestors (a long poem published in 2001 that presents digitally inspired typographic layout and cultural/personal/historical content) to Ralph Ellison's sociological treatise in novel form, Invisible Man (1947). Far from just a higher-level literary concern, sociocultural identity is a central feat of human cognition. The fallout from this perspective is striking in at least two ways: (1) if our identities are largely imaginative, what are the implications for social scientists, humanists, and technologists grappling with the everyday lived reality of human identity categories (stereotypes, ideals, salient examples, etc.); and (2) how can theories of conceptual metaphor and blending aid in elucidating the types of ideologies, social relationships, political configurations, and global conflicts that result in our everyday lived experience as humans? Fiction has a unique ability to articulate nuanced subjective experiences of phenomena such as those related to identity, and computational meaning-making systems are uniquely poised to offer dynamic experiences that change based upon user interaction if only authors can take heed of the insights and progress already made in domains such as the literary arts.

Recall that a theory of phantasmal media must also address a second gap: the striking difference between more mature art forms, with established conventions and strong accompanying communities engaged in meaning-making, interpretation, and criticism, and digital works that often remain comparatively remote and focused on self-reflexive exploration of the medium itself as opposed to content. Noticing this gap does not mean devaluing many computational works that operate as self-reflexive formal systems much like those of the literary group Oulipo. It also does not seek to undermine popular games or innovative game-like systems that have stirred rich emotional, dramatic, and even motor-sensory experiences for players such as the influential console game Shadow of the Colossus (2005) or the AI-based interactive drama Façade (2005). Such works are valuable not only for the stunning content produced by practitioners, but also for insights gained into the extremely difficult nature of using procedurality as a basis for dynamic, artistic expression. For this reason, the literary/procedural gap described here calls for further works to build upon these advances and to do even better in invoking literary modes of meaning-making in computational domains, engaging multimodal sensory experiences in cinema, the performing arts, and, indeed, everyday life. The coordination of literary and computational models and media further calls for the grounding of expressive new phantasmal media systems in diverse, cross-cultural artistic practices.

Cognitive scientists have described the hallmarks of narrative imagining - event stories, action stories, parable, metaphor, metonymy, force dynamics, and more. Understanding how cognitive processes of narrative imagining comprise the building blocks for expressive works of fiction can serve as a bridge to understanding the nature of effective phantasmal media works. Modeling aspects of these cognitive processes that are regular enough to be amenable to procedural description, and crucially leaving the rest up to the facilities of human artists, can undergird a range of types of computationally expressive works, including not only canonical forms, but also new experimental systems. The tragedies and triumphs of the human condition, from descriptions of intensely personal melancholy to grand statements on the ability to view other humans as resources and to then exploit those resources, are all imaginative cognitive feats as well. If we see that substantial aspects of our experiences are, in fact, cognitive fictions as opposed to objective realities, the implications are profound. The supposed real curls up and drifts away, spectrally, like a pale line of smoke. Data-structures do not attempt to capture objective truths, but rather are subjective forms to be manipulated according to rules like sentences in novels. Algorithms for processing data become expressive tools also, limited to Turing-machine style processing, but robust in transforming content along within those confines. The key for harnessing computational systems to produce meaningful, potentially transgressive and transformative fiction then is to better understand patterns underlying imaginative cognition along with the non-deterministic nature of those patterns, and never losing sight of the fact that all data is comprised merely of smoky, subjective phantasms, and all meaning, even computationally produced meaning, is ultimately human imagination.

Cite this Essay: