How do we think about things — like electronic literature — that combine the operational aspects of computing systems with the affective and representational aspects of the arts? We could view them through the frameworks of computer science, the literary arts, or critical interpretation. These can all be valuable. But they are all, inevitably, partial. Wardrip-Fruin proposes that digital humanities frameworks can provide a way of thinking about the dual elements of electronic literature simultaneously. Here he provides a case study: a strand of research that is both in computational approaches to social simulation and in the creation of works that build upon, and guide the development of, these simulations. He discusses the digital humanities concepts of operational logics and playable models that help him and his collaborators understand their work as they carry it out.
In digital humanities work, we bring humanities insights and methods to bear on digital projects. Those projects are often themselves what we might call “humanities products.” That is to say, these projects operate within humanities domains: analyzing data from humanities disciplines for humanities publications, making humanities arguments in new media forms, opening access to humanities collections, or presenting humanities knowledge to the general public. But these are not the only types of digital projects that can benefit from a humanities approach.
This article presents a strand of work we have been doing in developing new electronic literature projects at the Expressive Intelligence Studio (EIS), a lab I co-direct with Michael Mateas at the University of California, Santa Cruz. These projects are pursued both as aesthetic works and as driving applications for technology research. As technology, these works often employ and develop artificial intelligence techniques. But standard AI evaluation approaches — looking at issues such as error rates, real-world fidelity, score maximization, or speed — would be inappropriate.
We can (and do) use the approaches of the arts and design to evaluate these projects as artworks. But to guide their development as technology, that is both integrated into the current project and may live on and be used for new works and in other contexts, we need a different approach. One approach would be to look at the AI embedded in the technology as expressing a set of ideas, open to hermeneutic explication and critique. Another would be to critically examine the reader/audience experiences that can be created by the procedural representations produced by the works. Another would be to look at each new technology as enabling a media form, amenable to media studies analysis. Many others could be imagined.
In a sense, each such idea is a way of imagining what we might mean by “digital humanities” beyond the creation of humanities products — beyond the application of computing to the analysis and dissemination of humanities data. I believe imagining such possibilities is a key activity we should be undertaking. The insights of the humanities are important for understanding our world broadly, and we must make a similar argument for the insights of the digital humanities — they are essential for understanding and shaping the increasingly digital world in which we find ourselves. The digital humanities have a key role in critically examining not only the experiences of — but also the systems that shape — our increasingly computationally-infused communication, work, and arts. And we must insist that this humanistic lens is not only of value in after-the-fact contemplation, but also in the process of exploring and developing new possibilities. Otherwise, we run the risk of ceding the discussion about the future of knowledge to those who see humanistic inquiry as fundamentally backward-facing and inward-looking.
In this article I attempt to provide an example of such work in practice.1Our work in EIS is certainly not the only example of such work in practice. For example, John Cayley’s The Listeners is work that explores new artistic possibilities of emerging language technologies while being guided by (digital) humanistic insights about language, literature, aurality, and technology. Such creative projects can also serve to explore and expose the potentials and limits of, and demystify, new technologies of language, writing, and interaction — as examined humanistically by Lori Emerson in Reading Writing Interfaces and by Christian Andersen and Søren Pold in The Metainterface. This can be contrasted with the work of authors such as Mark Marino, myself, and others who have taken digital humanities (specifically, software studies) approaches to looking at human interaction with computational language technologies through the lens of Eliza/Doctor, apart from the creation or examination of explicitly artistic systems and works. All these approaches are valuable, but Cayley’s creation-oriented and humanistically-guided work is particularly in the vein discussed in this article. The text below is divided into three main sections. First, it discusses the challenge of computing research for media. Second, it describes a strand of projects we have developed in EIS. Third, it discusses two of the digital humanities ideas that have been key to our approach.
Computing for Media
Computing research for media has a tendency to confound traditional engineering approaches. For example, take computer graphics. A traditional engineering approach would define a goal, such as photorealism, with all movement toward that goal as progress. And this is certainly how some work in the field has proceeded. But when these techniques are applied to making full artworks, rather than demonstration images, the problems with this approach become clear. For example, motion pictures that are recognized for using graphics technology to make an early push toward photorealism — such as Final Fantasy and Beowulf — are also recognized as aesthetic failures. On the other hand, deliberately non-photorealistic films of the same era — such as Pixar’s Monsters, Inc and Finding Nemo — are now seen as classics.
The problem for the traditional engineering mindset is that “progress” can take you backward. In the case of computer graphics, the issue was the now-well-known “uncanny valley.” The pursuit of realism is also a problem more generally. For example, in the domain of fiction, we find that real people are surprisingly different from fictional characters — so a dialogue generator that produces lines just like those spoken by average people is not going to serve authors any better than a story generator that produces the events of an average day. “Realism” is a niche aesthetic, and one best explored with a strong artistic compass, rather than with a naive goal of making things just like real life.
We need to guide technology research through knowledge of media, as found in the arts and humanities. Companies like Pixar demonstrate how this can be done for mainstream media — Finding Nemo required as much or more research investment than Beowulf, just focused on different questions. For the future of electronic literature, much of this research seems likely to be done in universities. The founding of the Computational Media department at UC Santa Cruz was driven by the goal of defining a place where knowledge of computing and media could be brought together on an equal footing, much like knowledge of computing and biology came together to found bioinformatics.
Different faculty in the department approach this work in different ways. Katherine Isbister’s lab combines human-computer interaction with both game design and creating physical objects meant to be touched and worn, operating in the space between the traditional fine arts (sculpture) and what have been considered crafts (jewelry, fashion). Nathan Altice’s students create new works informed by media archaeology, from exploring the potential of rarely-engaged media platforms like the Magnavox Odyssey to translating between digital games and physical tabletop games. Leila Takayama’s group brings together techniques from animation and human-robot interaction to explore the performative dimensions of the increasingly capable physical objects around us. Angus Forbes’s group joins the modes of information visualization, computer graphics, and contemporary visual art and design, creating works that range from bioinformatics tools to interactive installation art. And so on.
In this context, Mateas and I — who might be the odd outliers elsewhere — are simply following a research path with another set of interdisciplinary commitments that enable the forms of computing research for media that interest us. Michael’s PhD is from the Computer Science program at Carnegie Mellon, focused on artificial intelligence for interactive drama. While there he also worked actively with the Studio for Creative Inquiry — a group heavily engaged in both contemporary art and critical theory — and before that he helped found a group focused on ethnographically-flavored human-computer interaction at Intel. My path to a PhD began with an MFA in Literary Arts from Brown University, which I came to after exhibiting digital language works primarily in visual arts contexts, but while working as a research scientist in Ken Perlin’s lab at New York University. After the MFA I continued into Brown’s self-designed doctoral option with a committee made up of computer scientists (Andy van Dam, David Durand), humanists (Wendy Chun, George Landow), and a fiction writer / digital art luminary (Robert Coover).
As suggested above, Mateas and I both have histories with electronic literature. Mateas’s collaborative works include the groundbreaking interactive drama Façade and the ideologically-biased interactive documentary generator Terminal Time. My collaborative works include the text-based Cave VR experience Screen and the long-form web customization project The Impermanence Agent. We bring our interdisciplinary backgrounds to our work with the Expressive Intelligence Studio not as a set of requirements (it is not necessary that every project in the group engage in technical, humanistic, and artistic research) but rather as a toolbox, from which we work with students to select appropriate methods and bodies of knowledge for each stage of each project.
Our hope in founding EIS (and helping found the Computational Media department) was in part to develop a creative community. We’ve been lucky to recruit students who have made this a reality. And in recent years it has been gratifying to see this community grow into an international diaspora — with UC Santa Cruz as just one node — as EIS graduates have become faculty at more than ten colleges and universities, while others have found ways to develop and contribute to daring projects as independent creators and in industry, and current and former students, post-docs, and colleagues have grown friendships and collaborations.
Social Simulation and Electronic Literature Research
Within EIS we pursue a wide variety of projects related to electronic literature, with technologies ranging from the dynamic construction of choice-based narratives to the generation of natural language, brought into the world in forms from video games to Twitter bots to live performances. Rather than try to offer a comprehensive list, here I will describe selected projects from a particular thread of our research — in social simulation. In selecting projects, I won’t focus only on complete, stand-alone works of e-lit. While I will include such works, I will also describe some more preliminary projects, at the level of demonstrations or prototypes. And I will also mention two tools, created by our group, which are available to create projects outside our group. In this I am following recommendations from the “Media Systems” workshop that Mateas and I convened. (I believe this workshop was the first-ever activity jointly funded by the U.S. National Science Foundation, National Endowment for the Arts, and National Endowment for the Humanities.) As described in the resulting report, it is important to recognize that this range of forms — not only the demos, technologies, and tools that computing research would value, nor only the completed works on which the arts and humanities tend to focus — each makes important contributions to research exploring new media possibilities.
“Simulation” is an odd term. For some it might suggest a quest for fidelity to a “real” thing that is being simulated, much like the quest for naive realism that has driven some research in computer graphics. Yet simulation for media covers much broader ground. Physics simulations, for example, could include those meant to hew close to our everyday world, as seen in flight simulators. But they also include experiences in which the simulation system is meant to produce quite different results, as seen in platformer games which allow “double jumping” — jumping while still in flight from an earlier jump — something far removed from what our everyday world allows.
What simulation systems allow for, in media, is a wide range of audience options and a consistent system response. For example, if we tried to create a Mario-style platformer game by hand, drawing the arc of each possible jump as an animation that can be triggered by the player at a particular point, we would run into two problems. First, it would be intractable to hand-specify all the possible jumps, from all the possible points, that a simulation system can enable. Second, it would be very difficult for even one of these to take into account all the information that we might want the player to understand about physics in the world — such as the impact of running at different speeds before the jump, or holding down the button for different periods of time. We would have to simplify, perhaps authoring one jump for players going “fast” and others going “slow.” As a result, it would be harder for players to learn what is possible in the world, and less likely for unusual things they might imagine to be made possible by the system (because they weren’t imagined at the time of authoring, or these unusual things were the first cut while looking to save authoring effort).
The wide availability of simulation models for domains like physics (along with the widespread design knowledge about how to use them effectively) is key to why physics plays a central role in so many works of computational media. One of the questions that has driven our work at EIS is, “What if there were ‘social physics’?” That is to say, what if we could provide simulation models that would enable the exploration of a wide space of interactions between human beings, rather than physical objects?2Our notion of social physics arose through connections between the arts, humanities, social sciences, and computing. The initial conception was influenced by strands of interdisciplinary social science, themselves influenced by arts and humanities practices such as dramaturgy (Erving Goffman’s The Presentation of Self in Everyday Life) and even frameworks of games and play (Eric Berne’s Games People Play). That said, because we explicitly wanted to create a playable model of social interaction that was learnable and actionable (as outlined by Mateas) our further development was inspired by our team’s humanistic engagement with media artifacts, to examine how they portrayed social relationships and dynamics. In particular, we focused on analysis of media about high school (such as Mean Girls) and other media focused on social relationships (such as Sex and the City). While one might expect that using humanistic analysis of artistic media would result in a model of social interaction that could be used for only artistic ends, the team developing the later Immerse project (which was informed by social science work on “good stranger” behavior) found our fundamental approach was also effective for encoding social states and possibilities in its domain.
This interests us in part because, when playing physics-based games, audiences tend to feel some responsibility for what happens — thinking, “I missed the jump” or “I caught the ball.” But in digital narrative experiences it is quite rare for audiences to feel a similar sense of responsibility. More often they think, “So, that’s what happens to those characters” or “Well, that’s how the story goes.” We wanted to know what would it would be like if audiences felt at least partially responsible for interactions between characters, while drawing a fictional situation with more linguistic depth than found in, say, The Sims.
Our first major exploration of this was in Prom Week and its accompanying AI system, Comme il Faut (or “CiF”), both primarily developed by EIS members Josh McCoy, Mike Treanor, Ben Samuel, and Aaron A. Reed. Prom Week is set in the time leading up to an end-of-year dance at a U.S. high school. Each character has their own personal characteristics and desires, but these are defined against an elaborate set of rules and beliefs that determine “normal” behavior in its fictional high school. The audience interacts with Prom Week by selecting pairs among the characters, examining what actions they most wish to take with one another, and choosing which actions to attempt. Each potential action has the goal of changing the social state in some way — one desired by the initiating character — which may succeed or fail. In either case, each attempt results in a short scene, which will play out differently depending on the specifics of the world history and character relationships.
Each scene also has a number of effects. These effects include changing the characters’ relationships with each other, giving one or both characters temporary statuses that may change how future exchanges unfold (e.g., personally “embarrassed” or “angry at” a particular character), recording events in the ongoing history of the world, and triggering events based on patterns in the world state and character relationships. As a result, players can explore a wide range of possible social worlds, make important events happen in the lives of the characters, and fulfill (or ignore, or subvert) some of the character-specific goals presented at the start of each level — the last of which will determine the shape of the ending to that character’s story presented at the close of the level.
Because Prom Week was a complete piece, released publicly in 2012, we were able to learn a number of things from the public reaction. One was that some players did experience the sense of responsibility we sought. For example, in Rock, Paper, Shotgun, Craig Pearson wrote, “I presumed I’d need to be nasty, but that route got me nowhere. Not that it wouldn’t have worked, and horribly it makes me want to see if I could destroy Buzz, but I won the game by accidentally being nice and friendly. So now I feel bad and impressed, and want to play it all over again” (n.p.).
On the other hand, we also learned that Prom Week’s extremely open-ended story structure doesn’t guarantee a sense of developing narrative. Alex Mitchell describes his experience with Prom Week this way: “What I was beginning to understand was the correspondence between character actions and changes in character relationships within the social simulation, changes that lead towards achievement of the goals set by the system. There is, however, no modeling or representation here of any story structure” (34). While some social physics systems may not desire to provide players the sense of a developing narrative, we also believed that our approach was one that had the potential to enable experiences that combined a set of story possibilities with audience agency.
Responding to the potential desire for more directed narrative, which was visible among some players of early Prom Week versions, EIS members Anne Sullivan and April Grow built a prototype system called Mismanor in parallel with the completion of Prom Week. Mismanor is a historical fantasy, set in a countryside home, where the focal/player character has been invited to an upper-crust dinner party. The end of the evening inevitably reveals that some of the characters are members of a cult, and the player character has not been invited to dinner for entirely innocent reasons. The Mismanor experience is further structured by quests, which different characters may offer the player character based on the current state of the social world, and which have as their endpoints the achievement of certain social states — which may be arrived at in many ways.
This experience required two new technology systems: CiF-RPG and Grail GM. CiF-RPG extends the CiF system (developed alongside Prom Week) to account for differential knowledge (as opposed to everyone knowing everything, in Prom Week’s high school) and the movement and manipulation of objects. Grail GM dynamically decides what quests to offer, and which characters to offer them through, based on the past progression and current state of the social world. This prototype convinced us that social physics could provide much more range of possibility than is typical in a more directed narrative, together with a sense of responsibility. Our belief was borne out two years later, when Emily Short published Blood & Laurels, a much more complete work (compared with Mismanor) created using the Versu social simulation tools on which she and Richard Evans collaborated.
The focus of our social simulation work has often been on “high fidelity” social interactions, unfolding at something like a conversational speed, with the interactions driven by the project’s audience. But another strand of our work has used lower-fidelity simulation to create bodies of events in a fictional world, before any audience is present. Our most developed project taking this approach is Bad News, developed by EIS members James Ryan, Ben Samuel, and Adam Summerville. This piece is presented as live, interactive, dramatic performance.3While I discuss Bad News in the context of the social simulation used during its generation phase, a fuller picture of the work would likely place it at the intersection of two computational media traditions. One is the generative tradition, which began with Christopher Strachey’s love letter generator — likely the first work of electronic literature, and first work of digital art of any kind. The other is the tradition of computationally-guided dramatic performance, as seen in the work of groups like Blast Theory. What connects them is the “story sifting” performed on the generative material to determine the material presented to the actor to guide their improvisation.
Each Bad News session begins with a social simulation of decades of history in a small U.S. town. Through this, characters who live in the town come to know each other, form friendships, families, and animosities, work with and for one another, be born, die, and so on. A unique town is created in this way for each performance, and each performance is done for an audience of one, after which that town is never experienced again. Performances take place on two sides of a curtain. On one side sits the audience and on the other sits Samuel — who in addition to being a computational media researcher is also a professional actor. Samuel draws the curtain aside and greets the audience member as the newly-hired assistant of his character, the local mortician. The audience member’s job is to locate the next-of-kin for an anonymous dead body, by talking with residents of the town, but first Samuel talks the audience member through crafting a “cover story” for their work, so that they do not unintentionally inform someone else of the death before the next of kin.
After this, Samuel draws the curtain closed again, and the audience member chooses to go where they wish in the town, and speak with who they wish. As they do, Samuel draws the curtain aside and improvisationally portrays each character. But before he begins, Ryan searches through the simulation record of the town, finding information about the character’s personality, the most dramatic events of that character’s life so far, connections they might have with characters the audience member has already met, and so on. While no performance can be planned ahead of time, the evocative social simulation that the team has authored inevitably provides intriguing and revealing character experiences for Samuel to work into his performance.
Audiences often engage Bad News lightly at first. But as they come to know the characters and their town more deeply, and as they are drawn into a feeling of relationship through Samuel’s performance, over the course of interaction, the tone shifts. By the time the next of kin has been located, audiences are often hesitant. They commonly draw out the conversation, not quite ready to share the bad news for which the piece is named. It is not uncommon for them to shed tears when they do so. In short, Bad News uses social simulation together with interactive performance to explore responsibility of another kind — not for what happened, but simply for being its witness and messenger.
As with Prom Week, because Bad News was a completed work, experienced by audiences, we were able to learn things from its reception. One thing we learned was simply how compelling people found the experience, such as when Bad News won the audience choice award at IndieCade 2016, or when Steven T. Wright wrote in Rolling Stone, “this marvel of procedural performance can only be played by a lucky few, and that's a crying shame” (n.p.). The other thing we learned is what it would take to make its kind of experience more widely accessible. While improvisational acting is not an easily-acquired skill, there is a body of knowledge about it, training is available, and even a beginner can get started with it. On the other hand, sifting through the record of the town, finding compelling story situations and character information, is something comparatively new. Creating Bad News and performing it with a variety of people gave us real-world experience with how to perform such “story sifting” in a wide range of interactive situations, each grounded in a different fictional world (for a preliminary framing of this concept, see: Ryan, Mateas, et al.; for a fuller treatment, see chapter 5 of: Ryan).
The story sifting approach is currently being taken further in a project titled Why Are We Like This? (WAWLT) led by EIS members Max Kreminski and Melanie Dickinson. This project grew out of a series of prototypes collaboratively created by EIS students (plus a few members of other labs) around the idea of a mystery story construction kit. The goal was to explore possibilities for two players working together with a computational system to collaboratively construct a mystery story that would be different every time. This would be a far cry from normal mystery game experiences, which usually frame mysteries as pre-defined puzzles to be solved by a single player. It also grew out of collaborative inquiry into mystery genre elements, resulting in a decision to focus on character and emotion over plot (as argued for by Raymond Chandler) and to try to produce a “cozy” mystery in a social sense (e.g., no characters are portrayed as monstrous). These prototypes employed a variety of technical approaches, from paper index cards to computational constraint solving. But the prototype that formed the seed of the now-emerging full project focused on story sifting.
In the case of WAWLT, what’s being sifted isn’t the large life patterns in the stories of the entire history of a town (as in Bad News). Instead the sifting is through more detailed interactions between a smaller cast of characters. And rather than sifting serving one primary purpose (as with the prompting for improvising a character in Bad News) in WAWLT there are two differentiated uses for story sifting. One is to provide ways for the audience (the readers/writers/players) to search through the record of the social simulation to understand its current state and past history. Another use of story sifting is to enable characters to subjectively reflect on the past and use that information to shape understandings of the world that can lead to new motivations.
These uses of story sifting both support the larger aesthetic goal of the system — to create an experience of collaborative sensemaking, and co-authorship, with a digital literary project and another human being. WAWLT isn’t simply a tool for co-authoring stories. If it was, many people would find it intimidating, and it would be difficult to feel the project was about anything. Instead, WAWLT provides characters with relationships, histories, and motivations. It presents a scenario and defines a set of possible actions that fit in with the “search for truth” theme of many mysteries. But instead of portraying one story that could arise from this material, it helps the audience to find sites of narrative potential within it, and offers two types of writing mechanics for interacting with it. One of these mechanics is allowing the audience to express system-understandable “author goals” for scenes, which shape the next character interactions recommended by the social simulation. The other is the editing of the ongoing language of the story — which is never left as an intimidating “blank page,” but instead seeded by outputs from the selected simulation actions.
Further, in addition to creating WAWLT, with its intriguing hybrid of story elements and tool-like elements, Kreminski and Dickinson are also the stewards of two EIS tools in this area. One is Ensemble, a successor to the CiF social simulation tool created in the context of Prom Week. The other is their recently-created Felt, a query language-based story sifting tool that can also be used as a simulation engine. It is our hope that tools like these will help others explore some of the possibilities of media creation and technology research in this area.
As we create work such as these projects, there are two digital humanities approaches that both help guide our work while it is in process and help evaluate its results when we are done. One of these I have called “expressive processing,” while the other Mateas has named “playable models.”
With “expressive processing” I aim to draw attention to two aspects of computational media. First, the internal processes of computational media are designed artifacts, like buildings, transportation systems, or music players. As with other designed mechanisms, processes can be seen in terms of their efficiency, their aesthetics, their points of failure, or their (lack of) suitability for particular purposes. Their design can be typical or unusual for their era and context. The parts and their arrangement may express kinship with, and points of divergence from, design movements and schools of thought. They can be progressively redesigned, repurposed, or used as foundations for new systems — by their original designers or others — all while retaining traces and characteristics from prior uses.
Second, unlike many other designed mechanisms, the processes of computational media operate on, and in terms of, elements and structures meaningful to humans. For example, a natural language processing system (for understanding or generating human language) expresses a miniature philosophy of language in its universe of interpretation or expression. When such a system is incorporated into a work of computational media such as an interactive fiction, its structures and operations are invoked whenever the work is experienced. This invocation selects, as it were, a particular constellation from among the system’s universe of possibilities. In a natural language generation system, this invocation might be a particular sentence shown to an audience in the system output. From looking at the output sentence alone, it is not possible to see where the individual elements (e.g., words, phrases, sentence templates, or statistical language structures) once resided in the larger system. It is not possible to see how the movements of the model universe resulted in this constellation becoming possible — and becoming more apparent than other possible ones.
How do these aspects of expressive processing serve to help evaluate works of electronic literature? I believe that, while approaches such as playtesting can give some insight into whether an experience like Prom Week is succeeding, audience members only see the surface of a game. Playtesting might tell us that the underlying ideas we intend to express through system design are not coming through, but it will not help us press, critique, or even understand those underlying ideas. For instance, I think Prom Week is a better game for the critical discussions we had about issues such as how “popularity” should function in a game, and how popularity should be expressed to players. Perhaps counterintuitively, some of our most important decisions were about what we chose not to encode in our processes, and how this absence was communicated to players. While Prom Week’s simulated school encodes many stereotypes and biases, it does not include, for example, racism and heteronormativity.4When playing a game, a primary task for players is to figure out how it works. In a sense, players must internalize its models — which they often do unconsciously and imperfectly. My memory of the Prom Week development process is that both people on the development team and people beyond it (e.g., Jane Pinckard) believed that we didn’t want one of our first experiments with social physics to train people to internalize models of prejudice and oppression, but rather help them imagine a different world. While I believe effective work could be made that sought to expose and subvert societal oppressions through social physics, such concerns were also borne out by the difficulty we had subverting heteronormative expectations with Oswald’s level (as discussed in the main text). The second of these proved particularly difficult for some players to understand, even though the first non-tutorial level is designed to encourage players to discover it. (This level focuses on Oswald, a character who wants a date for the prom. His level is populated by a number of female characters with whom he has little connection, as well as with Nicholas, a “frenemy” with whom there is a lot of potential. But, despite the nudging of the level introduction, this potential proved difficult for some players to discover, or perhaps to accept.) Evaluating what we are expressing through our processes, through critical interpretation and discussion, is something I believe is indispensable in this kind of work.
The expressive processing approach is one that I have disseminated in traditionally humanistic modes, such as publishing a monograph of the same name. The playable models approach, on the other hand, so far has mostly been communicated through mentoring and oral presentation. Those things that have been written about it are generally phrased more in a computing vein. For example, Mateas contributed the following text to a report from a 2010 workshop, focused on the fact that playable models (in order to be playable) must be “learnable” and “actionable”:
To be learnable, a player must be able to make inferences about a game’s state and build up a mental image or model of the underlying system as they interact with the rules of the game. This may not mean that they are able to completely reverse-engineer the system. Rather, a learnable computational model is one supporting the incremental development of simplified and partial mental models that successfully provide guidance for future exploration and interaction within the game rule system. This exploration is afforded through mechanisms of engagement, that is, a means for a player to affect the state of the game in a manner consistent with his or her desires. To meet this requirement, the [model] must also be actionable. Defining games as systems that employ such playable models distinguishes them from traditional systems and computational models in other disciplines such as physics or engineering, where [the] cognitive properties of learnable and actionable are not factors. (Boellstorff et al. 33)
This does not sound like a particularly humanistic concept, when described in this language. But I would argue that it is, because the focus on playability, in this context, is on the way that audiences come to understand such models as procedural representations. While we have many humanistic tools for discussing our experiences with traditional, fixed representations (as found in print literature, cinema, and so on) we are greatly in need of tools for grappling with procedural representations.5A number of scholars have been at work developing ways of thinking about procedural representations, especially in relationship to games. For example, Jesper Juul’s book Half-Real explains how the “real,” rule-based aspects of many games can only be interpreted through their fictional, representational aspects. Ian Bogost’s Persuasive Games presents “procedural rhetoric” as a way to interpret, and make, arguments that employ a game’s rules together with traditional representation modes. Mary Flanagan’s Critical Play describes how players can engage the meaning of games’ systemic representations critically, and creators can design to foster that critical engagement. D. Fox Harrell’s Phantasmal Media argues that we can organize the internal structure and operations of systems around the subjective and cultural meanings we aim to evoke and expose. Doris C. Rusch’s Making Deep Games seeks to guide readers in creating games that have appropriate metaphorical mappings to human experience, both in their operations and in the interpretations and feelings they evoke through play.
Viewing such models and procedural representations also gets us away from the pitfalls of viewing them as naive simulations — just as it is important to get away from the naive realism of some computer graphics (and artificial intelligence). The important issues for us to discuss are not in what ways, and how closely, a playable model mirrors the results of the same actions in our everyday world. Rather, the key questions are what kind of world the playable model is leading its audience to understand, how that understanding is developed, and what kinds of actions become possible to imagine and carry out once that happens.
For example, returning again to Prom Week, our goal was to not just express the way the world works, but also show the way the world does not have to be, and help the player learn how to make a different world. A player who understands the game can make radical shifts in the social landscape — one beta test player joyfully reported inverting the high school’s entire popularity structure. It is that experience of changing the world, and of coming to understand some of the many ways the world could change, that is (hopefully) the core experience of Prom Week. Critically examining our work as a procedural representation helped clarify this goal for us, and guide us in our development.
As I write this, the publication date is drawing near for a book I’ve written that explains and explores the concept of playable models: How Pac-Man Eats. It dives further into the case of Prom Week, in particular how we worked to create a model of character volition: attempting to make our model of character desires learnable and actionable. We eventually learned that we were leaning too heavily on media conventions from games (which tend to focus on things like numerical values, often presented as bars, meters, or numbers) and instead needed to draw more on conventions from other media forms. In particular, borrowing the concept of a “thought balloon” from comics proved to be powerful, offering a legible way for characters to present what actions they are thinking of taking.
More generally, this process was important for clarifying what is necessary to construct a new procedural representation. Specifically, I argue that there are three key elements: how it operates, how its operations are understood, and how it is situated in the design of the larger experience. The first of these, how a procedural representation operates, might seem like a purely technical question. But if we think of procedural representations as simulations, it becomes clear that it is not — just as the map is never the territory, the simulation is never the thing being simulated. Further, the goal of the procedural representation is to create an authored experience. This is why, to return to an earlier example, the physics for platformer games is so far removed from physics in our everyday world (there is no double jumping when obstacles are encountered in our world). Further, for an experience like Prom Week the operations of the interpersonal models are by necessity informed by the ideas we wish to express, rather than some attempt to reproduce our everyday world, given there is no consensus about how interpersonal relationships operate in our world — only theories that assist in thinking from different perspectives.
The second issue, how a procedural representation is understood by audiences, is similarly more complex than it might at first appear. To return to the example of jumping in games, its physics are not communicated by simply moving characters up and down. Rather, many tools from other media forms are employed in its communication, especially tools from animation — exaggerating movements and environmental responses, such as a character crouching down on impact, with dust rising around her feet, and perhaps, when landing after a long fall, shaking the “camera” looking at the scene. Similarly, in Prom Week we learned to focus on what other media forms communicate about character intention — not some numerical value of affinity, which might seem like a pure exposure of how the system is operating, but rather indications of the actions characters are considering. We also learned to focus on the means other media use for communicating interior thoughts — for an experience like ours, without communication modes like detailed facial expressions, the thought balloons of comics worked well.
The final question, of how a procedural representation fits into a larger experience, is perhaps the most challenging. For well-established types of playable models there is a great deal of craft advice available from the game design community. For example, the book Game Mechanics, by Ernest Adams and Joris Dormans, is a good introduction to the different ways that economic models are used. A good introduction to issues of “game feel” (for games using physics-based spatial models) can be found in Steve Swink’s book by that name. But when procedural representations are employed in areas such as electronic literature, there is much less accumulated wisdom to fall back on. In one sense this is exciting — as practitioners and critics we will create and see many different approaches. We will have the opportunity to reach audiences with representational approaches that don’t need to be defamiliarized before they can be seen with fresh eyes. On the other hand, it will be an experience akin to flying without instruments or a map, as we found when attempting to create a meaningful audience experience and set of available audience actions that integrated with Prom Week’s procedural representations. A humanistic frame — simply thinking of what we were trying to create as a procedural representation, and interpreting what was produced and how it was experienced as such — was one of the most valuable guides we had in this work.
Of course, the concept of playable models can also be used for more traditional humanistic ends. In How Pac-Man Eats I often use it to interpret the work of others, including with popular games such as entries in the Grand Theft Auto and The Sims series. I also demonstrate how thinking in terms of models can be useful for historical interpretation, showing how it was more likely a difference in spatial models, rather than an issue of interface complication, that led to the different historical receptions of Spacewar!, Pong, and Computer Space. But as we explore new futures for electronic literature, I believe one of the most important roles for digital humanities concepts and work will be to provide types of guidance that neither traditional computing approaches nor traditional arts approaches can offer.
I believe this can serve as an important complement to, rather than exist in competition with, other modes of digital humanities.6To put it another way, if this article is read as a salvo in a “hack versus yack” debate, I’ll have failed. In the introductory video games class that I teach at UC Santa Cruz, I engage students in game making and analysis as complementary activities — the underlying structures of games, which particular game themes and surface presentations can obfuscate, become much more apparent when viewed from the perspective of one who has used such structures to create their own games. (This obfuscation is one of the many instances in which, as Lori Emerson writes, “the glossy surface of the interface further alienates the user from having access to the underlying workings” (xi).) Similarly, one of the urgent tasks of the digital humanities is to examine, expose, and critique the digital systems that increasingly infuse and guide our world — from racist policing and sentencing algorithms to recommendation systems that promote conspiracy theories and hatred, as discussed in books such as Cathy O'Neil's Weapons of Math Destruction and Safiya Umoja Noble's Algorithms of Oppression. These systems are not only obfuscated by design choices, but often actively kept secret by governments and corporations. To examine such systems effectively requires what Mateas has called “procedural literacy,” a topic explored in much greater depth in Annette Vee’s Coding Literacy.
This is part of why I am so happy that the digital humanities includes such a rich vein of work in developing digital projects. Not only are the projects themselves valuable, but they are a key means of developing the procedural literacy that the humanities needs for urgent work in our contemporary world. So while what I propose here may seem radical from the perspective of computing (in which the humanities are often seen as having little role) from the perspective of the digital humanities it is perhaps quite modest — merely extending our thinking about digital humanities making to include technical and aesthetic research projects.
Acknowledgements: Many thanks to Alex Saum Pascual and Scott Rettberg for organizing the “Electronic Literature as a Framework for the Digital Humanities” workshop at UC Berkeley, to Jeremy Douglass for hosting the follow up at UC Santa Barbara and suggesting this article’s title, and to the participants in both workshops for their thoughtful (and thought-provoking) responses to earlier iterations.
Adams, Ernest, and Joris Dormans. Game Mechanics: Advanced Game Design. 1st ed., New Riders Publishing, 2012.
Andersen, Christian Ulrik, and Soren Bro Pold. The Metainterface: The Art of Platforms, Cities, and Clouds. MIT Press, 2018.
Atari, Inc. Pong. Arcade. Atari, Inc., 1972.
Berne, Eric. Games People Play: The Psychology of Human Relationships. Grove Press, 1964.
Boellstorff, Tom, et al. The Future of Research in Computer Games and Virtual World Environments: Workshop Report. Technical Report UCI-ISR-12-8, Institute for Software Research, University of California, Irvine, July 2012, http://www.isr.uci.edu/tech_reports/UCI-ISR-12-8.pdf.
Bogost, Ian. Persuasive Games: The Expressive Power of Videogames. The MIT Press, 2007.
Cayley, John. The Listeners. Amazon Alexa Skills Kit. 2015–present.
Docter, Pete, et al. Monsters, Inc. Pixar Animation Studios, Walt Disney Pictures, 2001.
Domike, Steffi, et al. Terminal Time. Apple Mac OS 8. 1999–2003.
Emerson, Lori. Reading Writing Interfaces: From the Digital to the Bookbound. University of Minnesota Press, 2014.
Flanagan, Mary. Critical Play: Radical Game Design. The MIT Press, 2009.
Goffman, Erving. The Presentation of Self in Everyday Life. 1 edition, Anchor, 1959.
Harrell, D. Fox. Phantasmal Media: An Approach to Imagination, Computation, and Expression. MIT Press, 2013.
Juul, Jesper. Half-Real: Video Games between Real Rules and Fictional Worlds. The MIT Press, 2005.
Kreminski, Max, Devi Acharya, et al. “Cozy Mystery Construction Kit: Prototyping toward an AI-Assisted Collaborative Storytelling Mystery Game.” Proceedings of the 14th International Conference on the Foundations of Digital Games, Association for Computing Machinery, 2019, pp. 1–9. ACM Digital Library, doi:10.1145/3337722.3341853.
Kreminski, Max, Melanie Dickinson, and Noah Wardrip-Fruin. “Felt: A Simple Story Sifter.” ICIDS, 2019. Semantic Scholar, doi:10.1007/978-3-030-33894-7_27.
Kreminski, Max, Melanie Dickinson, Michael Mateas, et al. Why Are We Like This?: Exploring Writing Mechanics for an AI-Augmented Storytelling Game. 2020, p. 11.
---. “Why Are We Like This?: The AI Architecture of a Co-Creative Storytelling Game.” International Conference on the Foundations of Digital Games, Association for Computing Machinery, 2020, pp. 1–4. ACM Digital Library, doi:10.1145/3402942.3402953.
Little Story People. Blood & Laurels. Linden Research, Inc., 2014.
Marino, Mark C. Critical Code Studies: Initial Methods. MIT Press, 2020.
Mateas, Michael. “Procedural Literacy: Educating the New Media Practitioner.” On The Horizon. Special Issue. Future of Games, Simulations and Interactive Media in Learning Contexts, vol. 13, no. 1, 2005.
Mateas, Michael, and Andrew Stern. Façade. Microsoft Windows. Procedural Arts, 2005.
Maxis Software, Inc. The Sims. Microsoft Windows. Electronic Arts, Inc., 2000.
McCoy, Josh, et al. “Prom Week: Designing Past the Game/Story Dilemma.” Proceedings of the 8th International Conference on the Foundations of Digital Games, Society for the Advancement of the Science of Digital Games, 2013, p. 8. http://www.fdg2013.org/program/papers/paper13_mccoy_etal.pdf.
McCoy, Joshua, Mike Treanor, Ben Samuel, Aaron A. Reed, Michael Mateas, Noah Wardrip-Fruin, et al. Prom Week. UC Santa Cruz Expressive Intelligence Studio, 2012, https://promweek.soe.ucsc.edu/.
---. “Social Story Worlds With Comme Il Faut.” IEEE Transactions on Computational Intelligence and AI in Games, vol. 6, no. 2, June 2014, pp. 97–112. IEEE Xplore, doi:10.1109/TCIAIG.2014.2304692.
Mitchell, Alex. “Reflective Rereading and the SimCity Effect in Interactive Stories.” Interactive Storytelling, edited by Henrik Schoenau-Fog et al., Springer International Publishing, 2015, pp. 27–39.
Noble, Safiya Umoja. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press, 2018.
O’Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, 2016.
Pearson, Craig. “Impressions: Prom Week.” Rock, Paper, Shotgun, 16 Feb. 2012, https://www.rockpapershotgun.com/2012/02/16/impressions-prom-week/.
Rusch, Doris C. Making Deep Games: Designing Games with Meaning and Purpose. 1 edition, Focal Press, 2016.
Russell, Stephen, et al. Spacewar! Digital Equipment Corporation PDP-1. Massachusetts Institute of Technology, 1962.
Ryan, James. Curating Simulated Storyworlds. UC Santa Cruz, 2018. escholarship.org, https://escholarship.org/uc/item/1340j5h2.
Ryan, James Owen, Ben Samuel, et al. Bad News. Performance and installation, interactive and computationally-assisted. 2016–17.
Ryan, James Owen, Michael Mateas, et al. “Open Design Challenges for Interactive Emergent Narrative.” Interactive Storytelling, edited by Henrik Schoenau-Fog et al., Springer International Publishing, 2015, pp. 14–26. Springer Link, doi:10.1007/978-3-319-27036-4_2.
Sakaguchi, Hironobu, and Motonori Sakakibara. Final Fantasy: The Spirits Within. Chris Lee Productions, SquareCompany, Square USA, 2001.
Samuel, Ben, James Ryan, et al. “Bad News: An Experiment in Computationally Assisted Performance.” Interactive Storytelling, edited by Frank Nack and Andrew S. Gordon, Springer International Publishing, 2016, pp. 108–20. Springer Link, doi:10.1007/978-3-319-48279-8_10.
Samuel, Ben, Aaron A. Reed, et al. “The Ensemble Engine: Next-Generation Social Physics.” Proceedings of the 10th International Conference on the Foundations of Digital Games, 2015. http://www.fdg2015.org/papers/fdg2015_paper_07.pdf.
Samuel, Benjamin. Crafting Stories Through Play. UC Santa Cruz, 2016. escholarship.org, https://escholarship.org/uc/item/6nw5x48d.
Shapiro, Daniel, Larry Lebron, et al. Composing Social Interactions via Social Games. 2015.
Shapiro, Daniel, Josh McCoy, et al. “Creating Playable Social Experiences through Whole-Body Interaction with Virtual Characters.” Proceedings of the Ninth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AAAI Press, 2013, pp. 79–85.
Stanton, Andrew, and Lee Unkrich. Finding Nemo. Pixar Animation Studios, Walt Disney Pictures, 2003.
Star, Darren. Sex and the City. Home Box Office, 1998–2004.
Sullivan, Anne, April Grow, Tabitha Chirrick, et al. Extending CRPGs as an Interactive Storytelling Form. 2011, pp. 164–69. ResearchGate, doi:10.1007/978-3-642-25289-1_18.
Sullivan, Anne, Michael Mateas, et al. “Making Quests Playable: Choices, CRPGs, and the Grail Framework.” Leonardo Electronic Almanac, vol. 17, Jan. 2012. ResearchGate, doi:10.5900/SU97819068971612012.17(2)_146.
Sullivan, Anne, April Grow, Michael Mateas, et al. “The Design of Mismanor: Creating a Playable Quest-Based Story Game.” Proceedings of the International Conference on the Foundations of Digital Games, ACM, 2012, pp. 180–187. ACM Digital Library, doi:10.1145/2282338.2282374.
Swink, Steve. Game Feel. Routledge, 2008.
Syzygy. Computer Space. Arcade. Nutting Associates, 1971.
Vee, Annette. Coding Literacy: How Computer Programming Is Changing Writing. The MIT Press, 2017.
Wardrip-Fruin, Noah. “Digital Media Archaeology: Interpreting Computational Processes.” Media Archaeologies, edited by Erkki Huhtamo and Jussi Parikka, University of California Press, 2011.
---. Expressive Processing: Digital Fictions, Computer Games, and Software Studies. The MIT Press, 2009.
---. How Pac-Man Eats. The MIT Press, 2020.
Wardrip-Fruin, Noah, Andrew McClain, et al. Screen. Virtual reality (projected). 2003–07.
Wardrip-Fruin, Noah, Adam Chapman, et al. The Impermanence Agent. Web browser and proxy server. 1998–2002, http://www.impermanenceagent.org/agent/.
Wardrip-Fruin, Noah, and Michael Mateas. Envisioning the Future of Computational Media: The Final Report of the Media Systems Project. Center for Games and Playable Media, UC Santa Cruz, 2014.
Waters, Mark S. Mean Girls. Paramount Pictures Corporation, 2004.
Wright, Steven T. “How the Mixed Reality Game ‘Bad News’ Brings Towns Like ‘Twin Peaks’ to Life.” Rolling Stone, Apr. 2017, http://www.rollingstone.com/glixel/news/mixed-reality-game-bad-news-brings-small-town-usa-to-life-w479065.
Zemeckis, Robert. Beowulf. Paramount Pictures, Shangri-La Entertainment, ImageMovers, 2007.
Cite this Essay:
- 1Our work in EIS is certainly not the only example of such work in practice. For example, John Cayley’s The Listeners is work that explores new artistic possibilities of emerging language technologies while being guided by (digital) humanistic insights about language, literature, aurality, and technology. Such creative projects can also serve to explore and expose the potentials and limits of, and demystify, new technologies of language, writing, and interaction — as examined humanistically by Lori Emerson in Reading Writing Interfaces and by Christian Andersen and Søren Pold in The Metainterface. This can be contrasted with the work of authors such as Mark Marino, myself, and others who have taken digital humanities (specifically, software studies) approaches to looking at human interaction with computational language technologies through the lens of Eliza/Doctor, apart from the creation or examination of explicitly artistic systems and works. All these approaches are valuable, but Cayley’s creation-oriented and humanistically-guided work is particularly in the vein discussed in this article.
- 2Our notion of social physics arose through connections between the arts, humanities, social sciences, and computing. The initial conception was influenced by strands of interdisciplinary social science, themselves influenced by arts and humanities practices such as dramaturgy (Erving Goffman’s The Presentation of Self in Everyday Life) and even frameworks of games and play (Eric Berne’s Games People Play). That said, because we explicitly wanted to create a playable model of social interaction that was learnable and actionable (as outlined by Mateas) our further development was inspired by our team’s humanistic engagement with media artifacts, to examine how they portrayed social relationships and dynamics. In particular, we focused on analysis of media about high school (such as Mean Girls) and other media focused on social relationships (such as Sex and the City). While one might expect that using humanistic analysis of artistic media would result in a model of social interaction that could be used for only artistic ends, the team developing the later Immerse project (which was informed by social science work on “good stranger” behavior) found our fundamental approach was also effective for encoding social states and possibilities in its domain.
- 3While I discuss Bad News in the context of the social simulation used during its generation phase, a fuller picture of the work would likely place it at the intersection of two computational media traditions. One is the generative tradition, which began with Christopher Strachey’s love letter generator — likely the first work of electronic literature, and first work of digital art of any kind. The other is the tradition of computationally-guided dramatic performance, as seen in the work of groups like Blast Theory. What connects them is the “story sifting” performed on the generative material to determine the material presented to the actor to guide their improvisation.
- 4When playing a game, a primary task for players is to figure out how it works. In a sense, players must internalize its models — which they often do unconsciously and imperfectly. My memory of the Prom Week development process is that both people on the development team and people beyond it (e.g., Jane Pinckard) believed that we didn’t want one of our first experiments with social physics to train people to internalize models of prejudice and oppression, but rather help them imagine a different world. While I believe effective work could be made that sought to expose and subvert societal oppressions through social physics, such concerns were also borne out by the difficulty we had subverting heteronormative expectations with Oswald’s level (as discussed in the main text).
- 5A number of scholars have been at work developing ways of thinking about procedural representations, especially in relationship to games. For example, Jesper Juul’s book Half-Real explains how the “real,” rule-based aspects of many games can only be interpreted through their fictional, representational aspects. Ian Bogost’s Persuasive Games presents “procedural rhetoric” as a way to interpret, and make, arguments that employ a game’s rules together with traditional representation modes. Mary Flanagan’s Critical Play describes how players can engage the meaning of games’ systemic representations critically, and creators can design to foster that critical engagement. D. Fox Harrell’s Phantasmal Media argues that we can organize the internal structure and operations of systems around the subjective and cultural meanings we aim to evoke and expose. Doris C. Rusch’s Making Deep Games seeks to guide readers in creating games that have appropriate metaphorical mappings to human experience, both in their operations and in the interpretations and feelings they evoke through play.
- 6To put it another way, if this article is read as a salvo in a “hack versus yack” debate, I’ll have failed.