Roderick Coover and Scott Rettberg reflect on the cultural values, political debates, power structures and architectures of exploitation underlying much of contemporary digital culture. As digital artists and collaborators, they also identify aesthetic reactions that actually combat what they critique. But for this to happen, we need literary works that are themselves produced, and actively circulating within digital environments.
A portion of the essay, focused on the ToxI*City project, is adapted from an earlier discussion published in ebr: Voices from Troubled Shores: Toxi•City: a Climate Change Narrative
Among other things, the Digital Humanities can be described as a reaction to a crisis. It is no exaggeration to say that the arts and humanities are under threat. In the USA, Europe, and elsewhere, humanities enrollments are sharply declining (a pattern which has been evident since peak enrollments in the 1970s but which has accelerated acutely since the 2008 financial crisis.) Many governments have been steadily reducing both humanities research funding and funding to the arts (during the Trump years in the USA these cuts have also extended to scientific research funding). We might say that our disciplines face a crisis of relevance as we are increasingly called upon to justify not only the utilitarian applicability of the humanities in an increasingly job-market focused university environment, but also our societal relevance in a more general sense.
Even in the comparatively progressive nations of Scandinavia, the humanities have not been spared the knife. In Denmark, higher education reforms linking the number of student places to job market needs as determined by industry are resulting in a scaling down of the humanities. Similar cuts have taken place in Sweden, where humanities courses have been cut from many technical universities. In Norway, the government conducted a large-scale evaluation of the humanities during 2017 with an express intention to evaluate the societal value of study in humanities courses. Thankfully, the findings of the “Humanoriamelding” were generally positive–the humanities were found to have enduring value to society—but the report also recommended some significant course corrections. Among them is that the humanities engage more directly with “great societal challenges,” such as climate change, mass migration, and how we as a society are adjusting to the digital turn. This is coherent with research funding trends in the EU, where much of the research funding in social sciences and humanities has been shifted from basic research to integrated projects in which the humanities are expected to partner with researchers on the hard sciences on large-scale projects.
This is neither an entirely positive trend nor one without potential opportunities for artists and humanities researchers and their disciplines. It offers particular opportunities for authors and artists working in the field of electronic literature and digital art to consider the potential role of their own practice. Without buying into reductionist or neoliberal notions of the value of particular forms of human knowledge, when we face the types of challenges that we do in this century, it remains worth asking: what can be done? In 2020, this question surfaces more prevalently than ever, in a moment when the world is simultaneously confronting a global pandemic and, through the Black Lives Matter movement, the consequences of systemic racism that stretch back at least 400 years. What can our work and our field do to effect positive change?
We do not argue here, as perhaps for example the funders of the European Union research framework might, that the humanities need relate directly to major challenges such as climate change, mass migration, and resurgent nationalism in order to remain significant, but instead that they can and will play a vital role in helping us to process and address these problems. A great deal of work in the field of electronic literature has been driven by innovation for innovation’s sake. During the past several decades writers and writers have been presented with an amazing new toolset and a sandbox as expansive as the World Wide Web, presenting us with opportunities to experiment and create new forms and discover new narrative and poetic uses for platforms intended by their developers for other purposes. A spirit of exploratory play has been essential to the development of electronic literature as a field of practice. But literature has never been essentially about the novelty of new techniques and formal innovations. Most writers adhere to the idea that a good story must ultimately be about something, and that beyond simple wordplay, a remarkable poem should offer some kind of meaning or at least some kind of intent to spur significant reflection.
Just as novels, beyond their aesthetic and entertainment value, have always served as reflections of the cultural values, political debates, and societal challenges of the time in which they were produced, contemporary electronic literature and media art are providing us with new toolsets for processing the significant shifts, from the digital turn to the Anthropocene mass extinction, that are defining our contemporary society and our relation to the planet. This is not to argue for a purely utilitarian approach to artistic production: indeed, artworks that are conceived first and foremost as the communication of scientific research or as ideological statements tend to fall flat. We argue instead that great stories and affective artworks tend to arise from human struggles with great challenges. There seems little point in continuing to produce reams of fiction focused on marital conflicts or family struggles in middle-class households when the sea level is rising at such a rapid rate that many of the homes in which such domestic dramas occur will be underwater before any generational shift can take place. We might take it as a truism that in the near future, virtually all fiction will be ecological fiction, by default. Our collaborative works, which we will discuss below, are driven by an awareness that we occupy a zone of contemporary crisis which is no longer avoidable. Zones of contemporary crisis are not only where artworks can have significant societal impacts: they are also where some of the most urgent and compelling story material will be found.
During a period when numerous governments and educational institutions are questioning the value of the humanities to society in comparison to the hard sciences or market-driven professional training, we argue that critical projects driven by scientific research, non-fiction documentary evidence, storytelling, and digital media aesthetics can help society to grapple with some of the most significant challenges of our time.
Digital Literacy and Critical Digital Media
Few people would disagree with the assertion that the digital turn has had significant impacts on the ways that most human cultures write, communicate, socialize, and structure the basic functions of our polity. While our logical understanding of the nature and the specific impacts of these changes lag behind the changes themselves, we take it as a truism that digital literacy has become imperative in the contemporary period. Certain aspects of digital literacy are obvious: students for example need to learn how to evaluate their sources of information differently than they did in the past, with an awareness of the effects of the filter bubble, fake news, and rampant propaganda on online discourse. Digital literacy also demands a heightened awareness of the media ecology in which individuals, their communications, and their social interactions are mediated by very large and very powerful corporate entities. Companies such as Google, Facebook, Twitter, and Amazon have more access to and control over repositories of human language, personal data, and contemporary cultural memory than any government has ever had. In comparison, even the Stasi during cold war East Germany had less control of human communications.
Digital literacy further demands new models of “critical making.” In their introduction to their anthology DIY Citizenship: Critical Making and Social Media Matt Ratto and Megan Boler articulate DIY citizenship as “a twenty-first century amalgamation of politics, cultures, arts and technology that in turn constitutes identities rooted in diverse making practices” that extends to the efforts of interventionists, makers, hackers, modders, and tinkerers” (18). Electronic literature and other digital arts practices can function as powerful forms of critical making, particularly suited to address the societal changes wrought by the digital turn.
Authors and artists including among others John Cayley, Daniel Howe, Helen Nissenbaum, Jason Huff, Mimi Cabell, and Ben Grosser have produced digital language artworks that illustrate particular practices of Google and Amazon. These corporations are impacting language, not only in the sense of how and what people communicate, but also how language is controlled, used, and commoditized. Google has gathered the largest archive of human language ever assembled and treats it as their proprietary corpus. Facebook has gathered the largest collection of “people data” ever harvested and retains the rights to use it as they see fit. Amazon has mapped the buying habits of everyone who uses its service and put its listening devices into our homes.
American Psycho (2012) by Mimi Cabell and Jason Huff considers the poetics of citation in the networked society. Cabell and Huff published a version of Brent Easton Ellis’s novel American Psycho that did not include any of the text of the novel but instead a series of chapter titles and annotations for web-based ads on each page of the book, which is otherwise blank. These were harvested by feeding Ellis’s novel through Gmail one page at a time and then copying the text of the ads that Google displayed with it in the Gmail browser. The project is not about Ellis’s novel per se but instead a consideration of how Google reads the texts we send to it and responds with related advertising.
Another type of response to the challenges of corporate control of the Internet are artistic reactions that actually combat what they critique. Daniel Howe and Helen Nissenbaum’s TrackMeNot (2012) is both a software artwork and a practical tool for netizens who want to intervene in violations of their privacy. TrackMeNot is a web browser plugin. When it is installed, every time a user sends a search query, the plugin also sends a number of other queries that are linguistically but not semantically related to the actual search term. This makes search query useless to Google in terms of developing an individual profile. AdLiPo (Howe, 2014) is a selective ad blocker that replaces selected ads with short generated critical language about the nature of advertising, in a process that is in part based on a machine reading of the contents of aw page. AdNauseam by Howe et al. (2015) is another ad-blocker that on the surface seems to work like any other. In addition to blocking the ads, however, the program simultaneously registers a click on every single ad on the page, both disrupting the economic system and trust relations between advertisers and their clients and obfuscating any user profiling on the basis of what ads are clicked. The plugin also harvests images from all of the ads that are blocked and clicked. This produces a collage representing a profile of the type of consumer that the advertisers on the websites you visit believe you to be. ScareMail (2011) by Ben Grosser is a Gmail extension that is intended to disrupt NSA surveillance. The program generates “scary mail” by attaching short nonsense stories to every message sent, using words from a list of terms that NSA crawlers use to trigger investigations, flooding the agency with useless nonsense information. The stories created are syntactically close enough to normal language to be interesting as texts, interventionist spam.
John Cayley’s The Listeners (2016-) functions as a critique and an engagement with the author calls “aurature”—a new form of literature that is being enable by transactive home assistants such as Alexa and the Google home assistant. These are high quality speakers with microphones that connect them to Amazon or Google, respectively, and their AI systems. The Listeners is an Alexa skill. The user can interact with The Listeners by describing their emotions: for example, “I am filled with anger” or by asking Alexa to continue. The base of the discourse the interactor has with Alexa while using the The Listeners is metafictional. The conversation that evolves is focused on this new form of relationship, between a person, an AI agent, and a corporate entity that is harvesting the “intents” of people who have welcomed these listening devices into their home. As Cayley writes, we have only barely begun to “to comprehend the implications of these extraordinary circumstances with regard to hospitality, privacy, language, human persons and their interrelations.”
Many of the “Netprovs” organized by Rob Wittig and Mark Marino also engage critically with the practices of the multinational corporations benefitting from it, and the habits of users who have to some degree become complicit in their own exploitation. These collective writing projects often take place on social media and are usually centered on some absurdity of contemporary digital culture, such as the “I Work for the Web” (2013) netprov, which proposed a formation of a union of likers and re-tweeters who objected to the practices of internet companies who monetize their online labor without paying them for it. In netprovs, the organizers propose a general story frame and seed some plot events, but the majority of the writing that takes place is produced by participants who create characters and write in their voices, interacting with each other in online platforms such as Twitter. This structure invites the players to write self-reflexively about phenomena such as the “Fear of Missing Out Online” syndrome – where social networks continuously send users notifications to lure them into spending more and more time on social networks (explored in the 2018 #BehindYourBak netprov), or the behavior of users who loudly declare that they are taking a break from technology, only to meticulously document their sabbatical from technology in social network postings (in the #1WKNOTECH netprov). While these interventions are comic and playful, they provide participants with an opportunity to defamiliarize their own online behaviors and perhaps recognize more plainly the power structures and architectures of exploitation underlying much of contemporary digital culture.
Electronic literature and related digital arts practices offer us new means of critical reflection by engaging with the societal consequences of the digital turn that are produced within the environments that they critique. It is important to emphasize that there is a distinction here between forms of critique that are medially embedded in this way from, say, a critical paper analyzing a particular digital culture phenomenon. Performing a netprov about addiction to social networks within social networks, or critiquing emerging relationships between corporate actors, human users, and the listening devices they are welcoming into their homes via an Alexa “skill” allow authors not only to critique aspects of the digital turn, but to illustrate the object of critique procedurally within the systems themselves.
The above examples illustrate how works of electronic literature and digital art can engage directly and critically with the changing societal circumstances of the digital turn, but beyond the challenges that are specific to the digital, authors and artists are critically confronting other pressing challenges, such as climate change. In what follows we turn to our own work to describe how we have tried to do this using the particular aesthetic affordances of digital media art. The group of cinematic and computational narrative projects we have produced through the collaboration “CRchange” engage in a direct way with the circumstances of the contemporary climate catastrophe and in particular to the human reaction to it, which has thus far been inadequate to the challenge at hand.
The Origins of a Collaboration: The Catastrophe Trilogy
We had been friends for many years and collaborated on projects such as events and exhibitions before it occurred to us to collaborate on creative work. An opportunity arose in 2010, when Rod was appointed as a distinguished scholar-in-residence at the University of Bergen. Rod’s arrival in Bergen came shortly after the eruption of the Eyjafjallajökull volcano in Iceland. Although this eruption was comparatively minor in comparison to other catastrophic volcanic eruptions, both elsewhere and in Iceland’s history, it was notable for the significant disruptions it caused for European air travel during a six-day period in April 2010, when many flights were grounded across the European continent and Nordic countries, due to microscopic particles of ejected tephra in an ash cloud that rose to altitude of nine kilometers and dispersed in the atmosphere over Europe, placing it directly in the path of hundreds of scheduled flights. Because the tephra can potentially interfere with the operations of jet engines, airlines had no option but to ground flights, and the skies over Europe went briefly silent of the roars of jet engines, as thousands of stranded travelers were left to find train and boat alternatives to their flights home. The ash spread wide from the volcano in Iceland. During that week if you left a cup on the balcony while it was raining in Bergen, Norway, you would find particles of volcanic ash floating on the surface. Scott began to research how historic volcanos had affected Europe in the past, and we discussed this phenomenon during Rod’s visit. This evolved into a discussion of cycles of catastrophe: how quickly we as a culture tend to absorb, dismiss, and move on from major natural disasters—even those that have significant impacts such as the loss of human life—shortly after they occur. We agreed that this might provide a compelling basis for a film project.
Rod shot a series of 360° panoramas around Bergen, including an evocative image shot from a pier off Nygårdsgaten in Bergen, overlooking the fjord at the center of town. The image included the body of water, a temporary Tivoli with carnival rides for children, a great deal of new construction, and a heavy mist rolling in over the mountains and a bridge in the distance. The image was pervaded with a sense of mystery and foreboding.
Det siste utbruddet/ The Last Volcano by Roderick Coover and Scott Rettberg, 2011
Scott used an 18th century source text, an account written by an Icelandic farmer, as the basis for the script. The script was written in English, and then roughly translated into Norwegian, before we recorded it with Gro Jørstad Nilsen. Gro, a Norwegian novelist, refined the translation. The monologue begins with the phrase “Some say that what happens once never happens again. Others say that nothing only happens once” before describing a summer in Bergen in which the “haze did not abate” and the “coppery clouds hung low and thick.” The speaker describes a time in which the “lambs fell in the field like so much rotten fruit when the youngest were vomiting blood and the oldest had died.” While the account was rewritten to accent some poetic details and to set it specifically in Bergen, the majority of the words in the script were those of the farmer describing life after the eruption. While we were recording, Gro’s husband, Jan Arlid Breistein, interrupted Gro to ask when this had taken place – perhaps the 14th or 16th Century? While this chance interruption meant that we had to re-record the reading, it occurred to Rod that the interruption might work into the finished piece. In the end, the film includes two readings of the text, and two cycles through the visual panorama. The first performance is a straightforward reading of the text, but as the second takes place, Breistein’s voice introduces a kind of temporal anxiety. Though what is described is a tragedy of the past, we are left to wonder whether it is actually suggests a present nearer to us, on the imminent horizon.
We produced two more short films over the following couple of years which returned to the theme of revisited catastrophe, recasting the cataclysms of the past as reflections of the problems of the present. Each of the three films in Catastrophe trilogy involve historical disasters. Rotter og Katter / Cats and Rats (2011) took as its subject matter the Black Death, the plague which devastated Norway and much of the rest of Europe during the 14th century, but recast as a first date in the form of a Skype call between a male American epidemiologist and a Norwegian woman (who may or may not be a witch). While a comic short that explores cross-cultural miscommunication, the film also exposes how cultural biases can make a bad situation—a pandemic—even worse. Norwegian Tsunami/Norsk flodbølge (2013) is focused on the Storegga Slide Tsunami, a megatsunami which is believed to have occurred around 6500 BC. The tsunami was unleashed when a submarine landslide occurred off the Western coast of Norway. The tsunami is believed to have devastated the coastal Mesolithic population living in Norway and England at the time, and to have flooded Doggerland, the land bridge that once connected England to mainland Europe, and thus creating the English Channel. In this case the meditation on the tsunami is set as a conversation on an oil platform in the North Sea, in an area where new drilling appears to be causing geological disruptions.
Rats and Cats :: Katter Og Rotter by Roderick Coover and Scott Rettberg, 2012
In The Catastrophe Trilogy we considered how historical disasters, removed temporally far enough from the present to have been forgotten by most but historians and geologists, might serve as lenses to help us contemplate our own relation to environmental catastrophes. These cycles of trauma and forgetting would prove to be a significant concern in our further work. We also began to consider how algorithmic variation, recursion, and chance could inflect the media ecology of our collaborations.
Three Rails Live: An Experiment in Combinatory Cinema
Three Rails Live (2011-13) is a work that was first and foremost an experiment in integrating some of the technical aspects of combinatory poetry generation with narrative cinema. It was also an experiment in collaboration through reiterative re-interpretations of the same set of materials. We collaborated on this project with Nick Montfort. All three of us had collaborated with each other on projects previously, but all three of us had never before collaborated together on one project. Each of the three of us came at the same set of materials with different types of approaches and perspectives which we eventually combined into a single work that ultimately forms a coherent whole. The title was derived from the fact that we were doing two events showcasing our various collaborations at Penn Writer’s House in Philadelphia and then at Brown University. The idea was we would each develop components of the project individually, then bring them together and finish a first draft on the train from Philadelphia to Providence.
The process of developing the project was recursive and reiterative. Rod sent a selection of short video clips and images to Scott and Nick. Scott viewed the clips and sorted them arbitrarily into themes (Landscape and Fate, Tourists, Death by Snake, Industrial Sites, Trains, Flood, Toxic, Flight, Stripped, and Third Rail), wrote short three short narrative segments for each theme, and then recorded readings of each of these narratives. As Rod read the text Scott sent, he added new materials in response to the themes that developed. Nick selected particular images, and, borrowing a technique from Harry Mathews, wrote “perverbs”—remixes of two different proverbs that subvert the original—for each of the texts paired to an image. Nick also constructed a title generator that arbitrarily creates a title for each run of the work. The system the authors constructed selects two image sets and two of the narrative recordings from a constrained random selection. A perverb with a moral to the story is then assigned and the process begins anew. The system thus results in short narrative videos with new juxtapositions of images, texts, and perverbs every time it runs. All of the texts and images emerge from this aleatory but thematically determined method.
Although the narrative texts were written separately as prose poems, as the themes developed, a narrative voice took shape and a fragmented but coherent story took form, focused on a middle-aged man, whose life and family connections had begun to dissolve after the loss of his spouse. His personal dissolution and disconnection from family and friends mirrored the degradation of the environment he encountered around him as he traveled, adrift, in a world in which connections between the individual, society, and nature were becoming increasingly untethered. The system assembles the individual “stories” in pairs and then randomly assigns them to videos (first in a masked set of images, and then an unmasked set), before assigning them a moral. We originally had the work coded so that this selection would be completely random on each run, which meant that narrative fragments might repeat after one cycle. But when we recoded the work for a subsequent version, we reset the parameters so that as each clip and each narrative segment are played once, they are removed from the pool. The work will now not repeat a clip or a narrative segment for 45 minutes, so that if viewers engage with the work over that time, they will experience the “whole story” albeit one in which the order of the narrative segments and the juxtapositions between image and text will never have been seen before.
In Three Rails Live the video clips and narrative segments are not yoked together in advance, but by the system at run time. The connections between the moving image seen and the story heard will thus sometimes seem strange or unintended. But two factors lend the video and narrative elements a sense of coherence. The first is that though the specific pairings are not determined in advance, the texts were written in response to the images in the set as a whole. You may hear a story that reminds you of a text you saw a couple a few minutes before even if you cannot make an explicit illustrative connection between the image you are seeing and the story fragment you are hearing simultaneously. The thematic threads are woven throughout the piece. Secondly, the human mind is drawn toward narrative closure. We tend to find metaphoric connections between whatever image we are seeing and the story we are hearing at the same time, whether those connections are authorially intended or not. Humans are meaning-making machines.
Sample screen recording of the combinatory film Three Rails Live by Roderick Coover, Nick Montfort and Scott Rettberg, 2013
In Three Rails Live the fragmented narrative structure and the random connections between image and text serve as a kind of objective correlative for the subject matter of the stories themselves, which often describe situations—grief at the loss of a loved one, the uncontrollable flooding of a family home, sudden die-offs of wildlife—which are difficult for the protagonist to process in any kind of cohesive way. Absent coherent explanations for phenomena that are difficult to process logically or emotionally, the main character seeks patterns in the phenomenological experiences presented to him. He struggles to understand his role in a narrative that is largely beyond his control. As we developed this project, we realized that generative combinatory structures are well-suited to depicting what Timothy Morton has described as “hyperobjects” such as climate change. Just as humans are driven to narrative closure when presented with images and narratives that may or may not in themselves cohere, large-scale catastrophes that unfold over on scales beyond our normal modes of reference leave us to make sense of hyperobjects which are unknowable to as complete entities, by connecting those fragments of the whole which are exposed to us.
Toxi•City: A Climate Change Narrative
Toxi•City: A Climate Change Narrative is a combinatory narrative film that uses computer code to draw fragments from a database in changing configurations every time it is shown. As some stories seem to resolve, others unravel. Just as with the conditions of ocean tides and tidal shores, the stories cycle and change without clear beginning or end. Rather, individuals grasp for meaning from fleeting conditions of a world in flux. As the characters' paths intersect, story threads come together. These intersections offer moments of resolution, contact and visions of the future, before the narratives are broken apart and a fresh cycle begins again.
The first iteration of Toxi•City was produced as a commission for the Chemical Heritage Foundation Museum in Philadelphia for an exhibition titled Sensing Change, which was intended to showcase artworks based on scientific research into the causes and effects of climate change. Toxi•City asks how conditions of life would change if repeated storm surges and tides flooded the densely populated lands with toxins from the hundreds of sea-level petrochemical industry sites and post-industrial brownfields. The fictions are interspersed with nonfictional accounts of deaths that occurred during storms in the area, most notably Hurricane Sandy. The narrative events are drawn from actual events and predicted conditions faced in the Delaware River Estuary as well as events along the nearby coastal shores of New Jersey and New York.
Toxi•City builds narratively on top of the gathering of non-fiction material. Rod began the process of developing the project by going out into the environment of Philadelphia where he lives and following sets of ideas, systems of images that he gathered, collecting buckets of related material, as he developed a kind of process of looking at conditions of change, particularly in recent times looking at a lot of sites where one might observe the impact of climate change.
The process of making Toxi•City began as one of visual research. Rod spent time kayaking on the Delaware River in its tidal zone, a long stretch of river that passes many of the large cities in the American seaboard. The river is home to many of America's largest oil refineries. It was once home to the American steel industry and shipbuilding industries and the central port for coal and many of the other dirty fossil fuels of the past. At the center of the Delaware the largest city is Philadelphia, which was in the seventeenth and eighteenth century the main economic hub for the US, only to be surpassed by New York quite late in the 19th century. The landscape of the Delaware River estuary is marked by centuries of early industrial development, including a number of other cities that are in rather poor state, like Camden, New Jersey, Wilmington, Delaware, and Trenton, New Jersey.
Rod’s initial research began with boating along these sites, recording an industrial landscape and beginning to speculate what would happen to these landscapes as rising seas from climate change would wash over a landscape of brownfields, of environmental cleanup zones, of active chemical repositories, and begin to mix all of this chemical waste into the land, washing over the cities, not just with salty corrosive seawater but with this toxic waste mix. He explored these landscapes and photographed them, recording what he could see within a potential tidal zone of future floods.
We started talking about how to develop a project around this situation in the Delaware River estuary. Rod lives in this area and Scott used to live in the region as well. We began by asking how people who live in the area have adapted and might adapt as terrible catastrophes related to sea-rise occur, and we followed outcomes of Hurricane Irene, Hurricane Katrina and the many more minor storms that impacted people's lives. While some leave, others die, most just make adjustments in their lives in landscapes that are imagined in new ways. Life goes on but life choices, feelings about places and human relations may all be altered.
Rod started by photographing landscapes of the Delaware River estuary from a kayak first to collect documentary material and then to gather the substance of what would become a narrative film. It was a process of looking for documentary evidence for a changing future imaginary: the worlds we imagine we might inhabit in the years ahead. We then began to think about the kinds of characters we'd like to follow, and the differing conditions they might face. Given the infinite permutations involved in looking forward, a combinatory model seemed to offer a perfect structure. An array of images constructs one vision of the future, which is then set against another, and another, and another, as a panorama of conflicting, but nevertheless coexistent, interpretations of the situation of climate change build upon and react to one another.
Mapping was also an important part of this initial process. Rod used historic maps to locate former industrial sites that might be disturbed with new floods. This landscape, which was so transformed by the 19th Century industrial revolution, is full of hidden repositories of toxic soils that have since been built upon or left fallow. This hidden landscape is complemented by a contemporary petrochemical industrial landscape of refineries and chemical plants, while other maps like those of human water use, agriculture and bird migration show how interconnected our lives are with both industrial and natural conditions. The research resulted in the development of The Chemical Map (2013), an online resource produced as an interactive map identifying over 200 sites within flood zones of potential toxicity, sites that would impact where people live, drinking water supplies and so forth. The map provided additional structure for the gathering of images and a means of setting an objective set of data against Rod's own personal observations.
The Chemical Map also grew into a kind of a nonfiction record in the form of an interactive log of exploration, which Rod passed on to Scott. He also started doing additional research on deaths that occurred during Hurricane Sandy and on the potential impact and what life would look like in a near future if events like Hurricane Sandy and this kind of regular flooding we see now became so frequent that it would force people to change where they live or their habits of life.
It's out of this kind of interactive writing and picturing a landscape in a nonfiction form that we move into narrative and we carry that research into a fiction and that's where we then began working back and forth, to develop characters and develop an approach to a feature-length film with a combinatory structure. We started working on this project right before Hurricane Sandy hit the East Coast of the USA. When the storm came, Scott was at a conference in Edinburgh, and one morning picked the newspaper outside the hotel room door to see a picture of President Obama standing on this devastated shoreline comforting a woman who had just lost her business in the storm. Scott looked at these pictures and thought “that place looks remarkably familiar.” It turned out that Obama was standing about two blocks from the house where Scott used to live. Looking at Flickr photos and local New Jersey newspapers online revealed images of the old neighborhood pub two blocks from Scott’s former home, the Rod n' Reel, completely washed out, inundated, destroyed. The whole north end of the island where Scott used to live was just completely wiped out. As we were working on this project, the situations we were writing and making a film about were not abstract. The places affected were places that we knew, the people affected were people we saw walking down the streets in neighborhoods we had lived in.
The contemporary situation of climate change is not abstract. Even though we could describe this film as a speculative narrative, it is a very near-term type of speculation, addressed more to the imminent horizon than the distant future. One of the functions of art about climate change is to look hard at the near future, in an analytical way, but also in an empathic way. One of the powers of media art is to enable a different sort of affect, and perhaps generate a different kind of identification, than written discourse alone, so that we have a better sense of the impact of these problems in human terms.
There are two narrative layers in Toxi*City: one which is absolutely non-fictional. These layers are structured in a way both algorithmically and rhythmically. In between the character-driven narrative segments that we call a chorus, made up of death stories. Scott adopted these stories mainly, and fairly directly, from obituaries. This is a very sort of odd genre that you find in newspapers when people die in a major disaster. They are both about the disaster itself, and they focus in some sense on how the people died, but they also try to do the work of marking the life and the passing of this particular person, this particular spark of human life that has been extinguished in the event.
Sample screen recording from the combinatory film Toxi•City: A Climate Change Narrative by Roderick Coover and Scott Rettberg, 2016
A number of our films have tried to engage in some way with how we, as humans, as individuals, as a society, process catastrophic events and deteriorating environmental conditions. We often rush past these catastrophes, perhaps to avoid dwelling on the trauma. Maybe we read that 100 people died in New Jersey, or 300 people died during the storm, and we absorb that as a statistic, and we move very quickly past it. By including those death stories, and by including enough of them that you will hear of different people’s deaths every time you experience the work, we're trying to keep them as a sort of persistent reminder, to some extent humanize what is happening as a result of climate change, through the inclusion of that layer.
The other layer of the story is a fragmented speculative narrative. We wanted to represent a number of different perspectives, a number of different types of voices. The six characters sort of represent different age groups, different socioeconomic groups, as well as different types of reactions to the events. Some of this was again based loosely on the documentary research that Rod and his students did. The voice of the fisherman character for example, and some elements of his story were adapted from interviews of longshoremen that Rod's students found in union archives. The voice of the FEMA worker, in a way serves an expository role, to bring in factual information about all of these toxic waste sites on the chemical map, this incredible concentration of hazardous material, all of this potential catastrophe lying in wait two or three feet beneath the topsoil. We also wanted to not simply represent these characters as victims. They're victims, we're all both victims and perpetrators of climate change, but we know that we will see different types of constructive and destructive responses to these types of events. Toxi•City includes voices like that of the pig farmer, who is constructive in his own way in that he is adapting to the new circumstances to survive, but is also the voice of some truly ignorant perspectives on climate change that are still very much part of the contemporary discourse around it.
The combinatory structure of Toxi•City is different from that of Three Rails Live—in this case the images and the texts are yoked together, but there is still a clear poetic reasoning for the combinatory structure. There are a few dozen of these death stories in the pool. We're trying to take a kind of a panoramic approach to what happens in these sorts of events. The fact that you can watch the film over and over again and each time hear about the passing of different people underscores a point we're trying to emphasize. Each trauma that climate change produces is a singularity, it is a human trauma that is experienced individually, as well as a planetary trauma. Each time you watch the film, you are going to hear different death stories, experience these singular traumas, each of which occurred to a different human being who actually lived this moment and actually died.
Toxi•City has a more clearly defined narrative flow than some of our other combinatory work in that there are actually clearly defined beginnings, middles, and endings but the mix of them and the order that they are presented in is variable. You never experience a whole story on any given run. Much of this has to do with the experience of simultaneity—and how the complexity of these events simply can’t be captured in any linear narrative. These types of events happen everywhere at the same time, and it is never possible to know them, or to understand them, completely. Nevertheless, to progress in any way, we need to try to come to grips with this kind of scattered understanding and not be paralyzed by it.
The narrative structure operates like the visual metaphor of the waves you see throughout the film. Each time the film runs, there's another array of stories to be told and another set of deaths that happened, but also new potential. With each refresh you've got another set of experiences, another wave of stories, but also fresh sense of possibility for these characters who need somehow to live with hope in facing these situations. We need to maintain some kind of positive sense, some consideration, of how we can manage to go on in spite of these conditions. The combinatory structure allows for a sense of the immensity, diffusion, and ultimate unknowability of hyperobject-level events in such a that it becomes an objective correlative for these panoramic catastrophes.
The CRchange projects are but one example of a growing group of electronic literature and digital artworks that are encountering climate change as a subject and exploring how digital narrative and media art techniques may be exploited to observe and communicate its effects both on the human condition and on the planetary ecology of which humans are a part. J.R. Carpenter’s The Gathering Cloud (2017) for example considers the environmental impact of cloud computing in a visual hypertext assemblage of both fragments of environmental texts and metaphoric imagery of animals appropriated from cloud services. David Jhave Johnston’s Extinction Elegies: A Post-Fukushima Interactive Video Poem Tht Introduces Mutations into the DNA of Meaning (2011) meditates on the problem of nuclear waste—the fact that while nuclear power plants are built to last about 30 years, their radioactive spent fuel need to be stored for thousands of years—through a video poem that introduces mutations into the text with each successive generation. Johannes Heldén’s Encyclopedia (2013-15) is a work that produces AI-generated descriptions of extinct species in a kind of endless elegy for animals that never existed, underscoring the fact that for future generations there will be very little difference between fictional species that never existed and real species that no longer exist: a planet flourishing with biodiversity will in any case be beyond the reach of human memory. In each of these cases, the authors are exploring how computational technology, in its complicated relation to the ecology, can provide different modes, metaphors, and procedural representations of climate change than narratives in other media could. It is likely the case that climate change will continue to be a focus of many works of electronic literature in coming years—not only because authors are particularly driven to activism but also because of the omnipresence of climate change itself. The conditions of a rapidly degrading planet are such that the majority of fictions produced in the near future will be climate change narratives, just as all novels written during the period around World War I and World War II were in some way war novels, whether or not war was their intended and specific focus. Climate change is a subject we no longer avoid, because it surrounds and engulfs us.
The Politics of Our Violence: Digital Narratives of Conflict
If climate change is dominant meta-narrative of our age—that which is framing other long-term crises derived from it: such as sea level rise and coastal flooding, mass migration, and mass extinction, it is far from the only great societal challenge we currently confront. From the breakdown of nation state alliances formed in the 20th century, to mass scale income inequality, to sectarian violence, to challenges to basic human rights, complex political and social conflicts abound. Narratives guided by ideology often reduce complex situations involving competing human interests in a way that only serves to reinforce biased views and to increase polarization. What role might electronic literature and digital artworks play in not only communicating such complex human conflicts, but also promoting constructive dialogues that might contribute to defusing them? Reflecting on the polarized state of contemporary political discourse, interactive digital narrative researcher Hartmut Koentiz claims that rather than conventional journalism, “What is needed instead are narrative representations of complexity. This means re-inventing narrative by using the opportunities afforded by digital media to create representations that contain competing perspectives, offer choices and show the resulting consequences while allowing for repeat experiences.” Situations which are immensely complex and variable may call for representational media, such as database structures and interactive forms, that are themselves variable. These structures may allow us to see how in some cases, making a simple choice or changing a single variable may result in an entirely different narrative, or a different reality.
A number of works that could be described as non-fiction or documentary electronic literature indicate some paths that interactive digital narratives might potentially take in representing the complexity of actual situations of human conflict. A number of Sharon Daniel’s works, such as Public Secrets (2008) and Inside the Distance (2015) have employed hypertext and hypermedia, digital cinema, and interactivity in projects that communicate narratives of difficult situations, such as the mistreatment of women prisoners in American jails, or mediation between assault victims the perpetrators of the crimes. The use of interactive techniques creates a different kind of affect and involvement in these narratives than would a simpler documentary style. For Daniel, each research problem is a site, not a story. Her interactive structures allow users to connect conditions shared by the subjects, and the format allows stories to emerge from the experience rather than drive it. The process gives voice to the disenfranchised, allowing users the time to hear differing views and choice in how voices and views are followed. With Inside the Distance, Daniel also uses interactive video. The dramatic work builds upon an innovative program in Belgium known known as restorative justice by which victims and offenders are offered the opportunity to take active roles in the conclusion of their case through mediation. Separated by a criminal justice mediator and the length of a table, victim meets criminal. Each listens to the other's account of the experience of the crime and its impact, and the users not only listen to all sides but find themselves within the positions. The interactive format allows users to switch among the stories given by victims, offenders, mediators. The format allows users, like the participants, to build understanding about how crimes impact lives, and how a restorative criminal system could address these issues in ways that our current system of punishment might not.
Egyptian-American author Amira Hanafi’s A Dictionary of the Revolution, winner of both the 2019 Public Library Prize for Electronic Literature and the 2019 New Media Writing Prize, is another compelling example of Critical Digital Media. In this case, Hanafi interviewed many people of different socio-economic, religious, and ideological backgrounds, asking them for brief responses to concepts written as a single word on a note card. She then cross-referenced and visualized these responses into a lexicon and network visualization, allowing the reader to move nonlinearly and conceptually between dramatic individual narratives and experiences of the 25th of July 2011 Egyptian revolution. The work both captures a panoramic impression of the chaos of a single revolution and perhaps reveals something conceptually about the nature of revolution as a concept itself. While the narratives of events that occurred during the revolution and afterward are often in conflict with one another, the work as a whole reiterates the fact that no single narrative of the events could account for the impacts that it had on all of the different parties involved. This fragmentation extends beyond easily reducible divisions of sect or class or gender: one’s perception of a revolution might even extend for example to one’s location on a given day or proximity to acts of violence. A Dictionary of the Revolution provides a truly multivocal representation of the kinds of intersectionality, shifting temporary alliances, and contingent perceptions that complex events of this kind generate.
Hearts and Minds: The Interrogations Project (2014-16) is a project that we co-created with Daria Tsoupikova and Arthur Nishimoto. Hearts and Minds was originally developed in the CAVE2TM virtual reality theatre environment in the Electronic Visualization Lab at the University of Illinois Chicago. The project is a multisensory artwork that addresses a complex contemporary problem: as American soldiers returned from wars in Iraq and Afghanistan, it became increasingly clear that some of them participated in interrogation practices and acts of abusive violence with detainees for which they were not properly trained or psychologically prepared. This has in turn left many soldiers dealing with Post-Traumatic Stress Disorder on their return home and left many unresolved questions about the moral calculus of using torture as an interrogation strategy in American military operations.
This project is based on scientific research that addresses a significant problem—the cultural inheritance bequeathed us by the tactics employed during the war in Iraq. Based on interviews of soldiers who participated in or observed acts of torture rather than purely statistical data, this problem was then further narrativized through visualization techniques. The immersive 3D environment of the CAVE is here intended to provide an affective environment that produces a space for interpretation. The visualization environment serves as a dispositif for enacting individual and cultural memory of an institutionalized atrocity.
The project was developed through a unique collaboration between artists, scientists, and researchers from four universities. The project is based on interviews of American soldiers conducted by political scientist, Dr. John Tsukayama. Scott adapted a script from material gathered in 17 interviews that Tsukayma conducted with American soldiers who had been directly involved with interrogations of prisoners in Iraq, many of which crossed the line into the zone of torture. Rod conceptualized and developed a visual environment that involved surreal panoramas of desert landscapes layered behind 3D modeled domestic spaces designed by artist and visualization researcher Daria Tsoupikova. Computer scientist Arthur Nishimoto did the majority of the custom coding of the project, and sound designer Mark Partridge contributed soundscapes. Dr. Jeffrey Murer, Lecturer on collective violence at St. Andrews University, Scotland also contributed as a consultant on the project.
Hearts and Minds: The Interrogations Project by Roderick Coover, Scott Rettberg, Daria Tsoupikova and Arthur Nishimoto, 2014-16
The project presents the audience with a narrative environment that begins in a reflective temple space with four doors opening to ordinary American domestic spaces: a boy’s bedroom, a family room, a suburban backyard, a kitchen. The performer navigates the environment using a wand, a 3D mouse, to interact with and control a VR experience in the CAVE2™. The virtual scene is continuously updated according to the orientation and position of the head, as measured with head and arm trackers, and the 3D view of the scenes is focalized on this perspective. Moving through and exploring each of these rooms inside the virtual scene creates a sense of being immersed in the virtual environment. Using a wand with buttons, the navigator triggers individual objects, such as a toy truck, a Boy Scout poster, or a pair of wire cutters. When each object is activated, the walls of the domestic space fall away and a surreal desert landscape is revealed in 2D surrounding panorama, and one of the four voice-over actors is heard recounting particular acts and memory related metaphorically to the object selected. The objects also function much like hyperlinks that move us from one narrative element to another. Viewers travel through the domestic spaces and surreal interior landscapes of soldiers who have come home transformed by these experiences, triggering their testimonies by interacting with objects laden with loss. When each trigger object is selected within the 3D visual space, the domestic space falls away and a surreal desert landscape is revealed. This transition serves as a metaphor for the interior state of the individual solider, as it is coherent with accounts of soldiers experiencing Post-Traumatic Stress. It is also intended to bring audience members into a “listening state” where they can focus on the individual voices and the issues they raise. Objects in a living room space and a suburban backyard move us further into the field of battle, and there we encounter harrowing stories of interrogation, torture, and moral conflicts confronted differently by each of the characters.
The original version was ported in Unity from the original CAVE performance version to a stand-alone version for exhibition and performance contexts outside of the CAVE. It has subsequently been shown in standard cinema environments, and other VR and 3D theater environments, such as the 360 Gallery at City University of Hong Kong and the Data Dome in Bristol. The work has also reached audiences in diverse arenas far afield from typical venues for electronic literature, ranging from SIGGRAPH Asia and the IEEE Visualization conference to the Human Rights, Human Wrongs festival in Oslo and the Nobel Peace Prize Forum in Minneapolis.
An important component of the performances of Hearts and Minds is that the collective experience of the artwork is followed by audience discussion. The ultimate purpose of this work is to promote dialogue and debate about the contexts and circumstances of the use of battlefield torture in recent history. In this sense the project shares an aim with Augosto Boal’s Theatre of the Oppressed, in that attempts to offer audiences “the aesthetic means to analyze their past, in the context of their present, and subsequently to invent their future, without waiting for it.” During a 2015 presentation of the project at the Oslo Human Rights / Human Wrongs Film Festival, discussion participants included both Asbjørn Rachlew, the Superintendent of Norwegian Police who interrogated the 22nd of July terrorist Anders Behring Breivik, and a number of prisoners of conscience in the audience who had themselves been subject to abusive violence and felt provoked by the work to share their own experiences as victims of torture. The dialogs and debates that follow the interactive screenings of Hearts and Minds are difficult, but they are essentially the main point of doing the work to begin with.
Introduction and Panel debate, "Interrogating Torture", following the screening of "Hearts and Minds: The Interrogations Project at the Human Rights / Human Wrongs Festival, Litteraturhuset Oslo, Feb. 13, 2015.
Conclusion
As many of the other essays in the [Frame]Works gathering will argue, we assert here that the digital humanities should not be only understood as the application of digital tools to the archival materials of the human past, but as a humanities that engages directly with the challenges of the present. This view is one that emphasizes the digital arts and humanities and recognizes that electronic literature and digital art can be forms of critical making that engage directly with the contingencies of a world in crisis. The multimedia, variable, computational affordances of digital media enable us to represent complex situations in affective ways that are different from how they would be presented in a printed book or standard video format. There is much talk of interdisciplinarity in contemporary digital humanities discourse: critical digital media projects illustrate ways in which that interdisciplinarity can be more than surface deep. These projects are not simply about the application of computational methods to humanities materials, but the application of digital poetics to the negotiation of the situation of the human within a troubled present reality. We argue for a digital humanities that grapples with the history of a future that is yet to be written.
References
Boal, Augusto. 1985. Theatre of the Oppressed. Theatre Communications Group.
Cabell, Mimi, and Jason Huff. 2012. American Psycho. Vienna: Traumawien.
Carpenter, J.R. 2016. The Gathering Cloud. 2016. http://luckysoap.com/thegatheringcloud/.
Coover, Roderick and Scott Rettberg. Cats and Rats / Rotter _og katter. 2011. https://vimeo.com/50355051.
Coover, Roderick et al. 2015. “Interrogating Torture” panel discussion at the Human Rights / Human Wrongs Festival in Oslo, Norway. https://vimeo.com/119752276
Coover, Roderick and Scott Rettberg. Norwegian Tsunami / _Norsk flodbølge. 2012.
Coover, Roderick. The Chemical Map. 2013. https://www.google.com/maps/d/u/1/viewer?hl=en&ll=38.81532351880875%2C-75.72728133212138&z=8&mid=1gDv_AWrkjdNesNiNcucE1ujcr5QJMT6X
Coover, Roderick and Scott Rettberg. The Last Volcano / Det siste utbruddet. 2010. https://vimeo.com/57377350 (English subtitles).
Coover, Roderick, Nick Montfort, and Scott Rettberg. Three Rails Live. 2012. Sample online: https://vimeo.com/73507741.
Coover, Roderick, Arthur Nishimoto, Scott Rettberg and Daria Tsoupikova. Hearts and Minds: The Interrogations Project. 2014-16. Application for performance in the CAVE2™ immersive virtual reality environment in the EVL at University of Illinois Chicago, and as a standalone Unity application for other platforms. Short documentation video excerpt: https://vimeo.com/107178964.
Coover, Roderick, and Scott Rettberg and Daria Tsoupikova. Toxi•City: A Climate Change Narrative. 2016. Documentation: http://www.crchange.net/toxicity/.
Daniel, Sharon. 2015. Inside the Distance. http://www.sharondaniel.net/inside-the-distance.
Daniel, Sharon. n.d. “Public Secrets.” The Electronic Literature Collection, Volume 2. http://collection.eliterature.org/2/works/danielpublicsecrets.html.
John Cayley. The Listeners. 2016-2019. https://programmatology.shadoof.net/?thelisteners
Hanafi, Amira. 2018. A Dictionary of the Revolution. 2018. http://qamosalthawra.com/en.
Howe, Daniel C. 2014. AdLiPo. 2014. https://rednoise.org/adlipo/.
Howe, Daniel C., Helen Nissenbaum, and Mushon Zer-Aviv. 2015. AdNauseum. 2015. https://adnauseam.io/.
Howe, Daniel C., and Helen Nissenbaum. n.d. TrackMeNot. https://rednoise.org/daniel/detail.html#trackmenot.
David Jhave, Johnston. 2011. Extinction Elegies: A Post-Fukushima Interactive Video-Poem Tht Introduces Mutations Into the DNA of Meaning. 2011. http://glia.ca/2011/extinctionElegy/.
Koenitz, Hartmut. “Documentation of Interactive Digital Narrative Lecture by Hartmut Koenitz.” https://www.uib.no/en/dnn/129600/documentation-interactive-digital-narrative-lecture-hartmut-koenitz
Marino, Mark C., and Rob Wittig. 2015. #1WkNoTech. 2015. http://1wknotech.org/.
Marino, Mark, and Rob Wittig. 2015. I Work for the Web. 2015. http://robwit.net/iwfw/.
Moll, Joana. 2014. CO2GLE. http://www.janavirgin.com/CO2/CO2GLE_about.html
Moll, Joana. 2019. The Hidden Life of an Amazon User. https://www.janavirgin.com/AMZ/
Morton, Timothy. 2013. Hyperobjects: Philosophy and Ecology After the End of the World. Minneapolis: University of Minnesota Press.
Ratto, Matt and Megan Boler, eds. DIY Citizenship: Critical Making and Social Media. Cambridge, MIT Press, 2014.