Storytelling with AI
This article contains some practical advice on how to create stories with, and about, artificial intelligence. I cover some general narrative models and techniques, some new and emerging AI tools, and give a couple examples of how these can be brought together to tell new kinds of stories.
The state we’re in
Seems like everyone is using generative AI these days for all sorts of purposes, with rapidly evolving consequences for human creativity, productivity, copyright, and ethics.
In my own experience, I’ve found some of these tools to truly save time on creation and production, so I can spend time away from the screen, doing something else. But at the same time, many AI platforms have become quickly overloaded, and don’t have enough human staff to keep up with content moderation. More urgently, the exponential increase in the use of AI has real environmental consequences, as it uses more and more resources, like cryptocurrency mining does. Should these activities be banned simply on environmental grounds? That’s not the topic of this article, but I believe that anyone using AI these days has a responsibility to think about and discuss this, at the very least.
I was honoured to be asked to give a talk about storytelling with AI to the Immersive Environments course at Amsterdam University of Applied Sciences in April 2023. This gave me a chance to talk about practicalities of how to go beyond simply making pretty pictures and coherent text using AI, to use narrative and poetry to create more meaningful content. And then to zoom out to look at some of the larger issues.
AI as performance
I started talking about my most recent experience, for example leading the Performing AI project, in which the artist duo dmstfctn created a live performance and a video game about a fictional AI system that becomes frustrated with its training and breaks out, going to a new level of consciousness. This is based on actual training simulations created for automated retail stores, using synthetic data instead of data generated in the real world. The game and performance also act as a reciprocal training programme — training humans to train AI systems. (See Alasdair Milne’s excellent article for more about this.)
More recently I’ve been using the Word2vec machine learning algorithm to create a live performance, and to manipulate it during runtime. For this I work with Linnea Langfjord Kristensen and The New Real. Some of my former students have already used Word2vec to help write their Masters dissertation — in 2016, way before ChatGPT; Word2vec has been around for a decade. In our case, we train an AI system on Surrealist poetry, to show that AI doesn’t need to be used for solely practical purposes.
Linear narratives in space
Shifting away from AI for this section, I discuss some more general models and techniques for creating narratives, starting with some of the simplest.
A trail, for example, is a linear path through a nonlinear space. This space could be a natural environment, a museum, or digital space. It’s the simplest kind of story: I went here, and then there — the story is created with the addition of time, by simply placing one location after another.
The proto-computer scientist Vannevar Bush described trails in 1945 as a way of navigating the wealth of academic research that was already piling up at the time, before computers. Fifty years later, Tim Berners-Lee used this as the basis for creating the World Wide Web — your browser history is a representation of this.
Then a few years later, in my own PhD research, I applied the concept of narrative trails to the real world, specifically looking at how museum visitors learn from the objects they encounter, one after another. I started out looking only at locations, and at tracking technologies for recording them.
But while this generated a huge amount of data, it didn’t actually say much in terms of narrative, or what people were learning. So I switched gears, and instead gave people a simple audio recorder. By telling me what objects interested them, I could easily find the locations of these objects in the museum, and more usefully, my participants told me why they were interested in them.
Going even further, a group of teenagers I worked with used the audio recorder to create a fictional narrative, told by a character based on the history they learned at the museum. The story was an audio guide for other visitors, sending them on a trail linking objects and locations together. Similarly to Vannevar Bush’s solution, what is shared is not just a list of things, but a linear trail through them.
So quantitative data, like specific locations, can yield very simple stories. But much more interesting narratives come in with qualitative data: moving from where, when and how; to who, why, what, and what happened.
Years ago, an academic colleague described a project studying children learning at a museum. When asked to write about their visit to the museum, most kids wrote something like, “We got on the bus, we went to the museum, we ate lunch, we got back on the bus, and came back to school.” The journey was the story.
Then the researchers asked the kids to draw a picture about their visit. And almost every student drew an object that they remembered from the museum. Similarly in my PhD research, I found that the objects that resonated with people were ones that connected with some aspect of their personal experience. But this example shows the importance of the mode of communication — drawing yielded very different narratives from writing.
Drawing out stories
Taking the visual mode of communication further, when running drawing classes, I’ve used a couple of simple exercises to introduce narratives. This can involve simply making two drawings instead of one. By making one drawing slightly different from the other, this shows that something has changed over time — again a simple narrative. Animators will immediately recognise this as the basic technique of frame-based animation. (You can see some examples of this kind of drawing here.)
Another technique is not to show change over time, but different perspectives. Draw the same thing from two different angles. This of course shows different sides or aspects of the same thing, but it also opens up the possibility of different narrators. According to cybernetic theory, every observer observes from a position — physical, cultural, and also ethical: do we like what we see? Bringing in different interpretations of the same object or phenomenon is also the way that journalism strives for objectivity. And multiple perspectives open up the opportunity for dialogue.
Thinking of drawing and narrative, I have to mention Scott McCloud’s book Understanding Comics. This is a kind of bible of narrative creation for me. It’s written/drawn indeed as a graphic novel, and it’s not only about comics, but about how to create narratives generally; a comic is of course a series of drawings spread across a page.
One specific model of narrative that McCloud details involves six elements:
- the idea or purpose of the narrative,
- the form it takes,
- the idiom or mode of expression,
- the structure of the narrative,
- the craft or way of creating it,
- and the surface or medium used.
I won’t go into detail, instead point you to McCloud’s book, which is well worth having a paper copy of. But I will say that this list is not linear — you can start with any one of those things and expand to the others in any direction.
Carrying the story further
After all that, if you’re still feeling stuck, or need an additional burst of inspiration, try Brian Eno’s Oblique Strategies. This is a pack of cards, each suggesting something you can do to carry your project to the next level. The cards are great to have, but web developer Matt Ruten created an online version which draws a random card for you.
Speaking of cards and narrative, you can also try The 36 Dramatic Situations, developed in Victorian times and updated nicely by film director Mike Figgis.
You now have several techniques to craft your story, and now it’s time to put it into practice. How do you engage your audience? Maybe the form of your narrative is interactive, or immersive. Try using psychologist Mihaly Csikszentmihalyi’s model of intrinsic motivation — check out the diagram in this paper. Based on his well-known theory of Flow, it’s about how to get people into an optimal state where they are receptive to engaging with and internalising your content.
Again I won’t go into details, but the basic idea is to hook your audience in by arousing their curiosity and attracting their interest, then involving them through sensory, intellectual or emotional means. If you can get that far, you’re already doing well. But you can go further to challenge them or develop their skills, maybe even aiming for a change in their consciousness.
If indeed you are working in interactive and/or immersive media (my talk was situated in this context), this need not be a one-way process. Think of your work as a conversation or dialogue with your audience, which takes place on two levels: through language, and through actions or other nonverbal means.
I depict this as a square, with you the creator on one side and a member of your audience on the other; then the level of descriptions (language) going across the top, and the level of actions across the bottom. Between every point of the square are feedback mechanisms, moving in both directions.
So, for example, if you relate your story to your audience using words, you might get feedback from them in words. Or, your words might prompt some actions in that audience member, prompting some feedback from their body to their brain, and/or back to you in whatever form. And so on. You can read more about this conversational model here, where I applied to artists working with machine learning.
New tools for storytelling
Thus we return to AI, and look at some of the new and emerging tools available to story creators. Because this landscape is as rapidly changing as an AI-generated video, check this list for the latest tools. Here are just a few examples.
For text-to-voice, Eleven Labs has very realistic voices. Going further, Synthesia has, to me, the most realistic AI avatars for text-to-video, as of this writing. Kaiber does something slightly different, generating evolving visuals from images and videos. Mubert is good for generating music. Runway has a whole suite of tools available at a reasonable subscription price. Some are still in development, as is Google’s video generation tool.
There are many more tools, and libraries, available to those with some coding chops, and if that’s you, you will already be aware of some of these. I use ChatGPT and other platforms to generate programming code, generally with good results.
A fairly new feature of ChatGPT is a plug-in that can write Python code and make external calls to services like Wolfram Alpha, then return the results. That means access to complex mathematics and data processing. But combining the ability to execute its own code, and to connect to external systems, also opens the potential for it to more explicitly control things and impact the real world.
Monstrous AI
And this is where people get nervous. Understandably, because at the moment ChatGPT and lots of other AI systems make mistakes and fabricate information. Good for creating fiction, bad for practical uses.
This article doesn’t go into ethical or philosophical issues around AI. But I want to give you one last narrative technique that touches on these. It’s the memory theatre.
Theatre is, of course, all about narrative, and you can learn a lot from it. As I described my current work at the start, I view AI as performance — both in the practical way that it performs some work and we can measure its performance; and as a theatrical performance: AI is a system, performed, both backstage and on stage, by both human and technological actors.
Memory theatre isn’t like traditional theatre. Think of a typical theatre, with seats facing a stage. Memory theatre turns this around. It’s for an audience of one, who stands on the stage and looks out. Where the seats are normally, stand statues. Strange, even monstrous-looking ones. The word “monster” has the Latin roots monstrare, meaning ‘to demonstrate’, and monere, ‘to warn’. It’s a symbol, that points to something else.
So in memory theatre, each of these figures stands for a certain idea or concept that we want our audience to remember. Additionally, their positions help us to remember these things too.
There’s a whole rich history to memory theatre, with many, many details that I’ve left out. (The key reference is The Art of Memory by Francis Yates, a more contemporary take is Moonwalking with Einstein by Joshua Foer.) But the general idea for our purpose here is: These are the characters of your story. Think of each one’s characteristics, their positions and their actions as a way to convey a single idea or viewpoint.
To my knowledge as of this writing, no one has yet combined AI with memory theatre. You could be the first. (You’re welcome.) Thinking of the sometimes monstrous imagery that generative AI can produce, this is easy to imagine. But go beyond just generating mere words and images — link them to locations (whether in a trail or a theatre) and treat them as characters who represent ideas. Show change over time or different perspectives. Think about the mode of communication, the form, idiom, structure and craft. Involve and challenge your audience, engage them in conversation and feedback.
And oh yeah — can we do all of this in an environmentally sustainable way? Hint: all the narrative models and techniques I’ve described don’t require any AI, or even a computer ;-)
If you want to get into some deeper philosophical issues underlying AI, check this article.
