Category: NaNoGenMo

I spent about 25 hours in November producing a novel via an LLM for NaNoGenMo 2024. It was an interesting experiment, although the book produced was not particularly engaging. There’s a flatness to LLM-generated prose which I didn’t overcome, despite the potential of the oral history format. I do think that generated novels can be compelling, even moving, so I will have another try next year.

Some things I learned from this:

• I hadn’t realised how long and detailed prompts can be. My initial ones did not make full use of the context. Using gpt-4o-mini was cheap enough that I could essentially pass it prompts containing much of the work produced so far.
• For drafting prompts, the ChatGPT web interface was more effective, because it maintains the full conversation as a state. Once I used this for experimenting with prompts, things moved much faster.
• Evaluating the output is incredibly hard here. In a matter of minutes I can create a text that takes hours to read. Most of my reviews were done by random sampling, and I didn’t have time to properly examine the text’s wider structure.
• It was also tricky to get consistent layouts from the LLM. Using JSON formats helped somewhat here, but at the cost of reducing the size of LLM responses.

22 books were completed this year and I’m looking forward to reviewing them. I have an idea for a different approach next year and will do some research in the meantime (starting with Lillian-Yvonne Bertram and Nick Monfort’s Output Anthology)

I’m now halfway through NaNoGenMo 2024. I’ve been working on my project every day this month and wanted to share some initial thoughts.

• Having a software project to tinker with is fun, particularly with NaNoGenMo’s time limit to keep me focussed.
• My tinkering has been distracted by working on refactorings rather than the GenAI-specific code. Adding design patterns into the codebase has been a useful opportunity to think about refactoring, and something I should be playing with coding projects more often.
• Working with the LLM fills me with awe. These things can produce coherent text far faster than I can read them.
• The output is readable without much work. I asked ChatGPT4 to produce a Fitzgerald pastiche (Gatsby vs Kong – about kaiju threatening a golden age) and it’s an interesting text to scan through.
• The question of testing is particularly tricky here. I’m producing novels which would take about 3-4 hours to read. I’ve been randomly sampling passages, picking out style issues, but structural ones/weird repetitions on a larger scale will be harder to fix.
• My overall plan is to produce a novel made of oral histories. Getting these to sound varied in tone is a challenge, and one I will dig into over the last two weeks. My pre-NaNoGenMo experiments suggested that LLMs were good at first person accounts – but getting an enjoyable novel out of them is difficult.
• I’m relying on the structured JSON outputs from ChatGPT to get consistent formatting from ChatGPT, as it gives me a little more control.

Technically, I’ve completed NaNoGenMo as my project has used a fairly basic technique to generate 50,000 words of Godzilla vs Kong. But, ultimately, the question is whether ChatGPT can produce an enjoyable novel. I thought previous entrant All the Minutes was a genuinely exciting piece of literature. That is the bar I want to aim at.

One of the most interesting apects of computer-generated novels is that you can produce text faster than anyone could read it. Producing compelling, readable text is another matter.

There was a lot of hype in the early days about how GenAI would be able to compete with human writers. This has not turned out to the be the case – most sophisticated LLMs are designed for general use and getting them to produce crisp literary text is hard. They have learned bad habits from reading everyday prose and beginner’s creative writing (they have also picked up some strange ideas).

In the afterword to Death of an Author, Aiden Marchine¹ wrote about his workflow, which required combining ChatGPT with other tools and his own intensive edits. The book reads well, but Marchine estimates only 95% of the text is compuer-generated. He also describes doing a lot of work to help the AI.

ChatGPT is helping people with writing on a smaller level. Some writers use GenAI to produce descriptions, as described in Verge article The Great Fiction of AI. There’s also some interesting recent discussion by Cal Newport about how people have used LLMs in academic workflows (see What Kind of Writer is ChatGPT).

We’re a long way from giving chatGPT a paragraph of description and getting a readable novel out.

Something that Marchine pointed out is that LLMs are very good mimics for some types of writing. Marchine went on to point out that Dracula is a novel made up of different types of document, and maybe an LLM can produce a novel made of found texts. Stephen Marche’s New Yorker article, Was Linguistic A.I. Created by Accident? describes how one of the first signs of LLMs’ power was the production of some fake wikipedia entries. Five entries were created for ‘the Transformer’, and the results included an imaginary SF novel and a hardcore Japanese punk band.

A narrative novel is beyond current LLMs. But that still leaves options for other types of fiction.

1. Aiden Marchine was a penname taken by Stephen Marche for the work he produced in collabortation with AI tools. ↩︎