So what happens next?
Improving Story Continuation in Creative Writing Tools
Maya Epps
Under the Mentorship of Dr. Mandy Korpusik
https://openai.com/blog/better-language-models/
Computer Science
2
The Narrative / Story Continuation task aims to “dynamically predict ‘what happens next’ in a story.”[4]
Prior work has studied the use of Natural Language Generation (NLG) and machine learning to create a creative writing support tool for authors that continues a story.
https://openai.com/blog/better-language-models/
The orcs’ response was a deafening onslaught of claws, claws, and claws; even Elrond was forced to retreat. “You are in good hands, dwarf,” said Gimli, who had been among the first to charge at the orcs; it took only two words before their opponents were reduced to a blood-soaked quagmire, and the dwarf took his first kill of the night. The battle lasted for hours until two of the largest Orcs attempted to overwhelm Aragorn. When they finally stopped, they lay defeated and lifeless for miles and miles.
“I take nothing,” said Aragorn. “But I give my word, at my peril and mine, that I will never forget this day of horror. None of us will forget. Ever!”
“I’ll never forget it!” cried Gimli, who had been in the thick of the battle but hadn’t taken part in it. One of the wounded orcs he had carried off, he was the only one of the survivors who remained uninjured. “We’ll keep the memory of that day of evil, and the war with it, alive as long as we live, my friends!”
“Then we’ll keep it alive as long as we live,” added Legolas. “And we won’t forget the first great battle of the night, even if we may have forgotten the final defeat.”
Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry.
3
Clark et. al. case study:
“participants wanted more coherent suggestions from the model.”
Should have a low level of intrusiveness[3]
Studies found users are interested in a creative writing support tool, but also suggest improvements are needed.
“Would you use this system again?”
Responses (out of 9):
Creative Help tool:
“the most common piece of feedback was that the suggestions were not coherent”
“avoid foregone conclusions” about types of suggestions users wanted[4]
Author Sigal Samuel on GPT-2 and “how I’m using AI to write my next novel”:
“Using AI as a replacement for all authorial work? I don’t see that happening anytime soon. But using it as a cognitive enhancement, a creative prosthesis? Sign me up.”[5]
4
GPT-2 is a transformer-based machine learning model that may be ideal for this task because it achieved “state-of-the-art” (highest score) on many NLP benchmarks in 2019.[1]
GPT-2 improves on the previous record and approaches human accuracy
https://openai.com/blog/better-language-models/
GPT-2:
1.5 billion parameters
40GB of Internet text
Does applying transformer-based models that are fine-tuned on specific book genres to a narrative continuation task improve coherence and creativity, compared to a RNN with GRU baseline for a creative writing support tool, as measured by user feedback?
5
* RNN (Recurrent Neural Network) with GRU (Gated Recurrent Units) is a neural network used in the aforementioned Creative Help tool
Research Question
Subjects will be asked to provide feedback on both GPT-2 and baseline models for a fair comparison.
Network models will be trained on different
genres of books from the Project Gutenberg website.
6
References
[1] A. Radford, J. Wu, D. Amodei, D. Amodei, J. Clark, M. Brundage, and I. Sutskever, “Better
Language Models and Their Implications,” OpenAI, 14-Feb-2019. [Online]. Available: https://openai.com/blog/better-language-models/. [Accessed: 04-Nov-2020].
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I.
Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NIPS), 2017, pp. 5998–6008.
[3] E. Clark, A. S. Ross, C. Tan, Y. Ji, and N. A. Smith, “Creative writing with a machine in the loop: Case studies on slogans and
stories,” in Proceedings of IUI, 2018.
[4] M. Roemmele, "NEURAL NETWORKS FOR NARRATIVE CONTINUATION," Ph.D dissertation, University of Southern
California, Los Angeles, 2018. Accessed on: November 4, 2020. [Online]. Available: https://roemmele.github.io/publications/dissertation.pdf
[5] S. Samuel, “How I'm using AI to write my next novel,” Vox, 30-Aug-2019. [Online]. Available:
https://www.vox.com/future-perfect/2019/8/30/20840194/ai-art-fiction-writing-language-gpt-2. [Accessed: 11-Oct-2020].
7
Thank you!