In an earlier post, we considered some general issues around artificial intelligence, writing, and editing. This topic is both technically and existentially interesting, so today we’ll focus in more on one well-known example of language AI: GPT-3. We’ll also revisit some anxieties that it’s raising for writers and why we think these anxieties are overblown.
What is GPT-3 and how does it Work?
GPT-3 stands for “Generative Pre-trained Transformer, 3rd generation”. It is a large neural network (at the time of its release, the largest neural network ever), with 175 billion (!) parameters. (For reference, GPT-1 and GPT-2 had 117 million and 1.5 billion parameters, respectively).
A neural network is a computer architecture that works well for detecting hidden patterns in a large amount of input data. The network uses processes that mimic our brain’s own processing to gradually get better and better at pattern-detection during its training process. Importantly, for GPT-3 and modern neural networks, the programmer does not have to pre-specify the patterns in the data. As long as the sample is reasonably representative of the world, the neural network learns patterns automatically—and often impressively. Some neural networks, like GPT-3, can also generate common patterns they have learned.
GPT-3’s task is to generate human-like language outputs from text prompts. It was trained on about 450 GB of text data gathered from the internet. Because of its huge number of parameters, GPT-3 is capable of picking up on patterns in the text from the very fine-grained to the very general. Media reports emphasize the astonishing human-likeness and fluidity of the language it produces based on prompts (for a deep dive, see here). Furthermore, it also learns to learn based on the prompts, so it can tweak its output to the user’s commands.
Is it Time to Worry?
GPT-3’s achievements are a big deal. Are human writers doomed? Are the machines coming for our jobs sooner than we expected? We think not, and for very specific technical reasons.
We need to unpack the idea of “generating human-like language”. To do this, GPT-3 works in a step-by-step fashion—it picks the first word after the prompt, then the second, then the third, and so on. This allows it to generate strings of text very quickly, leveraging its pattern-detecting superpowers at every step of the process. However, this technical feature limits GPT-3’s usefulness for building coherent long texts—those that develop a theme, take asides but respect a basic simplicity of the narrative, and stay internally consistent. (To be fair, humans have trouble with this too, and GPT-3 does deal with context reasonably well, but the ways it deals with context is not very human-like.)
GPT-3 doesn’t Understand
There’s also a deeper limitation to GPT-3. Put simply, although GPT-3 outputs remarkably human-like language, it doesn’t really understand how the world works, what its words mean, or the general cohesion of its writing. This is not a limitation of computing power. The problem is that GPT-3 has an impoverished relationship to the world; it merely represents the world, and doesn’t engage it.
We humans, whether writers or editors, don’t have that problem. We are always already engaged with the world, and our language is responsive to that world. Human writers have other problems, like our relatively slow output, limited energy, and fleeting motivation. But because we are engaged, we usually have some grasp on a topic. Our language use is exquisitely context-sensitive in a way that GPT-3 cannot yet match. (This article discusses some of GPT-3’s failures of context-sensitivity.) This makes us resilient, to an extent, to automation-induced unemployment, at least for the foreseeable future.
Of course, there are many, many nuances here. We will pick up on them in later posts. Stay tuned!