What Is a Transformer? The 'Attention' Mechanism That Changed AI

The “GPT” in ChatGPT stands for Generative Pre-trained Transformer. Google’s “BERT” stands for Bidirectional Encoder Representations from Transformers². The names of modern generative AI often hide a word in common—and that word is Transformer.

The Transformer is the “blueprint” technology underpinning today’s AI boom. This article explains what a Transformer is, what made it so groundbreaking, and why it led to the rapid evolution of LLMs (large language models)—based on the original 2017 paper, written for non-experts.

What Is a Transformer: A New Design Born in 2017

The Transformer is a new neural network architecture (structure) proposed in the 2017 paper “Attention Is All You Need,” published by researchers at Google².

The paper’s claim is captured by its title. It states: “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”¹. Discarding the complex mechanisms that had been mainstream and narrowing down to a single idea called “attention”—that was the Transformer’s starting point.

So what is this “attention”? And what was the “conventional mechanism” it discarded? Let’s look at each in turn.

Innovation 1: Looking at the Whole Sentence to Decide Where to Focus

Attention (the attention mechanism) is a mechanism that calculates “which words relate to which words” in a sentence and directs attention to the important information. A Japanese explainer describes it as “calculating the relevance between each element of sequence data (such as text or audio) and the other elements, and directing ‘attention’ to important information to achieve contextual understanding”².

Especially important is self-attention. This is a mechanism for capturing how words relate to one another within a single sentence: “by capturing the relationships between words, self-attention obtains representations of words that take the whole sentence’s context into account”².

For example, in the sentence “the bank by the river’s edge,” whether “edge” connects to “river” or to “bank” can’t be determined without surveying the entire sentence. Self-attention calculates exactly this: “looking at the whole sentence and deciding which other words each word should attend to.” This enabled AI to correctly understand words within their surrounding context.

Innovation 2: Moving from “Sequential Processing” to Parallel Processing

The other innovation lay in how processing is done.

The RNN (recurrent neural network), which had been mainstream before the Transformer, had a weakness. According to the Japanese explainer, “the RNN, a conventional sequence-transduction model, had to process sequence data sequentially”². In other words, it could only read a sentence one word at a time, in order, and couldn’t move to the next until the previous step finished. This took longer for longer sentences and made learning from large amounts of data difficult.

The Transformer removed this constraint. Because “the Transformer can process sequence data in parallel”², it can handle the words in a sentence together at the same time. The original paper, too, reports that the Transformer is “superior in quality while being more parallelizable and requiring significantly less time to train” compared to conventional models¹.

This property of “learning fast through parallel processing” was the decisive factor that made the later scaling-up of LLMs possible. Being able to learn from vast amounts of text in a realistic timeframe opened the road to “Large” models.

The Results: From Proof in Machine Translation to the Era of LLMs

The Transformer’s capability was demonstrated in machine translation from the outset. According to the original paper, the Transformer “achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU”¹. For English-to-French, it reports, the model “establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature”¹. BLEU is a metric for measuring translation quality—the higher the number, the better the translation.

What’s worth noting here is the training efficiency of “3.5 days on eight GPUs.” Reaching the state of the art of the time with modest computing resources led to the idea, “maybe making it bigger will make it smarter.” Indeed, both GPT and BERT, which appeared afterward, are built on the Transformer, as their names indicate².

In other words, the Transformer is not a specific product but the shared “foundation” on which modern generative AI stands. The reason an LLM can predict the next word token by token, and respond with context within its context window, is that this attention mechanism captures the relationships within a sentence. The reasons for AI’s intelligence—and for its limits—both trace back to this 2017 design. In understanding generative AI, “Transformer” is a word worth knowing.