What Is Chain-of-Thought (CoT)? Why Making AI Show Its Work Makes It Smarter

Ask an AI a slightly tricky arithmetic problem or a logic puzzle, and it may breezily hand back a wrong answer with a straight face. Yet on the very same problem, simply adding “show your reasoning too” can sharply raise the odds of a correct answer. This approach of making the model write out not just the answer but the path of thinking that leads to it is called chain-of-thought (CoT).

CoT is the idea underpinning today’s reasoning models and prompt engineering. This article explains, for non-specialists and drawing on Google’s research, what chain-of-thought is and why simply making an AI show its work makes it smarter.

Why It’s Needed: Trying to Answer in One Leap Leads to Mistakes

An LLM (large language model) is, at heart, just predicting the words that follow the input text. That made it weak at problems requiring multiple steps. Ultralytics, which publishes an AI glossary, explains that standard models stumble on multi-step problems “because they attempt to map the input directly to the output in a single pass” ³. In human terms, it’s like blurting out an answer through pure mental arithmetic, without scratch paper or notes. Easy problems land, but as the number of steps grows, the reasoning breaks down partway.

The fix that emerged was to deliberately make the AI output its “intermediate thinking” as text. Ultralytics defines chain-of-thought as a prompting technique that “enables large language models (LLMs) to solve complex reasoning tasks by breaking them down into intermediate logical steps” ³. Instead of rushing the answer, you have the model write out the thinking process itself, as if on paper.

How It Works: Make It Write the “Intermediate Thinking” Before the Answer

The first work to lay out chain-of-thought systematically was a 2022 paper by Google researchers, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” ¹. The paper reports that “generating a chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning” ¹.

The method itself is simple. When you have the AI solve a problem, the examples you show it as models include not just the final answer but the intermediate steps: “think this way, calculate that, and so the answer is this.” The AI then imitates that format, producing its intermediate thinking in order before arriving at an answer on the actual problem.

A key advantage is that you don’t need to retrain a new AI. CoT works without changing the model’s architecture or redoing fine-tuning—you only change how the examples in the prompt are written ¹. Ultralytics notes that by explicitly generating intermediate reasoning this way, “the model has more opportunities to correct itself and provides transparency into how it arrived at a conclusion” ³. Because it advances by checking the path one step at a time, it is easier to catch mistakes than when answering in a single leap.

The Effect: Accuracy Jumps with a Hundredth of the Effort

Chain-of-thought drew attention because its effect was shown clearly in numbers. In Google’s paper, prompting a 540-billion-parameter (540B) language model with just eight examples containing chains of thought achieved the state-of-the-art accuracy of the time on “GSM8K,” a benchmark of grade-school-level math word problems—surpassing even a GPT-3 that had been fine-tuned with a verifier ¹. Instead of additional training on massive data, simply showing the way of thinking through eight examples beat a model trained specifically for the task. This effect was confirmed not only in arithmetic but across multiple domains, including commonsense reasoning and symbolic reasoning ¹.

Going further was a 2022 paper, “Large Language Models are Zero-Shot Reasoners,” whose authors include University of Tokyo professor Yutaka Matsuo ². This study showed that even without showing a single example, simply adding the sentence “Let’s think step by step” to the end of the prompt raises the AI’s reasoning performance ²³. Whereas showing examples is called “few-shot CoT,” this works with zero examples, so it is called “zero-shot CoT.”

Its effect was large too: the paper reports that on OpenAI’s model (text-davinci-002), accuracy on MultiArith, a set of multi-step arithmetic word problems, jumped from 17.7% to 78.7%, and the aforementioned GSM8K from 10.4% to 40.7% ². A single, almost spell-like instruction drew out reasoning ability that had been dormant inside the AI. The result suggests that the power to reason was latent in the model all along, and the key was “how to elicit it.”

Relationship to Reasoning Models and Prompts

Chain-of-thought has taken root in today’s AI in two forms.

One is prompt engineering, how we use AI day to day. When asking a hard question, simply adding “show your reasoning too” or “explain step by step” tends to raise the quality of the answer. This is no special technique—it’s an application of chain-of-thought that anyone can try right away.

The other is the evolution into reasoning models. Reasoning models such as OpenAI’s o1 and o3 are built to unfold a long internal thinking process before producing a conclusion, without the user having to ask each time. Even though the form visible to the user differs, the root idea—“generate a path of thinking before answering”—is continuous with chain-of-thought.

Limits: Not a Universal Magic

Chain-of-thought is powerful but not all-purpose. Even if the intermediate thinking is written plausibly, if the reasoning itself is wrong, the AI still arrives at a wrong conclusion. Hallucination, where an AI states things at odds with the facts while laying out a coherent-sounding argument, cannot be fully prevented just by making it show its work. Also, chain-of-thought shows clear benefits mainly with sufficiently large models, and for simple questions whose answer is obvious at a glance, there is little point in having the model write out its steps.

Even so, the idea of chain-of-thought—“don’t rush the answer; put the thinking process into words”—is one of the most basic and reproducible ways to draw out an AI’s reasoning ability. If you want to understand further why AI became able to “think” before answering, reading this alongside reasoning models, which unfold thinking internally before answering, and prompt engineering, which changes answer quality through how you give instructions, brings the workings of conversational AI’s intelligence into clearer relief.