OpenAI’s o3, Claude’s extended thinking, Google’s Gemini Deep Think—recent AI models increasingly include a type that “thinks carefully before answering.” These are called reasoning models, and they outperform conventional models on problems requiring math, coding, and complex logic.
Rather than answering instantly, they first work through a problem step by step in their head before responding—a behavior that comes naturally to humans. How did AI acquire it? This article explains what a reasoning model is, the idea of “Chain-of-Thought” that gave rise to it, and the mechanism of “spending compute on thinking,” using papers and official sources and aimed at non-specialists. Reading it alongside our guide to large language models (LLMs) will help you see the direction of recent AI progress.
The Starting Point: Chain-of-Thought
The starting point for understanding reasoning models is a 2022 paper by Google researchers titled “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” It showed that “generating a chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning” 1.
The key is that instead of having the model produce only the answer, you have it write out the intermediate thoughts that lead to the answer, in order 4. For an arithmetic word problem, for example, rather than blurting out a number, you make it go step by step: “First there are 3 apples, then 2 more are given, so 5 in total, and 1 was eaten, so…”
The effect shows up in the numbers too. According to the paper, prompting a 540-billion-parameter model with just eight chain-of-thought exemplars achieved state-of-the-art accuracy on the GSM8K benchmark of math word problems, even surpassing a fine-tuned GPT-3 with a verifier 1. Performance gains were confirmed not only in arithmetic but across multiple tasks, including commonsense and symbolic reasoning 1.
Why does simply writing out the steps make a model smarter? Explanations cite reasons such as: (1) externalizing working memory by writing intermediate results into the output text, (2) breaking a complex problem into small, simple steps, and (3) making errors easier to spot because the intermediate process is visible 4. This technique is also one of the methods covered in our guide to prompt engineering, and it’s known that you can trigger it simply by adding “Let’s think step by step” 4.
What Is a Reasoning Model? The Model Does the Thinking Itself
Chain-of-Thought was initially a technique where the user prompted the model to “think before you answer.” A reasoning model is one designed to do this on its own, by default.
Microsoft’s documentation positions reasoning models as ones “designed to take on reasoning and problem-solving tasks,” noting that with these models “more time is spent processing and understanding the user’s request,” making them very powerful in fields such as science, coding, and math compared to earlier models 3. Instead of answering instantly, they take time to think things through internally.
Anthropic’s official documentation likewise describes Claude’s extended thinking as giving “Claude enhanced reasoning capabilities for complex tasks, while providing varying levels of transparency into its step-by-step thought process before it delivers its final answer” 2. When extended thinking is enabled, Claude generates “thinking” blocks that write out its internal reasoning before producing the final response 2. OpenAI’s o3 and Google’s Gemini Deep Think, though named differently, are built on the same idea of “thinking before answering.”
How It Works: Allocating Tokens (Compute) to Thinking
What characterizes reasoning models is that they spend cost on the act of “thinking” itself.
The unit of processing for AI is the token. In reasoning models, tokens are used for internal thinking separately from the final answer. Microsoft’s documentation describes the “reasoning tokens” included in a reasoning model’s response as “hidden tokens that are not returned as part of the message response content but are used by the model to generate the final answer to the request” 3. In other words, tokens are also consumed by what amounts to a “draft” or “introspection” that never appears on screen.
How much it thinks can be adjusted. Microsoft’s documentation states that raising the reasoning effort means “the model spends more time processing the request, and generally the number of reasoning tokens increases” 3. With Claude’s extended thinking as well, you can set an upper limit on the tokens used for internal reasoning as a “budget,” and a larger budget is said to improve response quality on complex problems 2.
There’s a trade-off here. Letting the model think more makes a smart answer more likely, but it costs more time and more token cost. That’s exactly why you shouldn’t use a reasoning model for every question. Microsoft’s documentation says reasoning models suit uses such as complex code generation, brainstorming on multifaceted challenges, and analyzing subtle differences in contracts and legal documents 3. Conversely, for simple questions or routine processing, an instant-answer model is faster and more economical.
Knowing When to Use One
A reasoning model is not an “all-around model that excels in every situation” but rather “a model good at tackling hard problems carefully.” It shines on tasks like math, science, and coding that require multiple steps to reach an answer 3. On the other hand, because responses are slower and cost more, the practical approach is to use it selectively alongside instant-answer models depending on the task.
One caveat: having an internal thinking process does not completely eliminate hallucinations, where the AI gets facts wrong. A reasoning model is a technique for raising accuracy on complex problems by granting “time to think”; it does not guarantee the correctness of the underlying knowledge. When adopting AI in your work, it helps to keep in mind that there are two options—“a model that answers fast” and “a model that thinks carefully”—and that the latter buys accuracy by paying a cost in thinking. Holding onto this picture lets you choose wisely depending on the task.
Sources
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - Jason Wei, Xuezhi Wang, et al. (Google, 2022). The paper showing that chain-of-thought improves reasoning in LLMs
- Extended thinking - Anthropic official documentation (Claude’s extended thinking and thinking budget)
- Azure OpenAI reasoning models - Microsoft official documentation (features of reasoning models and reasoning tokens)
- What Is Chain-of-Thought (CoT)? The Basics of Eliciting Step-by-Step Reasoning from LLMs - Japanese-language explainer by Nexaflow