What Is Chain-of-Thought Prompting and Why LLMs Reason Step by Step

Chain-of-thought (CoT) prompting is a technique that gets a large language model (LLM) to write out intermediate reasoning steps before it gives a final answer. Instead of asking for the answer directly, you show the model worked examples or simply ask it to reason step by step. The idea was popularized by Google researchers, who found that prompting with a few step-by-step examples significantly improved performance on arithmetic, commonsense, and symbolic reasoning tasks, reaching state-of-the-art accuracy on the GSM8K math word problem benchmark at the time (Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”).

Why it matters now: step-by-step reasoning has moved from a prompting trick into the models themselves. So-called reasoning models generate internal reasoning before they answer, and vendors document how to use them differently from standard chat models. Understanding chain of thought helps you decide when to ask for it, when the model already does it for you, and what it costs.

How chain-of-thought prompting works

An LLM produces text one token at a time, and each new token is conditioned on everything written before it. When the model writes intermediate steps, those steps become extra context for the final answer. For multi-step problems such as math, logic, planning, or reading a messy table, breaking the problem down gives the model room to compute instead of jumping straight to a guess.

Few-shot chain of thought

You include a handful of example questions in the prompt, each followed by a worked solution and the final answer. The model imitates the pattern on the new question. This is the original form described by Wei et al.

Zero-shot chain of thought

You add a short instruction such as “Let’s think step by step” without any examples. Kojima et al. reported that this single phrase raised accuracy on the MultiArith benchmark from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7% with InstructGPT (Kojima et al., “Large Language Models are Zero-Shot Reasoners”).

Self-consistency

Instead of one reasoning path, you sample several and pick the answer that appears most often. This simple majority vote often improves accuracy further over a single chain (Wang et al., “Self-Consistency Improves Chain of Thought Reasoning”).

Chain of thought and reasoning models

Reasoning models are trained to produce a chain of thought on their own. OpenAI explains that its reasoning models use reasoning tokens to “think” before responding, and those tokens count toward usage even though they are not shown in full (OpenAI reasoning guide). Anthropic’s documentation, in turn, explains how to prompt Claude to think step by step and how to separate the thinking from the final answer with tags (Anthropic: chain-of-thought prompting).

The practical takeaway: with a reasoning model, you usually do not need to write “think step by step.” Focus on a clear goal, constraints, and the right context. With a standard model, explicit chain-of-thought instructions still help on multi-step tasks.

When to use it (and when not to)

• Use it for math and calculations, multi-step logic, comparing options against several criteria, debugging code, and extracting answers from complex documents.

• Skip it for simple lookups, classification with obvious labels, short rewrites, or anything where latency and cost matter more than a small accuracy gain.

• Watch the cost: reasoning text is made of tokens, so it increases both latency and price. Our explainer on what a token is and how it drives API costs shows how to estimate the impact.

Using chain of thought in real applications

In production apps, teams rarely show raw reasoning to end users. A common pattern is to ask the model to reason inside a dedicated section and then return the final answer in a structured format, such as JSON, that the application can parse. Another is to keep the reasoning for logs and debugging, which makes it easier to understand why an agent chose a particular tool or answer.

Keep in mind that a written chain of thought is not a guarantee of correctness. A model can produce plausible-looking steps and still reach a wrong conclusion, or give an explanation that does not reflect how it actually arrived at the answer. That is why chain of thought works best alongside grounding, tool use for exact calculations, and systematic testing; see how teams test AI systems in our guide to AI evals.

Chain of thought for data work

For analysts using LLMs on data, chain of thought is useful when a question needs several steps: identifying the right tables, deciding on filters, computing a metric, and explaining the result. Asking the model to lay out its plan before writing SQL makes mistakes easier to spot, such as a wrong join or a filter on the wrong date column. Still, the final numbers should come from the query results, not from the model’s arithmetic.

FAQ

Does chain-of-thought prompting work with every model?

The original research found the biggest gains in large models; smaller models benefited much less. Modern reasoning models already reason internally, so explicit instructions matter most with standard chat models.

Is chain of thought the same as prompt chaining?

No. Chain of thought happens within a single response. Prompt chaining splits a task into several separate model calls, where the output of one becomes the input of the next.

Should I show the reasoning to users?

Usually not. Show a clear final answer, and keep the reasoning for debugging or for expert users who need to audit how a conclusion was reached.

Want to design prompts and AI apps that reason reliably? Explore these Coderhouse courses:

• AI Engineering Course — build LLM apps and agents with prompting, tool use, and evaluation.

• Data Analytics Course — master SQL and analysis so the data behind AI answers is accurate.