Reasoning and chain of thought: why thinking in tokens works, and what it costs

In 2022, researchers found that asking a model to show its working made it dramatically better at maths. Two years later, models were being trained to do it on their own, at length, and “reasoning model” became a product category. My inference box runs with reasoning on and an 8,192-token thinking budget. Here’s why the trick works, and what it really costs.

From prompting trick to discovery

• Chain-of-thought prompting (Wei et al., 2022): include worked examples with intermediate steps in the prompt, and the model writes its own steps before answering. On the GSM8K maths benchmark it took PaLM 540B from 17.9% to 56.9%.
• Zero-shot chain of thought (Kojima et al., 2022): no examples needed. Appending “Let’s think step by step” produced large gains on its own.
• Self-consistency (Wang et al., 2022): sample several reasoning chains and take the majority answer. Different paths to the same answer are evidence it’s right, and accuracy improved again.

The effect appeared mostly in large models. Small models produced fluent but wrong chains.

Why it works: computation you can’t do in one pass

A transformer does a fixed amount of computation per token: one pass through L layers. Some problems need more sequential steps than that depth provides, such as multi-step arithmetic or tracking state through a long procedure. Writing intermediate tokens gives the model a scratchpad. Each generated token is new input to the next pass, so the model can carry intermediate results forward and spend more serial computation on harder problems.

Theoretical work backs this up. Transformers with intermediate reasoning steps can solve classes of problems that fixed-depth transformers provably can’t (Feng et al., 2023; Merrill and Sabharwal, 2023). Chain of thought isn’t the model “showing” its thinking; to a large extent, it is the thinking.

Training models to think

The next step was to stop asking for reasoning and start training for it. OpenAI’s o1 (September 2024) was trained with reinforcement learning to produce long internal chains before answering. OpenAI reported that accuracy kept improving with both more RL training and more thinking time at inference. That’s test-time compute as a scaling axis alongside parameters and data.

DeepSeek-R1 (January 2025) published how to do it in the open. RL with GRPO and simple correctness rewards made long reasoning, self-verification and backtracking emerge without being demonstrated first (the DeepSeek note and the RL note have the details). Distilling those reasoning traces into small models transferred much of the ability cheaply.

What thinking costs

Thinking tokens are ordinary decode tokens. They’re generated one at a time at memory-bound speed (the decode note explains the ceiling). An 8,192-token thinking budget costs, before the answer even starts:

Decode speed	Time to spend 8,192 thinking tokens
82 tok/s (my box at peak)	~100 s
50 tok/s	~164 s
30 tok/s	~273 s

Thinking also fills the KV cache and the context window. And API providers bill reasoning tokens as output tokens, usually the most expensive kind.

That’s why serious deployments budget reasoning instead of leaving it unlimited. My production server runs with --reasoning-budget 8192. My fork’s server also accepts a reasoning_effort setting, either a word or a raw token budget, so a client can ask for less thinking on easy steps and more on hard ones, per request.

Is the visible reasoning faithful?

Not reliably. Turpin et al. (2023) showed models giving plausible chains of thought that omitted the real reason for an answer, for example a bias planted in the prompt. Later work on reasoning models found the same: their traces don’t always mention what actually drove the answer. Treat reasoning traces as useful but untrusted. They’re good for debugging and often right, but they’re not an audit log of the model’s computation.

What it changes for agents

• Planning improves. Reasoning before a tool call makes better plans and fewer pointless calls, especially for multi-step tasks.
• Latency multiplies. Every step of an agent loop pays for its thinking. A 30-step task with heavy reasoning per step takes much longer than the same task without it.
• Tool discipline can suffer. On my own agent benchmark, a reasoning-mode variant of Qwen3.6-27B scored 723.3 against 816.2 for the plain model, with a loop-discipline score of 63.2 against 89.0. That’s one harness and one model, but it shows that more thinking isn’t automatically a better agent (the leaderboard has the rest).
• Format pressure. Let the model reason freely, then constrain only the final tool call (the constrained decoding note covers how).

Rules I use

1. Budget reasoning per step, not per session. Easy steps get little or none; planning and debugging steps get more.
2. Keep old reasoning out of the context. Past thinking blocks usually add cost and noise to later turns; keep the conclusions, drop the traces.
3. Measure task success per minute, not benchmark accuracy alone, when choosing a reasoning setting for an agent.
4. Don’t trust traces as explanations. Verify outputs, not the story the model tells about them.

Sources: Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022). Kojima et al., Large Language Models are Zero-Shot Reasoners (2022). Wang et al., Self-Consistency Improves Chain of Thought Reasoning (2022). Feng et al., Towards Revealing the Mystery behind Chain of Thought (2023). Merrill and Sabharwal, The Expressive Power of Transformers with Chain of Thought (2023). OpenAI, Learning to Reason with LLMs (2024). DeepSeek-AI, DeepSeek-R1 (2025). Turpin et al., Language Models Don’t Always Say What They Think (2023). Chen et al., Reasoning Models Don’t Always Say What They Think (Anthropic, 2025). Timings are computed from the decode speeds shown.