LLM Reasoning · Language Models · AI Agents
CoT vs ReAct vs Toolformer: Three Classic Fixes for Multi-Step Tasks
CoT writes reasoning into the prompt, ReAct grounds that reasoning in tool observations, and Toolformer trains the model to call APIs itself. Same problem, three bets on where reliability comes from.
The one question all three papers answer
By early 2022 it was clear that large language models could absorb enormous amounts of knowledge and still fail at a grade-school math problem or a date calculation. Three papers, published within fourteen months of each other, attacked that gap from three different directions, and the modern agent stack is the descendant of all three. Chain-of-thought prompting (Jan 2022) says: keep the model in its own head, but make it write the steps. ReAct (Oct 2022) says: let it keep thinking, but force every few thoughts to touch the outside world. Toolformer (Feb 2023) says: stop relying on the prompt entirely, and train the model to call a calculator or a search API as a native act of generation.
They are not competitors so much as three layers. Almost every production agent today reasons step by step (chain-of-thought), interleaves that reasoning with tool observations (ReAct), and calls tools through a trained interface rather than a hand-written prompt (Toolformer and its successors). But each paper alone has a distinct mechanism, a distinct supervision story, and a distinct failure mode, and knowing which layer is failing is most of the debugging work.
Key numbers
These papers were never run against each other under one harness, so the honest table has one row of design comparison and three blocks of per-paper results. Do not read across the blocks as a ranking.
| Dimension | Chain-of-thought | ReAct | Toolformer |
|---|---|---|---|
| What changes | The few-shot exemplars include worked reasoning steps | The output space adds free-form thoughts interleaved with actions | The model is fine-tuned on self-annotated API calls |
| Supervision | None beyond hand-written exemplars | None beyond one or two hand-written exemplars | Self-supervised: keep a call only if it lowers next-token loss |
| Grounding | None; reasoning is a closed monologue | Each thought is conditioned on a real tool observation | Tool results are read back into the text |
| Tool chaining | n/a (no tools) | Sequential use of one tool (Wikipedia API) | Not supported; each API is called independently |
| Base model in the paper | PaLM 540B (also GPT-3, LaMDA) | PaLM 540B | GPT-J 6.7B |
Per-paper headline results:
| Result | Number | Setting |
|---|---|---|
| CoT on GSM8K | ~57% vs ~18% standard prompting | PaLM 540B, 8 exemplars, no training |
| CoT emergent threshold | No benefit below ~100B, sometimes hurts | 540B gains, 10B-class models flat or worse |
| CoT vs prior SOTA | Beats fine-tuned GPT-3 with a trained verifier | GSM8K, early 2022 |
| ReAct on ALFWorld | +34% absolute success over imitation- and RL-trained agents | 1-2 in-context examples vs thousands of trajectories |
| ReAct on WebShop | +10% absolute success rate | Same few-shot setup |
| ReAct on HotpotQA / Fever | Removes the hallucination and error propagation of pure CoT | Interleaving beats reasoning-alone and acting-alone |
| Toolformer downstream gains | Competitive with GPT-3 175B despite being ~25x smaller | GPT-J 6.7B base, zero-shot evaluations |
| Toolformer supervision | A handful of demos per API | Calculator, QA, two search engines, translation, calendar |
| Toolformer side effect | Core LM perplexity preserved | Tool use added without degrading base ability |
Chain-of-thought: the cheapest fix, with a hard size gate
The method is a one-line change to the prompt. Standard few-shot shows the model Q: ... A: <answer>; chain-of-thought shows Q: ... A: <reasoning steps, then answer>. Roughly eight hand-written exemplars are enough. With them, PaLM 540B jumps from about 18% to about 57% on GSM8K, beating the prior state of the art, which was a fine-tuned GPT-3 with a separately trained verifier. No gradient updates, no new data.
The under-appreciated finding is the gate. Below roughly 100B parameters, the same prompt does nothing or makes things worse: small models produce fluent, logically broken chains and land below where standard prompting put them. Chain-of-thought is an emergent ability; the capability has to be latent in the model already, and the prompt only surfaces it. That fact is why “just add think step by step” is not a universal recipe and why the reasoning-model era (o1, DeepSeek-R1) switched from prompting to training once it became clear the ceiling was in the weights, not the prompt.
Its failure mode is hallucinated certainty. The chain is a closed monologue: nothing in the loop checks an intermediate fact against reality, so one wrong premise snowballs into a confident wrong answer, and the written steps are not a faithful record of the model’s internal computation anyway.
ReAct: ground every few thoughts in an observation
ReAct’s fix is structural. The model alternates Thought → Action → Observation → Thought → ..., where actions are real tool calls and observations are what the tools return. Thoughts decompose the goal, track what is unknown, and reformulate failed searches; actions pull real facts in. Neither component wins alone on the knowledge tasks: reasoning-only repeats chain-of-thought’s hallucination problem, acting-only loses the plan. Interleaving is what removes hallucination and error propagation on HotpotQA and Fever.
The striking result is ALFWorld: a prompted model with one or two examples beats imitation- and RL-trained agents by 34% absolute success rate, and by 10% absolute on WebShop, where the baselines were trained on thousands of trajectories. ReAct is not an architecture or an objective; it is a format, and its contribution is the discovery that the format itself closes a reliability gap for a strong enough base model.
Its failure modes follow directly. The gains ride on the base model and the tool: a search that returns nothing useful gives the reasoning nothing to ground against, and the model stalls or hallucinates anyway. Trajectories can loop without converging, there is no principled stopping rule, and the benchmarks are narrow, mostly single-tool, mostly-text worlds.
Toolformer: make tool use a trained behavior, not a prompt trick
Toolformer takes the opposite bet on supervision. Instead of trusting the prompt to elicit tool use, it fine-tunes the model to emit API calls as part of plain text generation. The data creates itself: the model samples candidate calls at many positions in its pretraining corpus, executes them, and keeps a call only if inserting the result lowers the loss on the following tokens. A calculator call survives where it actually resolves an arithmetic span; a search call survives where the snippet genuinely helps the next words. Each of the five tools needs only a handful of human demonstrations to bootstrap.
The payoff is that a 6.7B-parameter GPT-J becomes competitive with the 175B GPT-3 across zero-shot downstream tasks, despite being roughly 25 times smaller, while its base language-modeling perplexity is preserved. This is the research ancestor of every function-calling API: the difference is that Toolformer learned when to call from data rather than relying on an inference-time schema.
Its ceiling is equally structural. The loss-reduction filter only recognizes locally useful calls, so tools whose payoff is diffuse or multi-step never make it into the data. Toolformer cannot chain APIs (one tool’s output cannot feed another), so compositional, multi-step tool use is explicitly out of scope, which is exactly the gap agent frameworks later filled on top of it.
When to use which
- Self-contained reasoning, large model, no tools needed: chain-of-thought is still the baseline. It costs eight exemplars and no training.
- Facts can go stale or must be verified mid-task: ReAct’s think-act-observe loop. Grounding beats longer reasoning in a vacuum every time the environment answers back.
- Shipping tool use to users who will not hand-tune prompts: the Toolformer lineage: train when to call, expose a schema at inference. Prompted ReAct is the research prototype; trained calling is the product.
- Multi-step composition across several tools: none of the three papers solves this alone, which is precisely why modern stacks combine all three layers rather than choosing among them.
Limits and open questions
The three papers use different base models (PaLM 540B twice, GPT-J 6.7B once), different tasks, and different measures of success, so no honest cross-paper ranking exists; the table above deliberately keeps each result attached to its own setting. All three were demonstrated in early-2023 regimes with hand-written exemplars and narrow benchmarks; how much of each result transfers to current models trained on reasoning traces and tool data is an open empirical question. And each method’s signature failure (unfaithful chains, grounding-free loops, no tool chaining) is still the failure you will meet first when the corresponding layer breaks in production.
FAQ
Which came first, chain-of-thought, ReAct, or Toolformer?
Chain-of-thought (January 2022), then ReAct (October 2022), then Toolformer (February 2023). ReAct explicitly builds on chain-of-thought, adding actions and observations to its reasoning traces; Toolformer is independent of both and replaces prompting with fine-tuning.
How much does chain-of-thought prompting improve GSM8K?
On PaLM 540B with eight exemplars, GSM8K solve rate rises from roughly 18% with standard prompting to about 57%, surpassing the prior best of a fine-tuned GPT-3 plus a trained verifier, with no additional training.
Does chain-of-thought work on small models?
No. The paper shows it is an emergent ability that appears around 100B parameters; below that, models write fluent but flawed reasoning and often score worse than with standard prompting.
How much better is ReAct than trained RL agents?
On ALFWorld, ReAct beats imitation- and reinforcement-learning-trained agents by 34% absolute success rate, and on WebShop by 10% absolute, using only one or two in-context examples instead of thousands of training trajectories.
Can Toolformer chain tools together?
No. Each API is called independently and one tool’s output cannot be fed into another, which the paper lists as a stated limitation and which later agent systems were built to address.
Is Toolformer the same as OpenAI function calling?
No. Both let a model invoke external functions, but Toolformer learns from data when to call and fine-tunes on the results, whereas function-calling APIs expose a fixed schema the model is prompted to use at inference time. Toolformer is the research precursor of that capability.