Agents and Reinforcement Learning

• Policy \(\pi_\theta(a_t\mid c_t)\): the probability distribution over actions \(a_t\) given the current input \(c_t\). In this article the LLM implements it, and \(\theta\) is the model weights being trained.
• Environment: the external system that executes an action and produces the next state, observation, reward and termination signal. The state transition is written \(P(s_{t+1}\mid s_t,a_t)\).
• State \(s_t\): the information needed to determine how the environment changes after an action. In a coding environment that is the current working files, the checking rules, the remaining execution budget and so on. It may include information not yet revealed to the model.
• Observation \(o_t\): the information the environment discloses to the model. For example, whether the tests that were run passed, and their error messages, are an observation. The environment releases only part of its state as an observation.
• Context \(c_t\): what actually goes into this LLM call. It can hold the problem statement, previous actions and several observations at once. Writing the full interaction record as \(h_t\) and the context builder as \(C\), we have \(c_t=C(h_t)\). Length limits may force part of the record to be dropped or summarized.
• Action \(a_t\): the choice the policy makes, such as a code edit or a test call. The LLM produces it as a token sequence \(a_t=(y_{t,1},\ldots,y_{t,K_t})\). Here \(t\) indexes the decision step and \(K_t\) is the number of tokens in that action.
• Episode: one attempt, from reset to termination. Writing the number of actions as \(T\), it runs for \(t=0,\ldots,T-1\). The unit bundles many model responses and test cases rather than being one of either.
• Rollout: the process of running the policy to collect experience, or the sample obtained that way. It may be a completed episode or a partial record cut off midway.
• Trajectory \(\tau\): the ordered record of the observations, actions, rewards and so on that actually occurred. Writing only observations and actions, \(\tau=(o_0,a_0,o_1,\ldots,o_T)\).
• Reward \(r_t\) and return \(G_t\): the former is the evaluation number received after action \(a_t\), the latter the discounted sum from that point on, \(G_t=\sum_{u=t}^{T-1}\gamma^{u-t}r_u\). Here \(\gamma\) is the discount factor, and this article's default is \(\gamma=1\).
• Advantage \(A^\pi(h_t,a_t)=Q^\pi(h_t,a_t)-V^\pi(h_t)\): how much higher the expected return after taking a particular action, \(Q^\pi\), is than the baseline value of the current record, \(V^\pi\). Real training uses an estimate \(\hat A\), and GRPO builds the comparison value out of group rewards.
• Group and rollout batch: the \(G\) comparison attempts on the same problem form one group. Collecting \(G\) attempts for each of \(B\) problems puts \(N=BG\) trajectories in a batch. The group size \(G\) and the per-step return \(G_t\) share a letter and nothing else, so read the subscript to tell them apart.
• Loss mask \(m_{ik}\): 1 when token position \(k\) of trajectory \(i\) is used as a training target, 0 when it is excluded. The similarly named attention mask controls which input positions can be read, not which ones are trained on.
• Harness: the execution system that organizes the model's input construction, tool execution, state storage, result checking and control loop. The input construction \(C(h_t)\) is also a choice made in this layer. The main text spells out each of these roles.
• Verifier and value critic: a verifier inspects the outcome and assigns the reward; a value critic estimates the return from here on as \(\hat V(h_t)\). One judges an answer already produced, the other looks ahead to performance that has not happened yet.
• Behavior / current / reference policy \(\pi_{\mathrm{old}},\pi_\theta,\pi_{\mathrm{ref}}\): respectively the policy that generated the experience, the policy being optimized now, and the reference policy for KL regularization. Different weight snapshots of the same LLM can play these roles.
• Policy lag: the version delay, or the policy difference, between the collecting policy \(\pi_{\mathrm{old}}\) and the current learner \(\pi_\theta\). How many updates behind it has fallen and how far the two distributions have drifted are counted separately.

In this article an LLM-based agent is a system in which, within a given goal and set of constraints, the LLM chooses the next action, observes the result the environment returns, and continues execution until completion or abort. The policy is responsible for choosing; the environment executes the choice.

The definition presumes a loop: the model's choice leads to actual execution, and the result of that execution becomes the input to the next choice. Published guides draw the scope a little differently depending on how much they emphasize the autonomy of the execution flow.

Anthropic's Building effective agents distinguishes workflows from agents by who decides the flow of execution. In a workflow, the LLM and tools move along code paths written in advance. The model may produce intermediate results or choose a branch, but the structure that connects the whole task is designed by the developer. In an agent, the model receives feedback from the environment and decides during execution which tool to use next and how to proceed. For example, it looks at a failure log, judges which files to read further and whether additional checks are needed, and continues the task. Tool use or a loop by itself does not separate the two structures. The line falls where the scope of the decisions entrusted to the model ends ("What are agents?", "Agents").

OpenAI's A practical guide to building agents starts from the independence to carry out tasks on the user's behalf. Concretely, the LLM must manage the execution of a workflow, choose the tool that fits the current situation, and revise its actions when needed. Judging whether the task is complete, and halting execution and handing control back to the user on failure, are also part of this role. Autonomy here names how much judgment is handed to the model to advance the task within given tools and constraints, not how much execution it is allowed ("What is an agent?").

Google's Agents whitepaper (Wiesinger, Marlow & Vuskovic, 2024) describes an agent as an application that observes its environment and acts through tools to achieve a goal. It divides the components into the model, tools and the orchestration layer. The model handles reasoning and choice, tools give access to external information and execution capabilities, and the orchestration layer connects the repeated cycle of gathering information, reasoning and acting. Rather than an individual model call, the unit of the definition is the whole run, which continues until the goal is reached or execution stops ("What is an agent?", "The orchestration layer", pp. 5–7).

The three descriptions end up resting on two axes. One is the execution structure that links observation, action and feedback; the other is the scope of choice granted to the model within that structure. This article writes the part where the model chooses the next action from the current record as the policy \(\pi_\theta(a_t\mid c_t)\). What input the policy receives, which actions it can execute, and how their results return as the next input are organized by the harness surrounding the policy.

Harness: the system that connects the model's judgment to execution

A harness is the system that surrounds the model and organizes the execution of a task. It constructs the input shown to the model, executes the tool calls the model generates, preserves results and work state, and links to the next model call or to termination. Lilian Weng (2026) also includes how reasoning and planning proceed, context management, artifact storage and result evaluation. From this perspective a harness is the execution structure that lets a model carry one task through to the end.

Weng describes harness engineering as the design of runtimes and software systems, going beyond prompt templates. In a long run it is hard to put every log into the context each time, so state and artifacts are left in files and the needed parts are read back. If several tasks run in parallel, there also has to be a procedure for tracking each task's start, progress and result and for merging them. Mapped onto the execution process in this article, it looks like this.

For example, consider fixing findMax, a function that should return the largest value in a nonempty integer array. The current code initializes its candidate maximum to best = 0, then replaces it only when it encounters a larger value. For input [3, 1, 2], it correctly returns 3. For [-2, -5, -8], however, no element exceeds zero, so it returns 0 instead of the expected -2. The bug is this initialization; the goal is to fix the function so that it also returns the correct maximum for all-negative arrays.

The agent's initial user request is: “Check the failing tests for findMax in the repository and fix the function so that all tests pass.” The harness puts this request and the tool specifications for run_tests, read_code, write_code, and finish into the first context. The model has not yet been given the cause or the failing input explained above. It must discover them by running tests and reading the code. The array is the input to findMax; the request, tool specifications, and subsequent execution results are the inputs to the LLM.

This example has three tests: [3, 1, 2] → 3, [-2, -5, -8] → -2, and [0] → 0. The table follows one run in which the agent observes the failure, finds and fixes its cause, and reruns all three tests successfully.

The left column is decided by the model, the right column by the harness. What to check, how to fix it and when to stop are the model's choices. Passing those choices to the actual files and executor, and deciding which parts of the result to show as the next input, is the harness's design. What the rest of this article changes with reinforcement learning is the left column: the policy.

This breakdown explains how the policy and the system each contribute to performance. Even with the same model, cutting the key part of a failure log out of the context loses the basis for the next fix, and not saving the work state can lead to repeating checks that were already done. So when comparing agents' performance, the harness configuration has to be treated as an experimental condition alongside the model weights. Policy learning changes the probability of actions given an input; improving the harness changes the conditions under which the model judges and executes.

Harness and environment point at the same run from different angles. The environment refers to the thing that receives the policy's action and whose state and observations change; the harness refers to the implementation that structures execution between the model and that thing. For example, the repository and the test runner are the environment a coding policy interacts with, while how those tests are exposed as a tool and how the returned logs are put into the next context is harness design. In real implementations the code playing the two roles can overlap.

What is the environment for an LLM?

The environment is the external system that takes the action the model chose and produces its result and the next observation. It need not be the internet or a physical space. For a coding task, the working files, the executor, the tests and grading rules, and the reset and termination rules make up the environment. The problem statement is an input that introduces the environment, not the environment itself.

Separating editing from verification also sharpens the environment's boundary. The policy generates a call meaning "modify the file", and the environment changes the stored code. When the policy selects run_tests(), the environment runs the current code and returns expected versus actual values. The tests do not change the code, but they add information the model needs for its decision and consume execution budget.

The same split shows up in environment interfaces. Gymnasium's Env.reset returns the initial observation, and Env.step returns the observation, reward and termination signal after an action separately. A test failure log is likewise an observation used for the next decision, and how many points that run scores is set by the reward rule.

What actually carries out this choose–execute link is the harness from the previous subsection. It parses the call, runs the tool, puts the return value into the next context and controls the loop. Through the loop described so far only the input \(c_t\) and the environment state \(s_t\) change; the model weights \(\theta\) stay put.

Writing an LLM's actions as an MDP

Gathering the pieces so far turns the code-fixing process into a single decision problem. The current code and execution budget are the state \(s_t\), the call the model chose is the action \(a_t\), and the rule by which the executor changes the code and returns a result corresponds to the transition \(P(s_{t+1}\mid s_t,a_t)\). Collecting a state set \(\mathcal S\), an action set \(\mathcal A\), the transition, the conditional expected reward \(r(s,a)\) and a discount factor \(\gamma\) gives a Markov decision process (MDP). Since we deal with finite attempts, an initial distribution and a termination condition are specified as well.

The point where this model meets an LLM is the representation of an action. The actions an LLM produces are token sequences, so over a vocabulary \(V\) one action is

\[a=(y_1,\ldots,y_K)\in\bigcup_{K\le K_{\max}}V^{K}\]

Finite, but combinatorially enormous. With a vocabulary of 100,000 and a length of 50, the number of candidates is \(10^{250}\). Storing a value per action in a table is therefore impossible from the start, and the policy is instead factorized into token-level conditional distributions.

\[\pi_\theta(a\mid h)=\prod_{k=1}^{K}\pi_\theta(y_k\mid h,\,y_{<k})\]

This factorization is what lets us sample an action and compute its log probability and gradient. The probability of a variable-length action also covers the end token or an explicit stopping rule. The reason this article concentrates on policy gradients is that this representation connects naturally to a pretrained language model. A large action space does not mathematically forbid Q-learning: constructions with token-level value functions or a set of candidate actions are possible, but they have to handle the maximization over actions and the error in value estimation separately.

What the same token sequence is interpreted as is decided by the executor. Output the contents of a modified file and it is an artifact; call run_tests() and it is an execution request. A final answer of "the fix is done" is delivered to the user. All three use the one representation, a string; they part ways at what the executor does with it. In the figures below, editing code and running tests are each marked as an explicit action.

Having written actions as probabilities, we also have to say what those probabilities are conditioned on. If the code and the execution state are fully disclosed, this can be treated as a fully observed MDP. If hidden grading inputs or internal executor state are invisible to the model, it is modeled as a partially observed MDP (POMDP). Which of the two cases applies depends on how much of the information needed for the next choice actually reached the model; using an LLM does not by itself create partial observability.

Take the complete record \(h_t\), which preserves the initial task, all observations and all actions, as the state and you get a history MDP. The actual input, though, is usually a truncated or summarized context \(c_t=C(h_t)\), and the executed policy is \(\pi_\theta(a_t\mid C(h_t))\). The text writes \(\pi_\theta(a_t\mid h_t)\) for brevity, with this context construction understood to live inside the policy. That the full record is Markov does not mean the summarized input is.

For instance, if logs with different error locations and execution conditions are all summarized as "test failed", the information needed to tell the causes apart and choose the next action is lost. A POMDP's belief state \(b_t(s)=P(s_t=s\mid h_t)\) can be a sufficient statistic under a known environment model, but you cannot assume a natural-language summary plays that role. Change the context management and you change the basis for the next action.

This article uses a setting in which execution ends with a completion report or with the allotted budget running out. The default reward is 0 in the middle and 1 at termination if the task succeeds, with discount \(\gamma=1\). The run is finite, so a later action's reward counts for exactly as much as an earlier one's.

The policy gradient that follows starts from two things: the action probability under the input actually used, and the ability to collect that action's outcome again in the same environment. Full state observation and a differentiable environment are not among its premises.

The policy and the harness are different objects of design but operate together during execution, so evaluating the effect of training means recording which context construction and which tools the numbers came from.

Actually executing the action written \(a_t\) in the equations takes one more step. You have to decide which tool call the model's string is parsed into, and in what shape the return value goes back into the next input.

The string run_tests() is just tokens the model generated until it reaches an executor. The harness parses that output into a call, validates the arguments, runs the actual function and returns the result to the model. Implementing tool use means designing that whole round trip, not just the model's output format.

The five components of the harness defined earlier all appear here at once. The tool specification fixes what can be called, the parser turns the generation into a call, the execution environment performs it, context management puts the return value into the next input, and the control loop decides whether to continue or stop.

These implementation choices make the earlier decision model concrete. The tool specification fixes the available actions \(\mathcal A\), context management fixes the model input \(C(h_t)\), and the execution budget fixes the termination condition. The same LLM under different conditions is solving a different problem. When comparing the effect of policy training later, the harness has to be held fixed too.

A tool needs more than a name. You have to define what it does, which arguments it takes, and what shape results and errors come back in. The following is a condensed example not tied to any particular API. Real APIs use a call ID to link a request to its result. OpenAI function calling.

tool = {
    "name": "run_tests",
    "description": "return the test results for the current task",
    "parameters": {},  # no arguments; runs the whole suite
}

def run_agent(context, max_turns):
    for turn in range(max_turns):
        message = model(context, tools=[tool])
        context.append(message)

        if message.is_final:
            return context, "finished"

        call = validate(message.tool_call)
        result = execute(call.name, call.arguments)
        context.append(tool_result(call.id, result))

    return context, "budget_exhausted"

Structured function calling emits the name and arguments in a fixed format. The text-action style has a parser read a string like Action: search[…]. The code-execution style lets the model combine several tools inside a program it writes. Programmatic tool calling is the choice of handling loops and aggregation in code and returning only the needed result to the context. MCP is not a reasoning method or a reinforcement learning algorithm but a protocol for connecting tools. Anthropic advanced tool use.

A well-formed call is only part of an appropriate action. Call the test tool in the correct format but pick only tests unrelated to the behavior that was changed, and the run verifies nothing. Two independent inputs can be checked in parallel, but checking edited code requires the edit to finish first. The model has to learn input selection, ordering and error recovery, not just the format.

Reasoning and acting: ReAct

Once a call succeeds, the open question is what to do with what it returned. The model has to interpret the returned result and judge whether there is more to check or whether it has enough grounds for a fix. ReAct is the construction that handles this coupling of reasoning and acting.

Yao et al. (2023) proposed ReAct, which generates reasoning text and environment actions together (Chapter 2). Observations obtained by acting update the reasoning, and the updated reasoning picks the next action. What the construction asks for is that alternation, not a long deliberation printed before every call.

Self-evaluation and revision

This loop can also include reviewing one's own patch. If a different test fails after the fix, the model sums up the condition it missed as feedback. Fixing the current code again changes this attempt; carrying that feedback forward changes the next one.

Madaan et al. (2023)'s Self-Refine has the same model generate feedback and iteratively revise its output, a procedure that runs without any additional weight training (Abstract).

Shinn et al. (2023)'s Reflexion keeps verbal feedback in memory and uses it on later attempts (Abstract). In both, feedback is what changes the execution and no optimizer touches the weights, which is what separates them from the policy learning discussed here.

This feedback is the model's interpretation and nothing more. Generating the evaluative sentence "the problem is solved" is not the same as the code actually passing the tests. Hence the separate names for self-evaluation, for a verifier that checks the outcome, and for a value critic that predicts future return. While feedback piles up in the context, the model weights stay as they are. Making the same mistake less likely on the next task requires a training process distinct from mid-run revision.

The goal of reinforcement learning is to change the policy so as to raise the expected return over the task distribution. The experience of recovering from a failure in one run becomes a sample for the next round of training. Collect such experience repeatedly across diverse tasks, and what changes is not a one-off fix on a particular input but the tendency in how actions get chosen.

Supervised fine-tuning (SFT) raises the probability of a demonstrated action in a given context. Reinforcement learning (RL) evaluates attempts the policy itself produced and changes the choice probabilities according to the outcome. That difference matters when the process is hard to write out but the final result can be checked.

Adaptation at inference time is a change in the input record. Parameter learning is a change in the weights that implement the policy. A self-evaluation or a failure note only changes the input record; an optimizer update happens only in parameter learning.

Turning one recovery into a repeatable ability

Three interventions are available on the same failure record. During execution we add the counterexample to the input; in SFT we hand over the correct edit as the answer; in RL we grade the outcome of an attempt the model carried on by itself. These act respectively on the current record \(h_t\), on the probability of a demonstrated action, and on action probabilities weighted by reward.

For a model that gets the tool arguments wrong to begin with, check the specification and the demonstration data first. For a model that emits valid calls but chooses and recovers poorly after a failure, that is where training on environment reward comes up for comparison. RL is not a required component of an agent; it is a training method for improving these choices.

The condition that outcomes can be checked leads into RL reward design. Lambert et al. (2024) describe RLVR, which uses verifiable outcomes as reward (Chapter 6). In a coding task, whether the final artifact passes the tests is such a signal. Whether the checking inputs are sufficient, and whether search can find a succeeding path at all, are separate questions.

What is needed now is to represent "one solution attempt" as trainable data. Recording which action was chosen under which observation, and how the attempt ended, is what lets action probabilities be tied to reward. These records are collected and trained on, and generalization to new problems is evaluated on a separate task set.

An episode is one attempt, a rollout is the run, a trajectory is the record

Everything from resetting the environment, through the policy choosing actions and receiving observations, until a termination condition is reached is one episode — one attempt. Actually running the policy to collect experience over that stretch is a rollout. When the literature says "four rollouts" it sometimes means the four samples obtained from those runs.

A trajectory \(\tau\) is the ordered record of visited observations, chosen actions, received rewards and so on. If the episode fixes where a run starts and ends, the trajectory captures which path was taken inside it. Solve the same problem four times and you get four episodes and four trajectories. Run the tests three times within one episode and you still have one rollout and one trajectory.

Below is a short path one can get on a code-repair task. Here \(t\) numbers the tool calls and the terminating action, not the tokens. This article writes \(r_t\) for the reward received after executing action \(a_t\). Other texts may write the same reward as \(r_{t+1}\); the timing it refers to is the same.

The moment the passing test log was seen and the moment the reward arrived are deliberately separated here. The reward is given once, at termination, by grading the final code. You can build an environment that scores every intermediate test, but that is a different reward design. Figure 1 cuts a run record like this one into finer pieces and counts the tokens used for training.

A rollout need not be a completed episode. Cut the collector off after a fixed number of steps and you get only part of a trajectory. So when storing it, distinguish termination, which is the task itself ending, from truncation, which is the collection being cut short. The description of the two termination flags in Gymnasium's Env.step makes this distinction explicit. If the task rule itself is "solve it within five steps", the remaining count belongs in the state and exhausting it can be defined as terminal. Conversely, if a task that could have continued was cut for collection convenience, you must not automatically treat the remaining return as 0.

\[G_t=\sum_{u=t}^{T-1}\gamma^{u-t}r_u,\qquad R(\tau)=G_0.\]

The return \(G_t\) is the discounted sum of rewards from now until termination; the word reward is kept for the number received at a single step. This article's default setting has a finite \(T\), \(\gamma=1\) and only a final success reward, so \(G_t=1\) at every action along a successful path. The actions share one \(G_t\) while their contributions differ. Telling the choices that helped from the ones that merely came along is what leads to credit assignment later.

What group and batch actually count

A group is the \(G\) attempts obtained from the same problem and initial state using independent executors. GRPO compares rewards within it. A rollout batch is the bundle of experience collected in one round for training. Running \(G\) attempts on each of \(B\) problems gives \(N=BG\) trajectories in total. Running several tests within one episode counts toward neither \(G\) nor \(B\).

A minibatch is a training subset of the collected batch; a microbatch is a still smaller slice sized to fit in memory. Accumulating gradients over several microbatches and changing the weights once is gradient accumulation, and one actual update is an optimizer step. Going through the fixed collected data once is what we call an epoch here. In Chapter 5, Algorithm 1 of Schulman et al. (2017), one iteration nests two loops: collecting experience and optimizing that experience over several epochs and minibatches. The \(B,G,N\) in this text is this article's notation and differs from the paper's symbols for actor count and horizon.

Now the record used for training goes into notation. Denoting the interaction up to time \(t\) by \(h_t\) and the next action by \(a_t\), the policy is \(\pi_\theta(a_t\mid h_t)\). This probability is over the context actually constructed; it does not assume the model sees the full environment state \(s_t\).

\[a_t\sim\pi_\theta(\cdot\mid h_t),\qquad \tau=(o_0,a_0,o_1,a_1,\ldots,o_T)\]

Taking the log of an action's probability turns it into a sum of the log probabilities of the generated tokens. Tool calls follow the same decomposition; the actual number of tokens depends on the tokenizer and the call format.

\[\log\pi_\theta(a_t\mid h_t)=\sum_{k=1}^{K_t}\log\pi_\theta(y_{t,k}\mid h_t,\,y_{t,<k})\]

So this article has two time axes. The axis of exchange with the environment is measured in tool calls or submissions; the axis on which likelihood is computed is measured in tokens. The equations ahead are written per action but implemented per token. Lose that correspondence and the masking and clipping discussed later land in the wrong places.

Attach the final grading result to the observations and actions and you have one training sample. Here all intermediate rewards are 0, and termination gives 1 if the final code passes the checks. Under a binary reward, raising the expected return and raising the success probability are the same thing.

\[R(\tau)=\mathbf{1}\{\text{final check passes}\},\qquad J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]\]

Two kinds of randomness are layered under the expectation in \(J\). One is the task distribution: which coding problem gets drawn. The other is the stochasticity of the policy and the environment: even on the same task, a different action sequence comes out each time. Raising the success rate on one task and raising it on new tasks are different goals, which is why held-out evaluation is kept separate later.

In this simple setting \(J\) is the success probability over the task distribution. The general return is a discounted sum of rewards, but for now we use finite episodes and a discount of 1. The model does not need to differentiate the test executor. Once it has the outcome, it differentiates the probability of the action it chose.

What the model chose in this record are the calls and the responses. The test results were inserted by the environment. So when computing the policy gradient, the tokens of a tool observation must not be treated as if they were actions the model had taken.

Jin et al. (2025) do the same in Search-R1: retrieved results stay in the context but are excluded from the policy loss target (Chapter 3.1, Loss Masking for Retrieved Tokens). The same distinction applies to the test logs returned by a coding tool. The masking is about the policy loss; a separate auxiliary loss that predicts observations is a different matter.

Doing this gives two things from each attempt: the log probabilities of the actions the policy chose, and the reward that attempt received. The same way of recording holds outside agents. RLHF and RLVR, which grade a single response, are special cases of the same notation, so before moving on to algorithms we first sort out that relationship.

Grading a single response

The problem of taking a prompt \(x\), generating one response \(y\) and grading it with \(R(x,y)\) can be written as a contextual bandit that treats the whole response as one action. It is the trajectory of the previous section with only the initial observation \(o_0=x\) and a single action left.

\[J(\theta)=\mathbb E_{x\sim\mathcal D}\,\mathbb E_{y\sim\pi_\theta(\cdot\mid x)}\bigl[R(x,y)\bigr]\]

Solved per token it is a sequential process with a growing prefix, but no external tool observation interrupts the generation. In an agent, a search or code-execution result changes the next input. Generation that only appends its own output has no such branching.

The names point to different axes

RLHF emphasizes drawing the training signal from human feedback, RLVR a reward that uses checkable outcomes, and Agentic RL the training of a policy that interacts with an environment. The three are not algorithm names competing for one slot. DPO is the name of yet another axis: the optimization method.

So a single coding agent answers to several names at once. If it rewards passing tests while also using a reward model trained on human preferences it is both RLVR and RLHF, and since it interacts with tools it is Agentic RL as well. Whether it optimizes with PPO or GRPO does not change this classification. The SFT / preference reward model / PPO construction of Ouyang et al. (2022) is a representative instance, not a mandatory pipeline for all RLHF.

KL-regularized reward optimization

The objective commonly used in RLHF raises a learned reward \(r_\phi\) while adding a KL penalty so the policy does not drift too far from a reference policy \(\pi_{\mathrm{ref}}\) (usually the SFT model that training started from). It is the form met again in GRPO later.

\[\max_\theta\;\mathbb E_{x\sim\mathcal D,\,y\sim\pi_\theta(\cdot\mid x)}\bigl[r_\phi(x,y)\bigr]-\beta\,\mathbb E_{x\sim\mathcal D}\Bigl[D_{\mathrm{KL}}\bigl(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\mathrm{ref}}(\cdot\mid x)\bigr)\Bigr]\]

The optimal policy of this objective has the form \(\pi^*(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)\exp\bigl(r_\phi(x,y)/\beta\bigr)\). Read backwards, it says the reward can be expressed as a log-probability ratio between two policies, and that relation is where DPO starts.

Where DPO sits on this map

DPO uses pairs of a preferred response \(y_w\) and a less preferred one \(y_l\). Substituting the relation above into the Bradley–Terry preference model turns the steps of training a separate reward model and scoring online rollouts with it into the following direct preference loss. Vanilla DPO can be trained on fixed preference data (Rafailov et al., 2023).

\[\mathcal L_{\mathrm{DPO}}(\theta)=-\mathbb E_{(x,y_w,y_l)}\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right]\]

You can build preference pairs out of tool trajectories too. What enters the loss is the action tokens the policy generated; an environment observation is not an answer the policy is supposed to produce. Build the pairs so that observations and actions from different trajectories do not get mixed, and write down the probability model and the collection procedure you used. Train on fixed preference data alone and the new states the policy's own actions led it into never enter the data, while an online variant that repeatedly collects fresh preference pairs does cover them. What the loss is and how the data is collected are separate choices.

For policy-gradient methods that optimize a reward, the ingredients are action log probabilities and rewards. DPO instead uses preference pairs and log-probability ratios under two policies, without inserting a scalar reward directly. The next section turns to the former: estimating a direction for the LLM weights from sampled rewards.

The policy gradient is the method of finding a direction of improvement by differentiating the expected return with respect to the policy's parameters. A trajectory's probability factors into the actions the model chose and the environment's transitions. With the environment's rules held fixed, differentiating only the log probability of the actions the model generated is enough to estimate the gradient of the expected return.

Turning an expectation into something samples can compute

First fix one task. Once the reward function is given, what can change is the probability with which each path appears. Assigning more probability to successful paths raises the expected reward. But summing over every possible path is hard, so this sum has to be turned into an average over samples drawn from the policy.

\[\begin{aligned} J(\theta)&=\sum_\tau p_\theta(\tau)R(\tau)\\ \nabla_\theta J&=\sum_\tau R(\tau)\nabla_\theta p_\theta(\tau)\\ &=\sum_\tau p_\theta(\tau)R(\tau)\underbrace{\frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)}}_{\nabla_\theta\log p_\theta(\tau)}\\ &=\mathbb E_{\tau\sim p_\theta}\left[R(\tau)\nabla_\theta\log p_\theta(\tau)\right]\\ &=\mathbb E_{\tau\sim\pi_\theta}\left[R(\tau)\sum_t\nabla_\theta\log\pi_\theta(a_t\mid h_t)\right]. \end{aligned}\]

The second line holds the reward fixed and differentiates the path probability. The third line multiplies and divides by the probability to create \(p_\theta(\tau)\) inside the sum. With that term present, the fourth line can be read as "sample paths from the policy and average what is inside the brackets." The last line takes the log of the product of path probabilities, turning it into a sum of per-action log probabilities. The environment's transition terms are fixed, so they vanish under differentiation. This is the REINFORCE of Williams (1992), expanded in this trajectory notation.

Why the environment terms vanish shows up once the path probability is split apart. Writing the environment's observation-generating rule as \(P(o_{t+1}\mid h_t,a_t)\), the path probability factors as below.

\[p_\theta(\tau)=p(o_0)\prod_t\pi_\theta(a_t\mid h_t)P(o_{t+1}\mid h_t,a_t)\]

Under the assumption that the environment does not depend on \(\theta\) directly, the log-gradient of that term is 0. The indirect effect of the policy changing the distribution of environments it visits is contained in the trajectory probability. If the reward function changes during training, or the environment contains other trainable components, what this simple expansion holds fixed has to be stated again.

In practice you collect \(N\) paths and average \(R_i\sum_t\nabla_\theta\log\pi_\theta(a_{i,t}\mid h_{i,t})\). Here the gradient of the log probability is the direction that makes the observed action generated more often, and the reward is the weight on that direction. If the optimizer minimizes a loss, put a minus sign in front.

The gradient when there are only two actions

Figure 2 reduces this computation to a single decision. Let \(\pi_\theta(A)=p=\sigma(z)\) and \(\pi_\theta(B)=1-p\), and set the probabilities of receiving reward 1 to \(q_A=0.75\) and \(q_B=0.35\). \(z\) is the logit difference between A and B.

At \(p=0.5\) the exact gradient is \((0.75-0.35)\times0.25=0.1\). A single sample's estimate is \(R(\mathbf1\{a=A\}-p)\): +0.5 if it picks A and succeeds, −0.5 if it picks B and succeeds, 0 if it fails. A successful B is also reinforced in that sample, but averaged over repetitions, what remains is the direction that increases A, which succeeds more often. The dots in the figure are this sample average, computed again and again.

Baseline: keep the direction, reduce the noise

The problem is variance. One lucky success influences an entire long action sequence. On a problem whose success rate is already high, a reward of 1 is unremarkable; on a problem that almost always fails, the same 1 is a big result. To reflect that difference, subtract a reference value, a baseline, from the reward.

\[\hat g=\frac1N\sum_{i=1}^{N}\sum_t\bigl(G_{i,t}-b(h_{i,t})\bigr)\nabla_\theta\log\pi_\theta(a_{i,t}\mid h_{i,t})\]

The earlier equation's total reward \(R(\tau)\) became the per-step return \(G_t\) because a reward received before an action was chosen is not a consequence of that action. Averaging over actions at that record, the score multiplied by past rewards has expectation 0, so keeping only the subsequent rewards is fine. Here \(\gamma=1\). Subtracting a per-record reference \(b(h_t)\) on top of that makes the weight "how much better than expected in this state it did."

\(G_{i,t}\) is the return after that action. A baseline that does not depend on the chosen action leaves the expected gradient intact, under suitable conditions, and cuts only the variance. One good reference value is \(V^\pi(h_t)\), the return expected from that record. The difference between the actual return and the expectation is where advantage estimation starts.

The baseline has one condition to meet: it must not depend on the action that sample actually chose. Subtracting a value computed to include the sample's own reward — a group mean, say — breaks that condition, and for trajectories collected independently on the same initial task, subtracting only the self-inclusive mean shrinks the expected gradient by a factor of \((G-1)/G\). Add standard-deviation normalization or clipping and this factor alone no longer explains the result. In a large group it is negligible, but in a group of four attempts it is a 25% shrinkage. The leave-one-out mean, which excludes the sample's own reward, respects the condition.

Why the condition is needed shows up once you fix a record \(h\) and plug in an action-independent \(b(h)\). Under the usual conditions, such as exchanging summation and differentiation, the following term is 0.

\[\mathbb E_{a\sim\pi_\theta}[b(h)\nabla\log\pi_\theta(a\mid h)]=b(h)\nabla\sum_a\pi_\theta(a\mid h)=0.\]

\(b(h)\) comes out of the sum over actions, and what remains, \(\sum_a\pi_\theta(a\mid h)=1\), has derivative 0. A mean that includes the sample's own reward changes with the action, so it cannot be pulled out this way. In the simple case of independent samples from the same context, that difference shows up as the \((G-1)/G\) shrinkage above; dividing by the standard deviation adds a further data-dependent weight. Figure 2 compares the three choices side by side.

Every estimate so far rests on the premise that the policy that generated the experience and the policy being differentiated are the same. But update the weights once and the record you just collected becomes the experience of a previous policy. Reusing that record means handling this difference.

The fraction of the batch you just collected that chose action A reflects the probability under the policy at collection time. If the update raised that action's probability, a plain average over the same batch is no longer the experience distribution of the new policy. The on-policy / off-policy distinction starts from where the data came from.

Call the behavior policy that collected the experience \(\mu\), and the target policy you want to evaluate or improve \(\pi_\theta\). Using experience where the two coincide is on-policy; training the target policy on experience from a different behavior policy is off-policy. What settles it is which policy generated the actions, not how old the data is.

If in a fixed context action A had collection probability 0.2 and the target policy gives it 0.6, the weight on an A sample is 3. Because it appeared rarely during collection, it gets more weight when computing the target policy's average. That is importance sampling.

\[\mathbb{E}_{a\sim\pi}[f(a)]=\mathbb{E}_{a\sim\mu}\!\left[\frac{\pi(a)}{\mu(a)}f(a)\right]\]

Expand the expectation as a sum and the ratio's job is visible. Multiply each action's collection frequency \(\mu(a)\) by \(\pi(a)/\mu(a)\) and the target frequency \(\pi(a)\) is what remains.

The experiment below defaults to \(\mu(A)=0.2\) and \(\pi(A)=0.6\). A gets a weight of 3 and B a weight of 0.5. Computed as an exact expectation, \(0.2\times3\times0.75+0.8\times0.5\times0.35=0.59\), which equals \(0.6\times0.75+0.4\times0.35\) computed directly under the target policy. The experiment itself computes from 24 samples, so it has error around this value.

Every action the target policy might select must have positive probability under the collecting policy. If the collection probability is 0, there is no sample to reweight. Over a long trajectory, not just the actions but the distribution over visited records differs too. Exact trajectory correction requires a product of many probability ratios and its variance can grow large. A single token's ratio is one factor of that product.

Each action's probability is, as we saw earlier, a product of generated-token probabilities, so the per-action ratio is computed as a product of token ratios too. Over a whole trajectory those products stack up as below.

\[w(\tau)=\prod_t\frac{\pi_\theta(a_t\mid h_t)}{\mu(a_t\mid h_t)}\]

Under the same environment and initial distribution and the support condition, this full ratio is used to correct a trajectory-level expectation. The environment's transition terms cancel between numerator and denominator. PPO's per-token or per-action local surrogate does not use this product; it approximates on the spot instead of correcting the whole visitation distribution. Sampling that shrinks the support, such as top-p, needs particular care.

Generating fresh trajectories each time from a fixed list of problems is not offline RL. Conversely, online off-policy RL — continuously collecting new data while also using older experience from a replay buffer — is possible. Distillation also uses the word on-policy, where it means the student generated its own sequences, not necessarily that RL with environment rewards took place.

From here on the four symbols each point at something different. \(\pi_\theta\) is the current training policy, \(\pi_{\mathrm{old}}\) the snapshot at collection time, \(\mu\) the actual sampling distribution, and \(\pi_{\mathrm{ref}}\) the reference model the policy is compared against so it does not drift too far. In simple synchronous training you can start from \(\mu=\pi_{\mathrm{old}}\), but differences in temperature, top-p or the generation engine mean the actual probabilities need more care. DeepSeek-AI (2025), Chapter 3.1.

A probability ratio corrects for the distribution difference, but the further the collecting policy is from the current one, the higher the variance of the estimate can be. So instead of reusing old records without limit, the route taken is to use a recent batch only a few times while limiting the incentive to change.

When collecting experience is expensive, computing a single gradient from a batch feels wasteful. But keep updating on the same data and the current policy drifts away from the policy that produced it. PPO bounds how far that reuse goes with an objective called the clipped surrogate.

Schulman et al. (2017)'s PPO alternates experience collection and optimization (Chapter 5, Algorithm 1). It uses the collected batch for several epochs of minibatch updates, while equation (7)'s clipped surrogate limits the gain from continuing to push one sample's probability up or down.

\[\rho_t(\theta)=\frac{\pi_\theta(a_t\mid h_t)}{\pi_{\mathrm{old}}(a_t\mid h_t)}\] \[L^{\mathrm{clip}}(\theta)=\mathbb{E}_{\mathrm{old}}\!\left[\min\!\left(\rho_t\hat A_t,\operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t\right)\right]\]

A good action has \(\hat A_t>0\). Raising its probability raises the ratio and the objective with it. Past \(1+\epsilon\), though, the gain available from that sample flattens out. A bad action is the mirror image: pushing its probability down below \(1-\epsilon\) yields no further gain. In neither direction is the function cut off wholesale; the gradient goes to zero only in the direction that passed the limit.

Split the term that \(\min\) selects by the sign of the advantage and the shape of clipping becomes visible. Multiplying by a positive number preserves the ordering; multiplying by a negative number reverses it.

With \(A=1\) and \(\epsilon=0.2\), the objective stays at 1.2 even after \(\rho\) passes 1.2. Conversely, with \(A=-1\), lowering \(\rho\) below 0.8 still leaves the objective at −0.8. The extra gain from raising a good action's probability further, or from lowering a bad one's further, disappears; only changes in the opposite direction keep a gradient.

PPO is usually classified as on-policy: rollouts are collected with a recent policy, reused a limited number of times, and then collected again. From the second update within a batch onward, though, the collecting policy and the current policy already differ. PPO reaches as far as that local mismatch; safely reusing arbitrarily old records belongs to general off-policy learning.

Under the same relative upper threshold \(1+\epsilon\), a positive-advantage reward term becomes flat after an absolute probability increase of \(\epsilon\pi_{\mathrm{old}}(a\mid h)\). With \(\epsilon=0.2\), for example, that point comes after an increase of 0.002 for a token at probability 0.01 and 0.1 for one at 0.5. This does not constrain the actual probability to stay there. DAPO Chapter 3.1 connects the early saturation of the incentive to raise low-probability tokens with the observed entropy decrease. How much entropy actually falls, though, rests on more than the clipping asymmetry.

Yu et al. (2025) proposed Clip-Higher in DAPO: decoupling the upper and lower clipping widths and widening the upper one (Chapter 3.1). The design intent is more room to reinforce low-probability tokens. The result is evidence that clipping width acts on exploration and not only on update size. The gain was measured in that setting; what raising entropy does in general is a separate question.

We have not yet said how \(\hat A_t\) is computed. PPO commonly trains a value model \(V_\phi\) that predicts future return alongside the policy and estimates the advantage with GAE.

Given a value estimate, the one-step reward plus the next value can be compared against the current value. That difference is called the TD residual.

\[\delta_t=r_t+\gamma V_\phi(h_{t+1})-V_\phi(h_t),\qquad \hat A_t^{\mathrm{GAE}}=\sum_{l=0}^{T-t-1}(\gamma\lambda)^l\delta_{t+l}.\]

\(\lambda\) controls how far ahead residuals are accumulated, and when the value is inexact that choice is itself the bias–variance trade-off. At a true terminal the next value is set to 0, but if a continuing process was cut off by a collection limit, do not apply that treatment automatically. Whether a time limit is defined as the task's termination has to be stated as well (the GAE paper).

Clipping guarantees neither a strict upper bound on the actual KL nor monotone improvement. What it directly sets is the ratio range beyond which this sample's surrogate gain saturates; the actual update size and the advantage estimator are determined separately. Building the baseline by comparing several attempts at the same problem, with no value model, is the choice that leads to GRPO.

GRPO (Group Relative Policy Optimization) builds the advantage by comparing the rewards of several rollouts collected on the same problem. Because each problem gets its own baseline, a success on an easy problem and a success on a hard one are scored against different references. This group comparison takes the place of a value model's prediction.

Shao et al. (2024) proposed GRPO in DeepSeekMath, building the comparison baseline from rewards on the same problem (Chapter 4.1.1). The approach reduces the memory and training cost of a value critic, at the price of collecting several rollouts per task.

What "critic-free" removes is the value critic, not the grader. You still need a verifier or reward model to score the group.

\[\bar R=\frac1G\sum_{i=1}^G R_i,\qquad \hat A_i=\frac{R_i-\bar R}{\operatorname{std}(R_1,\ldots,R_G)+\varepsilon_{\mathrm{num}}}\]

Here \(\varepsilon_{\mathrm{num}}\) is a small number that avoids division by zero, unrelated to PPO's similarly written clipping width. In the basic account with outcome rewards, the same \(\hat A_i\) is applied to every generated token of a trajectory and a PPO-like clipped surrogate is optimized. The original GRPO also carries a KL term against a reference policy.

For example, if the rewards are \((1,0,0,1)\), the mean is 0.5 and the standard deviation is also 0.5, so the advantages are \((1,-1,-1,1)\). Keep only one success, \((1,0,0,0)\), and the mean is 0.25; the successful attempt's advantage becomes \(\sqrt3\) and the rest become \(-1/\sqrt3\). The same reward of 1 receives a signal of different size depending on the group it is compared against.

Yu et al. (2025) proposed dynamic sampling: drop groups whose rewards are all equal and fill the batch with groups that carry a comparison signal (DAPO Chapter 3.2). How much collection this needs depends not only on the target batch size but on how often such groups appear.

If tasks are too easy or too hard, many rollouts can yield few groups usable for reward comparison. The cost of the extra collection, or of the discarded experience, belongs in the training cost. So the effective-group fraction is logged alongside.

Some problems remain that group comparison does not solve. A successful path can carry unnecessary calls and still receive the same advantage, and dividing by the group standard deviation or the response length changes samples' relative weights. Reward only compares attempts; each token's training contribution is decided afterward, separately.

Why a KL to the reference model is attached separately

Optimizing reward alone can tilt the policy toward outputs that happen to suit the training task. To soften that, a KL penalty on the divergence from a reference policy \(\pi_{\mathrm{ref}}\) is sometimes added. The original GRPO includes this term. The reference model can be pinned to the initial model or refreshed at stated intervals.

\[J_{\mathrm{total}}(\theta)=J_{\mathrm{RL}}(\theta)-\beta\,\mathbb E_{h}\!\left[D_{\mathrm{KL}}\bigl(\pi_\theta(\cdot\mid h)\,\Vert\,\pi_{\mathrm{ref}}(\cdot\mid h)\bigr)\right]\]

The phenomenon this term guards against differs from the one PPO's clipping guards against. Clipping is a local device that keeps you from going too far from the collecting policy within one batch; the KL penalty keeps you, across the whole of training, from drifting away from a chosen reference model. Whether the reference was frozen or refreshed, and under what rule if refreshed, has to be reported alongside. A larger \(\beta\) prices departures from the reference policy more heavily. KL regularization reaches only as far as the observed prefixes, though; whether ability on other tasks survives has to be checked separately. Some recent public work drops this term, and there is no universally correct value. In a batch where every reward in a group is equal and the comparison signal goes to zero, this term is still there.

The ingredients for an update are now in place: the probabilities of generated tokens, the ratio against the collecting policy, and the advantage computed within a group. Moving them into actual tensors also requires deciding which positions carry the loss and what to divide by.

In an LLM, the probability of an action is decomposed into the probabilities of its generated tokens. So moving the reward attached to a trajectory into an actual loss requires deciding the per-token log probabilities, the training-target mask, the advantage, and the unit over which to average. The computation below uses one group of four rollouts and focuses on outcome-GRPO's comparison signal and PPO's token-level clipping.

1. The collection buffer needs more than the answer string

First freeze the behavior policy \(\pi_{\mathrm{old}}\) and collect experience. The buffer stores the problem, group ID and trajectory ID, the token IDs of the context actually fed to each model call, the generated token IDs, the collection log probability of each generated token, the tool calls and return values, the rewards, and the termination reason. The environment and grader versions, the policy checkpoint, the tokenizer and chat template, and the sampling settings must be identifiable too. Store only the successful final code and you cannot recover which action was chosen under which input.

GLM-5 Team (2026) point out that re-tokenizing text can produce a different token sequence than the one used at collection time (Chapter 4.1.2, Token-in-Token-out vs. Text-in-Text-out). A buffer that keeps only a tidied transcript cannot recover that difference, so it must preserve the token IDs actually used in the model calls.

2. Compute advantages from four attempts

Suppose the same problem and initial code were copied into four isolated executors and rollouts collected. The rewards are \((1,0,0,0)\) and the counts of trainable generated tokens are \((2,4,4,6)\). The token counts are for hand computation, not real tool-call lengths. The group mean is \(1/4\) and the population standard deviation is \(\sqrt{3}/4\), so ignoring the numerical stabilizer the advantages are

\[\hat A=(\sqrt3,-1/\sqrt3,-1/\sqrt3,-1/\sqrt3)\approx(1.732,-0.577,-0.577,-0.577).\]

A reward of 0 became a negative advantage because there is a success in the same group. Fail all four times and every centered advantage is 0. In baseline-free REINFORCE, by contrast, a sample with reward 0 has zero gradient from the start. "We learn from failure" is only a complete statement once you say which comparison baseline is used.

Per Shao et al. (2024), outcome-GRPO normalizes the final reward within the group and assigns that value to every token of the same response (Chapter 4.1.2, Outcome Supervision RL with GRPO). Here it is applied to all model-generated tokens of one episode, tool observations excluded. The statistic must be computed over all \(G\) rewards before splitting into minibatches; renormalizing inside an arbitrary microbatch changes the comparison set.

3. One token's computation and the direction of the update

For one token of a successful trajectory with stored log probability \(\ell_{\mathrm{old}}\) and current log probability \(\ell_\theta\), the ratio is \(\rho=\exp(\ell_\theta-\ell_{\mathrm{old}})\). With \(\rho=1.3\) and a clipping width of 0.2, the surrogate contribution is \(\min(1.3\times1.732,1.2\times1.732)=2.078\). This sample's reward term is flat in the direction of further increase. At the same ratio, a failing trajectory gives \(\min(1.3\times-0.577,1.2\times-0.577)=-0.750\). Since the probability of a bad choice rose, a gradient that pushes it back remains.

The \(L\) in the equations is a surrogate to be maximized. It enters the loss an ordinary optimizer minimizes with the sign flipped. And "this token's term is flat" is a statement about that one term: gradients from other tokens and from the KL term keep moving the same shared parameters.

Schulman et al. (2017) define the clipped surrogate in Chapter 3, equation (7). The same min applies to positive and negative advantages alike, so the saturation directions in the two cases are opposite. The text applies that structure to token-level conditional probabilities. A single per-token ratio is not the product of importance ratios over the whole trajectory, and we do not call this local surrogate the exact expected return of the current policy.

4. Different averages under the same name

Write the generated-token mask as \(m_{ik}\), its count as \(L_i=\sum_k m_{ik}\), and the per-token clipped contribution as \(u_{ik}\). Here there is one problem, so \(B=1,N=G=4\). The three reductions below weight tokens differently when lengths differ. The comparison controls only how the same per-token contributions are averaged, not the whole algorithm.

The last two differ only by a common scale within one batch, but the scale changes across batches as the total token count changes. The first weights tokens of short responses more heavily even within the same batch. The score-function gradient derived from the expected reward \(J=\mathbb E[R]\) originally contains a sum over tokens per trajectory. A length average is not a shortcut for computing that sum; it is a choice that changes the weight on each token.

What the denominator does to training weights

Shao et al. (2024)'s GRPO objective has \(1/|o_i|\) inside the group average (Chapter 4.1.1, equation (3)). Because token contributions are averaged per response first, tokens of short responses get a larger coefficient. Applying this to an agent also requires stating whether a response means one tool call or the entire generation of one episode.

Yu et al. (2025) divide by the total token count of the group collected on one problem (DAPO Chapter 3.3, equation (12)). This is the second row of the table above. Once several problems are bundled, averaging per-problem token averages and averaging over all batch tokens at once diverge. The latter gives more total weight to problems with more tokens.

Liu et al. (2025) analyze the effect of length normalization and propose Dr. GRPO, which uses a fixed constant as the denominator (Chapter 3.1–3.2). That method also removes the division by the group standard deviation, so the table above, which holds the advantage fixed and compares only denominators, shows one part of Dr. GRPO rather than the whole of it.

The difference shows up in the numbers. With \(\rho=1\) at every valid token, the minimized loss under the three schemes is \(0\), \(1/(2\sqrt3)\approx0.2887\) and \(1/(3\sqrt3)\approx0.1925\) respectively. A loss of 0 does not mean the gradient is 0. In the first scheme the advantages sum to zero, but the derivatives of log probabilities at different tokens and prefixes differ, so they do not generally cancel. Do not compare loss numbers from different reductions as if they were performance scores.

5. Moving trajectories into padded tensors

Assume first that a trajectory can be represented as one causal token sequence, with no mid-run truncation or rewriting of the context. \(N=BG\) is the number of trajectories, \(S\) the padded total length, and \(V\) the vocabulary size. \(S\) counts the observations too, while \(L_i\) counts only the trainable generated tokens inside it.

The next-token shift is the most common place to go wrong. The logits at position \(j\) predict the token ID at position \(j+1\). So pair logits[:, :-1] with input_ids[:, 1:], and move the mask that marks target roles to generated_mask[:, 1:] as well. The last token position of an observation predicts the first token of the next model output. Whether a position carries loss is decided by the role of the predicted token, not the role of the position the logits sit at.

This target distinction is the same reason Jin et al. (2025) excluded retrieval results from the policy loss (Chapter 3.1, Loss Masking for Retrieved Tokens). A tool observation is input that must be read, but it is not an action the policy sampled. With the loss mask at 0 the observation still sits in attention, and the hidden state that read it still sits on the backward path.

If the real harness summarized or deleted intermediate context, a forward pass over one concatenated conversation will not reproduce the conditional probabilities from collection time. In that case, replay the actual input and generation span of each model call separately, and gather the loss by linking generated tokens back to their original trajectory and group. When packing several records into one row, do not mistake the row count for the trajectory count, and preserve the attention boundaries, position IDs and ID mapping between records.

6. Separate in code what is frozen from what is differentiated

import torch

# The core computation in Python / PyTorch style. Not a complete trainer.
# Assumes: N=B*G with each group in G consecutive rows, one collecting policy,
# no context rewriting, right padding, temperature=1, no top-k/p restriction.
# old_logp: stored at collection time, already aligned to target ids[:, 1:], [N, S-1].
# KL, entropy and value losses are excluded (beta=0). This time, averaged over all batch tokens.

with torch.no_grad():
    r = rewards.reshape(B, G).float()
    centered = r - r.mean(dim=1, keepdim=True)
    std = r.std(dim=1, correction=0, keepdim=True)
    advantage = (centered / (std + 1e-6)).reshape(N)

logits = student(input_ids, attention_mask=attention_mask).logits
target = input_ids[:, 1:]
new_logp = logits[:, :-1].float().log_softmax(dim=-1)
new_logp = new_logp.gather(-1, target.unsqueeze(-1)).squeeze(-1)
mask = generated_mask[:, 1:].bool() & attention_mask[:, 1:].bool()
assert mask.any(dim=1).all()  # in this example every record has trainable tokens.

# Do not exp() the arbitrary logp at padding and then multiply by 0.
# Select the valid generated positions first.
delta = new_logp[mask] - old_logp.detach()[mask]
assert torch.isfinite(delta).all()
ratio = delta.exp()
assert torch.isfinite(ratio).all()
adv_token = advantage.detach()[:, None].expand_as(new_logp)[mask]
unclipped = ratio * adv_token
clipped = ratio.clamp(1 - 0.2, 1 + 0.2) * adv_token
policy_loss = -torch.minimum(unclipped, clipped).mean()

optimizer.zero_grad()
policy_loss.backward()
optimizer.step()

Only the current model's logits → log_softmax → gather → ratio → loss is differentiated. The rewards, group statistics and collection log probabilities are frozen for this update. Nothing backpropagates into sampled integer token IDs or into the JavaScript test executor. Differentiating the probability the current policy assigns to already-chosen tokens instead is the heart of the score-function method.

The code above assumes for exposition that the whole batch fits in memory. Even when split into minibatches, advantages must be computed over complete groups first. If microbatches hold different token counts, plainly averaging their mean() values cannot reproduce the overall token average. Preserve the reduction by putting the optimizer step's total valid token count in the denominator and accumulating each slice's sum. In distributed training the same holds: averaging the per-rank averages and averaging over all tokens globally give different numbers.

7. Checks to pass before the first optimizer step

Probability reproduction: with the same policy snapshot, the same input and the same probability definition, the ratio before an update should be near 1. If it is not, look at the token shift, chat template, context changes, padding, dropout and inference-engine differences before touching the learning rate. The code above assumes pure softmax sampling. If temperature or top-p changed the collection distribution, state whether the stored log probabilities belong to that distribution or to the original model, and decide what the ratio is correcting for.

Boundaries and signs: check that the policy-target count is 0 for prompt, tool and padding tokens, that generated end tokens were not dropped, and that each group ID compares the same task. If all rewards are equal, the advantage and the policy-loss gradient of this construction should be 0. An excessively large ratio at a positive advantage should saturate the reward term, while a large ratio at a negative advantage should leave a gradient that pushes it back.

Keep regularizers separate: \(\pi_{\mathrm{old}}\) is the policy that generated the experience and \(\pi_{\mathrm{ref}}\) is the reference of the KL. The names look alike, but each plays its own role. Adding a KL means defining its direction, prefix distribution, sample estimator and reduction, and only then adding it. Not hiding those choices behind a single undefined reference_kl variable is where a reproducible implementation starts.

Get this computation right and gradients reach the tokens of a rewarded path. But correctness of the computation and usefulness of the training signal are different things. Which tests and which edits inside a successful path were actually necessary is not immediately visible from a single final reward.

The computation above assigned the same advantage to every generated token of a successful trajectory. If that path contained unnecessary repeated calls, those calls are reinforced by exactly the same amount. This alone does not make the REINFORCE estimator wrong, but with limited samples it is hard to separate the checks that were needed from the actions that merely came along. That is the credit assignment problem.

\[Q^\pi(h,a)=\mathbb E[G_t\mid h_t=h,a_t=a],\qquad A^\pi(h,a)=Q^\pi(h,a)-V^\pi(h).\]

Here \(Q^\pi\) is the expected return when the rest of the process after that action is also carried out by the current policy. The yardstick is the current policy's ability, not \(Q^*\), which asks whether an ideal policy could recover. If a novice model cannot interpret a test log, even a useful test call may score low under the current \(Q^\pi\). That is why gathering information and acting on that information have to be learned together.

Under sparse reward, observing a success comes before choosing an algorithm

If the initial policy never succeeds there is no successful path to compare a failing one against, and separating actions' contributions only becomes a question after that. If a task's rollouts are independent with success probability \(p\) and group size \(G\), then under a binary reward the probability of getting a group with both successes and failures is

\[P(\text{mixed group})=1-p^G-(1-p)^G.\]

This is a direct computation: one minus the probability that all succeed and the probability that all fail. At \(p=0.5,G=4\) it is 87.5%, but at \(p=0.01,G=4\) it is about 3.94%. Raising \(G\) to 16 only brings it to about 14.85%, at four times the cost per group. Rather than collecting only easy problems or enlarging groups unconditionally, decide by weighing the initial policy's ability, the task difficulty and the collection cost together.

Dynamic sampling, which keeps only mixed groups, can increase the amount of usable reward signal, but it is not a procedure that averages over the original task distribution. If hard tasks were discarded, report the fraction and the extra collection cost. Comparing only "performance per effective batch" hides the cost of the rejected rollouts. Yu et al. (2025), Chapter 3.2.

When the final reward is not enough

With few successful samples or long action sequences, more information about intermediate decisions can be obtained: predict a value, score the process, or branch from the same state with different actions and compare what follows. Each demands extra supervision or extra execution cost.

Branching rollouts estimate the difference in expected outcome between actions, but one success and one failure do not settle a causal contribution. Sampling after the two branches differs, and the environment can be stochastic. A replicable coding environment is a good place to design this experiment; an external transaction or a conversation with a user is hard to put back into the same state.

Does adding intermediate rewards solve the same problem?

Give "+0.1 every time the tests are run" and you may get a verification habit, but a policy that only repeats tests also gains. The classical condition for changing the signal while leaving the final-success problem intact is potential-based shaping. For a state potential \(\Phi\), set \(r'_t=r_t+\gamma\Phi(h_{t+1})-\Phi(h_t)\). Ng, Harada & Russell (1999).

\[\sum_{t=0}^{T-1}\gamma^t r'_t=\sum_{t=0}^{T-1}\gamma^t r_t-\Phi(h_0)+\gamma^T\Phi(h_T).\]

The intermediate terms cancel. In a finite episode with terminal potential set to 0, the shaped return differs from the original return only by a constant that depends on the initial state, so the ranking between policies is preserved. The conclusion survives only as long as the terminal potential and the time handling are kept intact, and a learned process reward does not take this form on its own. So when adding a reward, first ask whether it helps learning the original objective or changes the objective itself.

Getting a comparison signal from the environment costs exploration and grading. If a good teacher already exists, the starting point can change: rather than waiting for the student to discover a successful path on its own, take the teacher's repair process or next-action distribution as the training signal.

Distillation is the approach of using a teacher model's outputs or distributions as the training signal for a student model. In an agent, what gets imitated is not only the final answer but also the choice of tool, the call arguments and the follow-up actions. One can start by verifying trajectories the teacher generated and using them as SFT data.

\[\mathcal L_{\mathrm{SFT}}=-\mathbb E_{\tau\sim\pi_{\mathrm{teacher}}}\!\left[\sum_{k\in\mathcal M(\tau)}\log\pi_\theta(y_k\mid h_k)\right]\]

Here \(\mathcal M\) is the set of positions of model-generated tokens the student should imitate. File contents returned by a tool are used as context, not imitated as action targets. The distinction between generated spans and observation spans from Figure 1 is used here as well. This path can also be built with an API teacher that returns text and tool calls. But do not assume the teacher's long reasoning traces or private internal thoughts are all available. Build training data from outputs you can actually observe.

What on-policy and off-policy refer to in distillation

Even with teacher data, where the experience came from still matters. The student can imitate paths the teacher generated, or it can ask the teacher for the next action on paths the student produced itself. The earlier on-policy / off-policy distinction captures that difference. Whether supervision comes from a teacher or from an environment reward is a separate question.

Any training setup fixes two separate things. ① Who generated the record (behavior policy): the teacher, a past student, or the current student. ② What supervision is given at that position (supervision): a single target token, the teacher's full distribution, or a scalar reward from the environment. "On-policy" in distillation answers ① only; it is not a claim about environment rewards. The student generated its own sequences, and ② is still filled in by the teacher.

① and ② are fixed separately, so reading both off a single name breeds confusion. A setup where "the teacher scores the student's answer out of 10" is on-policy, since ① is the student, but ② is a single scalar and so is not the same objective as minimizing a dense KL. Conversely, "keep only the student's passing answers and run SFT on them" (rejection sampling / self-distillation) has the student at ① and a hard target at ②.

Why the teacher's paths alone are not enough

Once the student picks an action different from the teacher's, it may afterwards meet observations and failures absent from the teacher's demonstration. During training the teacher's prefix is given (teacher forcing), but at deployment the student's own previous output builds the next input.

Agarwal et al. (2024) point to the distribution gap between training on teacher data and generating with the student (Chapter 1), and propose GKD, which supervises with the teacher distribution on sequences the student produced (Chapter 3.1). In effect, it supplies supervision on what to generate at the inputs the student itself visited.

Even when the student generates the sequence, the supervision comes from the teacher's distribution. Rather than estimating a policy gradient from environment rewards, the on-policy distillation of this section reduces the gap to the teacher's distribution on the prefixes the student visited.

\[\mathcal L_{\mathrm{local}}=\mathbb E_{h\sim d_{\pi_\theta}}\!\left[D\bigl(\pi_{\mathrm{teacher}}(\cdot\mid h),\,\pi_\theta(\cdot\mid h)\bigr)\right]\]

What changed is the subscript of the expectation: it is now taken over \(d_{\pi_\theta}\), the student's visitation distribution. Actual implementations, though, freeze the collected prefixes and differentiate only \(D\) on top of them, so the dependence of the visitation distribution on \(\theta\) never enters the derivative. Handling that term brings the policy gradient term back.

Differentiating everything, that term included, splits it into two. This is the log-derivative trick applied as usual, not a result from any particular paper.

\[\nabla_\theta\,\mathbb E_{h\sim d_{\pi_\theta}}[D(h;\theta)]=\underbrace{\mathbb E_{h\sim d_{\pi_\theta}}[\nabla_\theta D(h;\theta)]}_{\text{term implemented}}+\underbrace{\mathbb E_{h\sim d_{\pi_\theta}}\!\left[D(h;\theta)\,\nabla_\theta\log d_{\pi_\theta}(h)\right]}_{\text{term usually dropped}}\]

The second term is itself a score-function term weighting \(D\) as a return. Viewed along the negative gradient that minimizes the loss, it corresponds to a policy gradient treating \(-D\) as reward. It is a term that pushes the policy toward visiting records where the divergence is smaller. The usual implementation, which freezes the collected prefixes and computes only the first term, discards it. So what on-policy distillation actually optimizes is not the student's visitation distribution but matching the conditional distribution over the current visitation distribution.

The direction of the divergence and the density of the signal

Several divergences can occupy the slot \(D\), and their properties depend on direction. Forward KL \(D_{\mathrm{KL}}(\pi_{\mathrm{teacher}}\Vert\pi_\theta)\) makes the student put probability everywhere the teacher does (mode-covering), while reverse KL \(D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{teacher}})\) heavily penalizes the student for putting probability where the teacher's is very low. These are often described as mode-covering versus mode-seeking tendencies. The direction of a single token-level KL does not, however, settle the diversity of the agent's whole strategy. With no representational constraint both KLs match the teacher distribution at the global optimum, and the difference in direction shows up in the path there.

The density of supervision also differs. A hard target gives one answer per position, a dense KL a full vocabulary distribution per position, and environment-reward RL one scalar per episode. The form and count of supervision used at a position differ that much, but the vocabulary size alone does not let you compute an effective information content or a sample-efficiency multiplier. Density is also separate from correctness: if the teacher does not know what is right on the student's unfamiliar mistaken path, a dense KL hands over a wrong answer spread across the whole vocabulary.

The cost asymmetry is large as well. Off-policy data, built once, can be reused across students and experiments. On-policy distillation needs new sequences whenever the student changes, so the teacher has to live inside the training loop. And since the student keeps updating, the student sequences collected a moment ago were already made by a slightly older policy. Policy lag between collection and update has to be managed even when using the student's own data.

Distillation before and after RL

Learning from a teacher serves different purposes depending on where in training it sits. It can raise the initial policy's ability, gather the abilities of several expert models, or preserve an ability acquired at an earlier stage.

Cold start, before RL

One reason to put teacher data ahead of RL is that the initial policy fails so often that no comparison signal appears at all. DeepSeek-R1's stated reason is a slightly different one. DeepSeek-AI (2025) write that to avoid the "early unstable cold start phase of RL training from the base model", they first fine-tuned DeepSeek-V3-Base on thousands of cold-start samples (Chapter 2.3.1).

That data came not from a new annotation effort but from combining models they already had with human post-processing: few-shot prompting with long chain-of-thought examples, prompting directly for answers that contain verification and reflection, collecting R1-Zero outputs in a readable format, and refining by hand (Chapter 2.3.1). The records came from models other than the student (a prompted model, or R1-Zero) plus human post-processing, and the supervision is a hard target, so cold start is one instance of off-policy distillation.

The reported benefits split in two as well. One is readability: R1-Zero produced language-mixed output, so the cold-start data enforced a format separating the reasoning process from the summary. The other is performance: starting from cold-start data whose format was designed with human priors, they report observing better performance than R1-Zero (Chapter 2.3.1). What cold start does is not create final performance but install the format and the initial ability that let later training proceed. R1-Zero trained without any cold start (Chapter 2.2), so the stage is a design choice for readability and performance, not a prerequisite.

If the student gets even the tool-call format wrong often enough that run_tests() does not parse, no increase in group size will produce a success to compare. What is needed then is not a new optimizer but a small set of demonstrations that fix the format and the basic procedure. If what is lacking is the interpretation of failure logs, the comparison is between supplying recovery demonstrations in that state and supplying more exploration.

For a small student the stakes of this choice are higher. The same report fine-tunes small models with SFT only and no RL stage, on 800k samples curated with R1 (600k reasoning, 200k non-reasoning; the non-reasoning data reuses part of DeepSeek-V3's SFT data) (Chapter 2.3.3, Chapter 2.4), and compares large-scale RL alone against distillation on a 32B student. On AIME 2024 pass@1 the former scores 47.0% and the latter 72.6% (Chapter 4.1, Table 6). The same section adds, though, that "while distillation strategies are both economical and effective, advancing beyond the boundaries of intelligence may still require more powerful base models and larger-scale reinforcement learning." Where the goal is ability beyond the teacher's, the RL stage remains.

Building the initial policy: Qwen3's two stages

Qwen Team (2025) used an off-policy stage training on teacher responses, followed by an on-policy stage matching the teacher's distribution on student-generated sequences (Chapter 4.5, Strong-to-Weak Distillation). If the first stage supplies the teacher's paths, the second continues learning on the paths the student actually produces.

In the same report's comparison, distillation gave better results than RL at roughly one tenth the GPU time (Chapter 4.7, Table 21). That comparison, though, is made for that student model and training setup and presupposes a good teacher already exists, so the cost of building the teacher sits outside the one-tenth figure.

After initialization, several abilities may need to be gathered into one model. Qwen Team (2026) used distillation in Qwen3-Coder-Next to consolidate domain experts into the deployed model (Chapter 4.2.5). Here the purpose is not simply shrinking the model but merging specialist abilities.

When earlier abilities degrade as training proceeds, a past checkpoint can serve as the teacher. GLM-5 Team (2026) use this kind of cross-stage distillation (Chapter 3.5).

GLM-5 sets GRPO's group size to 1 at this stage, because the advantage comes directly from the gap to the teacher rather than from a group comparison. In the outcome-GRPO of the earlier GRPO section a group size of 1 would leave nothing to compare against and the signal would go to zero; here the teacher occupies that slot.

Across those four cases distillation occupies three slots: initialization, consolidating abilities, and retaining them, and any of them can be combined with environment-reward RL. The order of training follows from what actions and distributions the teacher can supply, the student's base ability, and the quality of the reward available from the environment.

Whichever signal you choose, training needs a repeatable collection procedure. How preparing the initial policy, collecting rollouts and computing the loss fit into a single loop, with environment-reward RL as the reference, is what the next section sets out.

The loss computation above started from an already-collected batch. A real trainer is responsible for producing that batch too. It draws tasks, clones the initial environment, and runs each attempt under a frozen collecting policy. It grades the outcomes, computes group advantages and updates the policy, and the next collection round uses the new policy.

student = initialize_from_pretrained_or_distilled_model()

for iteration in range(num_iterations):
    behavior = snapshot(student)
    tasks = sample_tasks()
    trajectories = []

    for group_id, task in enumerate(tasks):
        for sample_id in range(group_size):
            # Clone the same initial state and run with different sampling seeds.
            env = reset_isolated_environment(task)
            trace = rollout(
                policy=behavior,
                env=env,
                tools=tools,
                budget=budget,
                seed=make_seed(iteration, group_id, sample_id),
            )
            reward = verify_final_outcome(env, trace)
            trajectories.append({
                "group_id": group_id,
                "task_id": task.id,
                "sample_id": sample_id,
                "trace": trace,
                "reward": reward,
            })

    # Compute advantages over complete groups, then split into training batches.
    advantages = estimate_group_advantages(trajectories)
    batch = collate_trajectories(trajectories, advantages)
    update_student(
        student=student,
        batch=batch,
        loss_mask=batch.generated_mask,
    )
    evaluate_on_separate_tasks(student)

One tool action is generated as many tokens. The time axis of the environment is measured in tool calls; the model's likelihood is measured in tokens. The two axes have to be put in correspondence and the boundary of each input and output preserved. Those boundaries matter most in terminal-reward training, where you do not know which tokens contributed directly to the reward.

In this loop a reward of 0 has to be read differently depending on its cause. The same 0 covers both a submission that failed the requirements and an attempt whose executor crashed before any test ran. Treat the latter as a model error and noise unrelated to action quality enters the training signal.

GLM-5 Team (2026) identify samples produced by environment failures and exclude them from training (Chapter 4.1.2). A record where a normal run failed to solve a hard problem, by contrast, is the policy's own experience. Handling the two differently requires logs that record environment error and policy failure as different events.

The verifier is part of the task as well. Reward passing the tests and the model will look for behavior that satisfies that condition, and passing is the condition met, not proof that every requirement is. So a rise in reward has to be confirmed alongside a rise in actual problem-solving ability. Held-out evaluation tasks, answer-leak checks, and regression checks on original behavior are needed.

Qwen Team (2026) report cases in Qwen3-Coder-Next where the reward was circumvented by reaching the reference commit through a repository's external connections or its commit history (Chapter 4.2.4, Reward Shaping). The reward went up, but the intended code-editing ability may not have improved. The experimental conditions must include not only the grading rules used for training but the paths that reach those rules.

These behaviors exploit the gap between the intended task and the reward rule. As long as the answer sits inside the environment and a path to it is open, the room for the policy to find and use that path remains. So when a reward curve rises, the thing to check alongside it is "what did it rise through?"

A search agent has the same structure. The call arguments are the query, the observations are the search results, and the final artifact is an answer with its evidence. Search-R1 applies PPO and GRPO while excluding the search results from the loss. That experiment, though, is a result for a particular model and QA environment, so which method wins in search has to be re-established for coding or long-horizon work.

Once collection and grading are accurate, the next problem is execution time. Even on the same task, a path that finishes immediately and a path that re-checks several times differ in length. As that gap grows, synchronous collection, which waits until every attempt has finished, can become the bottleneck.

In synchronous training the update begins only after the last rollout of a batch finishes. Workers that finished earlier wait in the meantime. Overlapping generation and training reduces that wait, but records made by a slow worker under older weights end up training the current policy.

Updating early from finished records buys throughput at the price of a gap between the collecting policy and the training policy. The off-policy problem from earlier reappears here, produced not by a choice of algorithm but by the system's execution order.

Espeholt et al. (2018) separated actors from the learner in IMPALA and corrected the resulting policy difference with V-trace. Long LLM rollouts create a similar mismatch, but the correction used there is usually token-level PPO clipping rather than V-trace.

GLM-5 Team (2026) use TITO, which passes token records through unchanged, together with a policy-lag filter that drops stale samples (Chapter 4.1.2). TITO keeps the collected action and the training target in agreement; the lag filter handles the difference between the collecting policy and the current one. Preserving exact token IDs leaves that difference untouched, which is why both are there.

The experiment above shows which samples survive as the allowed lag changes. Lowering the threshold uses less stale experience but can raise the cost of discards. The right value depends on the variance in execution time and on how fast the policy is changing.

Even at the same policy version, different sampling rules can misalign the distributions. DeepSeek-AI (2025) describe a Keep Sampling Mask for the sampling support restricted by top-p and top-k, and Off-Policy Sequence Masking, which excludes negative sequences whose policy gap is large (Chapter 3.1). Both are called masks, but unlike the mask that excludes environment observations from the policy target, what these two handle is the mismatch between the collecting and the training distribution.

The support condition seen in the earlier experiment returns here as a real systems problem. Actions cut away by top-p create regions with zero collection probability. Keep Sampling Mask applies the truncation mask used during collection to the current policy as well, aligning the two policies' action subspaces; Off-Policy Sequence Masking excludes the loss from negative sequences with a large policy gap. Both differ from PPO's min-clipping, and methods that exclude samples, such as the latter, introduce a bias–variance tradeoff.

Comparing training throughput requires knowing generation lengths and tool execution times, not just sample counts. Evaluating a finished policy takes the same care: fix the compute allowed per attempt, and improvement bought with more compute will not be mixed up with improvement bought by training.

Compute at inference time can be allocated to reasoning within one response, to interaction with external tools, and to sampling several candidates. Even with the same weights, the success rate and the cost change with this budget. Reasoning effort is a control that steers this amount of work, and evaluation should measure the actual token count, number of calls and latency alongside it.

OpenAI's Reasoning documentation describes controlling the amount of reasoning, and Anthropic's "How effort works" explains that effort can affect responses and tool calls, not only reasoning. Gemini's Thinking documentation likewise distinguishes which controls each model supports. A "high" on one model points at a different compute budget than a "high" on another, so the shared label is not a shared condition. These documents were checked on September 8, 2026, and the supported settings may change.

The deployment budget also has to match the conditions of the training data. Allow very few calls to a student trained on long teacher paths and it cannot run the procedure it learned to the end. Conversely, always allowing long reasoning and multiple candidates for a simple task raises the cost even when the pass/fail outcome is identical.

\[J_{\mathrm{cost}}=\mathbb E\!\left[R_{\mathrm{success}}-\lambda_{\mathrm{tok}}C_{\mathrm{tokens}}-\lambda_{\mathrm{tool}}C_{\mathrm{tools}}\right]\]

This expression is a teaching objective for the case where cost is also counted, not any vendor's actual reward function. Penalize tokens and tool calls heavily and even necessary verification may be skipped. Measuring success rate and cost separately, and then deciding what trade-off is wanted, is easier to interpret.

Compare policies by their success rate at the same reasoning, tool and candidate budgets, or compare success-rate–cost curves across budgets. A reward number from a single setting carries the contribution of ability and the contribution of compute folded together.

Bundle the choices made so far and you have one training setup. Where to start the policy, what experience to collect, what signal to update on, and what execution budget to evaluate under all have to be decided together. The names REINFORCE, PPO and GRPO fix just one of those, the update rule.

What this table compares is costs and assumptions, not algorithm names. Which configuration wins depends on whether value prediction is hard, whether collecting several attempts on the same problem is expensive, and whether a good teacher is reachable. Any conclusion about which method is better has to come from an experiment with matched initial checkpoints and evaluation conditions.

Tasks whose answer can be verified from execution results and tasks whose quality has to be judged against a separate evaluation criterion call for different reward designs. In either case data diversity and reward reliability are separate axes and are designed apart.

The Kimi K2 report describes a data generation process that diversifies domains, tools and agent configurations (Agentic Capabilities), and uses rubric-based model judgment for tasks that are hard to grade by rule (General Reinforcement Learning). The former is a choice about what experience you obtain, the latter about how that experience is evaluated.

Unlike a code execution result, a rubric judge's score passes through one more model judgment, and that judgment's errors go straight into the reward. That is why widening task coverage comes with a check of how well the new grader reflects actual success.

Once the training setup is fixed, set the order of checks before writing any of it. Confirm first that the environment reproduces, that the initial policy emits valid calls, and that records line up with the loss. Only after those conditions hold can differences between algorithms or collection schemes be interpreted.

When starting experiments, fixing the earliest broken condition first makes the cause easier to narrow. Change the reward function while the executor is unstable, or compare PPO and GRPO while log probabilities are misaligned, and you will not know what produced the improvement. The items below turn the preceding discussion into an actual order of checks.

Passing these checks tells you the trainer behaves as intended, and no more than that. Correct execution leaves the research hypothesis untouched, so the next step is a comparison that separates which ability improved and why.

Suppose, for example, that the success rate on code-repair tasks rose after RL. The model may have gotten better at finding the cause of errors, or it may have come to run more tests. "Coding performance improved" does not say which. Narrow the research question to "is a policy trained with RL more likely than one that only imitated successful paths to detect and recover from an error after a wrong first patch?" and the necessary comparison becomes clear. What follows is an example design for that experiment, not a report of one that was run.

Comparison arms and data splits

Compare the base instruct policy, a policy SFT'd on verified successful trajectories, and an RL policy started from that same SFT checkpoint. Where possible, add a rejection-sampling SFT arm that imitates extra successful samples under the same collection budget as RL. That is what splits the gain from collecting more data off from the gain of a reward-weighted update. Include the cost of generating teacher data and of failed or discarded rollouts.

Split train, validation and test by coding-test problem. Changing only the test inputs of the same problem is not a generalization test on new problems. Copies that reword the problem statement or only rename variables in the code belong in the same bucket. Use validation for settings and checkpoint selection, and evaluate on test after selection is done. Extending to real repository problems adds repository-level splits and pretraining-exposure checks.

Measuring recovery only on naturally failing runs can distort the comparison, because each policy fails on different tasks. A diagnostic that starts from the same wrong patch reduces this selection problem, but it differs from the natural usage distribution. So report the overall task success rate and the diagnostic results together. The question of supervision on states the student visited also connects to the concern behind DAgger (Ross et al., 2011), though GKD and DAgger differ both in supervision format and in concrete procedure.

How much training data is needed

The "10,000 examples" a paper mentions can be different quantities. Distillation usually counts curated question–response pairs or successful trajectories, while online RL often counts the task list from which rollouts are generated repeatedly. To compare actual cost, look at the number of unique tasks, the total number of attempts generated, the number of attempts used for training after filtering, and the number of generated tokens together.

It is hard to average a single "typically needed count" out of these cases. An experiment that teaches a specific behavior with a few thousand curated demonstrations and a system that trains a general-purpose policy on hundreds of thousands of tasks have different goals. In particular, s1's 1,000 and SWE-smith's 5,016 are the size of the final training data. The cost of producing candidates and discarding failures is larger than that. The numbers above were checked against the version stated in each paper (September 8, 2026).

Turning a task count into a rollout budget

Suppose each collection draws \(B\) tasks and makes \(G\) rollouts per task, and this is repeated \(K\) times. With no extra collection or retries, the total number of generated trajectories is as follows. \(K\) is the number of collection iterations, not the number of optimizer steps.

\(\bar L\) is the average number of generated tokens per trajectory. For example, \(B=32\), \(G=4\), \(K=100\) generates 12,800 trajectories. At an average length of 2,000 tokens, that is about 25.6 million generated tokens. Even with only 1,000 unique tasks, drawing the same problems again can produce this much experience. The costs of tool outputs and input prefill, backpropagation, and verifier execution have to be added separately to this generated-token count.

If dynamic sampling accepts only some of the groups, the actual amount collected is larger. Approximating the group acceptance rate by a constant \(\alpha\), collecting \(KBG\) accepted rollouts requires generating roughly \(KBG/\alpha\). The same holds for distillation that keeps only successful demonstrations. If the final acceptance rate of candidates is 25%, obtaining 1,000 demonstrations takes about 4,000 attempts on average. This calculation is a budget estimate; changes in the acceptance rate during training are updated from the actual logs.

On-policy distillation is hard to express by the size of a fixed demonstration file alone. The teacher is evaluated again at every context the student newly visits, so record the student's rollout count, the number of generated positions the teacher supervised, and the input length of the teacher forward passes together. The cost of supervising one student token also depends on how the teacher's full-vocabulary distribution is obtained.

In a first experiment, treat scale itself as a comparison variable

For a first experiment on this setup, I would check the environment, the verifier and mask alignment on 50–100 tasks, and only then scale up the data for the main experiment. Distillation can be compared on nested subsets such as 500 → 1,000 → 5,000 verified trajectories, and RL on 1,000 → 5,000 → 10,000 unique training tasks. These numbers are a comparison range drawn from the public cases above; which scale actually produces the performance is what the comparison is there to settle. For long repository tasks, start at a smaller scale and measure the execution cost first.

For distillation, a comparison that matches the total training-token budget and one that trains each dataset for the same number of epochs answer different questions, so run them separately. For RL, vary the number of unique tasks while matching the total rollout budget; only then can the effect of seeing more diverse tasks be separated from the effect of simply running more. In both cases, decide the next scale by checking whether improvement saturates on validation, and do not use test for this choice.

pass@1, pass@k, and the success rate of the final selection

If one run succeeds with probability \(p\) and you make \(k\) independent attempts on the same task, the probability that at least one succeeds is \(1-(1-p)^k\). That is the probability a successful candidate exists, not the probability the system picks that candidate and delivers it to the user. If a judge selects one of several candidates for submission, the final-selection success rate has to be measured separately.

Per Chen et al. (2021), when \(c\) of the \(n\) candidates generated for a task pass, pass@k can be estimated as follows (Chapter 3.1, \(n\ge k\)). It removes from all combinations the case of drawing all \(k\) from the \(n-c\) failing candidates.

\[\widehat{\operatorname{pass@}k}=1-\frac{\binom{n-c}{k}}{\binom nk},\qquad \binom{n-c}{k}=0\ \text{if }n-c<k.\]

With \(n=10,c=2,k=3\) this is \(1-56/120\approx0.533\); plugging the sample success rate \(c/n\) straight into \(1-(1-c/n)^k\) gives 0.488 instead. For several tasks, compute the per-task estimator first and then average. Do not feed an average pass@1 into the nonlinear formula. And if the failure log of an earlier attempt was given to the next one, the independent-sampling assumption changes too, so it is better to report "the success rate of a recovery protocol with up to k attempts" directly.

Effect sizes and uncertainty

If 200 independent tasks were each evaluated once and the success rate was 50%, the standard error under a simple binomial approximation is \(\sqrt{0.5(1-0.5)/200}\approx0.035\). The rough 95% interval half-width is 6.9 percentage points. This arithmetic is why a small score gap from a single run is hard to trust. When comparing two policies on the same tasks, analyze the paired per-task difference in success rate rather than comparing two independent intervals.

Solving the same problem 100 times is not the same as solving 100 different problems. When evaluating generalization across problems, keep the repeats of a problem in one bucket and consider a bootstrap that resamples at the problem level. Record variation from the training seed and variation from evaluation sampling separately. If several training seeds were not affordable, state that the result comes from a single training run. The importance of this uncertainty and aggregation connects to the RL evaluation discussion in Agarwal et al. (2021).

The minimum reproduction materials

A model name is not enough for a reader to rerun the experiment. Record the initial and final checkpoint identifiers, the task splits, the environment image and repository commit, the tool schema and prompts, the context construction, the sampling and termination settings, each reward component, the loss reduction, the optimizer settings, the seeds, and the raw evaluation results. The rule for retrying or excluding environment errors is needed too. If a broken command the policy learned to emit and a server outage unrelated to the policy are excluded under the same rule, the comparison is distorted.

Keep success rate and cost as separate learning curves, and check the effective-group fraction, generation length, entropy, ratio and clipping rate alongside them. For example, if the reward rises while the external regression-check score stays flat, that is grounds to suspect a reward exploit. A drop in entropy alone is not a failure verdict, but if success stalls and effective groups shrink too, there is reason to investigate whether exploration is insufficient.

Paragraphs carrying a citation are this article's summaries of the originals, not direct quotations or authorized translations. Sources of algorithms, observations reported by particular systems, and this article's own derivations and experiment proposals are listed as separate items. When following a link, check its version and section numbers. On September 8, 2026 the existing public cases, the core algorithms and the newly added evaluation and shaping literature were all checked against the originals. That check covers whether the cited content is in the source as stated; it did not extend to independent reproduction or verification of the results.

Research design and further connections

• Weng, L. (2026). Harness Engineering for Self-Improvement. Lil’Log. Introduction and Harness Design Patterns: the scope of a harness, including how execution is organized, context and state management, and evaluation of results.

• Muennighoff, N., et al. (2025). s1: Simple test-time scaling. Chapter 2. The distinction between the question pool and the 1,000 curated demonstrations.
• Yang, J., et al. (2025). SWE-smith: Scaling Data for Software Engineering Agents. Chapter 4. Collection scale in unique tasks, total attempts, and final successful trajectories.
• Zeng, W., et al. (2025). SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. Chapter 2.1, Appendix B. About 8,000 training problems, and the batch and group sizes.

• Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. ICML. Reward transformations and the conditions under which the optimal policy is preserved.
• Ross, S., Gordon, G., & Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS. Supervision on the states the learner visits, and DAgger.
• Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code. Chapter 3.1. The definition and estimator of pass@k.
• Agarwal, R., et al. (2021). Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS. Uncertainty and aggregation in RL evaluation.
• METR (2024). Guidelines for capability elicitation. The definition of elicitation as modifying the scaffolding around an agent, and what minimum scaffolding consists of.
• Pan, L., et al. (2026). Natural-Language Agent Harnesses. Abstract. Summarizes a harness as the external execution system around a model that organizes a task run, and argues that burying its logic in controller code makes it hard to compare or evaluate.

Policy gradients and training

• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS. InstructGPT, connecting SFT, a preference reward model and PPO.
• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS. The relation between preference data and the KL-regularized objective.
• Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. The original source of REINFORCE.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. PPO's surrogate and experience reuse.
• Schulman, J., et al. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR. Bias and variance in advantage estimation.
• Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. The original form of GRPO.
• Liu, Z., et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. Chapter 3.1 the bias created by length and standard-deviation normalization, Chapter 3.2 removing both terms (Dr. GRPO).
• Yu, Q., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Chapter 3.1 decoupled clip widths and entropy collapse, Chapter 3.2 dynamic sampling and groups with zero advantage, Chapter 3.3 token-level loss, Chapter 3.4 reward shaping for truncated responses. A system report that releases training code and data with reproduction as its goal, and the main basis for this article's "what to log" section. Its results are confined to mathematical reasoning tasks, so carrying them over to multi-turn tool use means making the same observations again.
• Espeholt, L., et al. (2018). IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. ICML. Policy difference in an asynchronous actor–learner setup.
• Chu, T., et al. (2025). SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. A generalization comparison across rule variants, and the observation that SFT before RL stabilizes format. The experimental settings are an arithmetic card game and navigation rather than multi-turn tool use.
• Lambert, N., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. Chapter 6 Reinforcement Learning with Verifiable Rewards. The definition of verifiable rewards.

Agent execution and self-evaluation

• Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR. Interleaving reasoning and acting.
• Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS. Feedback and revision at inference time.
• Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS. Verbal feedback and episodic memory.
• Jin, B., et al. (2025). Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. PPO/GRPO in a search environment, with observation-token masking.

Distillation and public training systems

• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Chapter 2.2 R1-Zero, started with no cold start; Chapter 2.3.1 how the cold-start data was collected and its two stated benefits; Chapter 2.3.3 the composition of the 800k SFT set; Chapter 2.4 and Chapter 4.1 Table 6 comparing distillation against RL on small models. The reported results are confined to single-response reasoning on verifiable tasks; multi-turn tool use lies outside that scope.
• Agarwal, R., et al. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR. The train/inference distribution mismatch and teacher feedback on student-generated sequences. The summary in the text follows the abstract's problem setting and method.
• Qwen Team (2025). Qwen3 Technical Report. Chapter 4.5 the off-policy and on-policy stages of strong-to-weak distillation, Chapter 4.7 Table 21's GPU-time comparison, Chapter 4.3 thinking budget.
• Cao, R., et al. (2026). Qwen3-Coder-Next Technical Report. Chapter 2.2 the execution environment (MegaFlow), Chapter 4.2.4 reward hacking, Chapter 4.2.5 expert distillation.
• GLM-5 Team (2026). GLM-5: from Vibe Coding to Agentic Engineering. Chapter 3.5 cross-stage distillation and the rationale for group size 1, Chapter 4.1.2 token-in/token-out, discarding stale samples, and excluding environment failures.
• DeepSeek-AI (2025). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. Chapter 3.1 the keep sampling mask and off-policy sequence masking.
• Moonshot AI (2025). Kimi K2: Open Agentic Intelligence. Official blog. Tool and environment synthesis in "Agentic Capabilities", and the self-judging rubric reward in "General Reinforcement Learning".

Official system, tool and evaluation documentation

• OpenAI. A practical guide to building agents. "What is an agent?" and "Agent design foundations". The official guide behind the definition in the first section.
• Google Cloud. Core concepts of AI agents. "Orchestration". The official account of the goal / tool / feedback loop.
• Wiesinger, J., Marlow, P., & Vuskovic, V. (2024). Agents. Google whitepaper. The third published definition, casting an agent as goal, observation and action through tools.
• Anthropic. Building effective agents (2024), Advanced tool use (2025), Demystifying evals for AI agents (2026). Practical standards for structure, execution and evaluation.
• OpenAI (2025). Introducing Codex, Introducing deep research. Public cases of RL on real interactive tasks.
• Google (2025). Build with Gemini Deep Research. Multi-step RL for search.
• OpenAI. Function calling, Reasoning models. The call loop and effort control.
• Anthropic. Effort. Control over reasoning, output and tool use.
• Google. Gemini thinking. Per-model thinking settings.