CS329A Self-Improving AI Agents, Part 6: Train-Time Scaling and Scaling RL

The sixth CS329A lecture moves from spending computation on an answer to changing the model that produces future answers. Its simplest loop is generate, evaluate, filter and train again. STaR, DeepSeekMath’s GRPO and DAPO show different ways to build and stabilize that loop.

The distinction from inference scaling remains important. Sampling more candidates leaves weights fixed; fine-tuning or reinforcement learning updates them. A gain in one setting does not determine the best allocation of all training and serving resources.

The lecturer emphasizes that implementation details matter. A reward must contain useful differences, the base model must reach some successful behavior and training must preserve enough exploration to improve. More compute does not substitute for those conditions.

STaR begins with a small set of examples containing a question, a rationale and an answer. These demonstrations prompt the model to produce rationales for a larger collection of questions whose final answers are already known.

The method keeps successful outputs and fine-tunes on the resulting question-rationale-answer examples. It repeats the process so an improved model can generate another round of training material.

This is a bootstrapping procedure, not data created without an external reference. The known answer supplies the filter. It also means the initial model must solve at least some relevant problems; a loop with no successful examples has little positive material to learn from.

For a question the model fails, STaR supplies the correct answer and asks it to construct a rationale leading there. The resulting example can then be used without exposing that answer hint in the normal input, as if the model had solved the problem directly.

This expands the training set beyond questions already answered correctly on the first attempt. But it depends on the model being able to generate a valid explanation when given the answer. A difficult question can remain beyond reach even with that hint.

The original procedure discussed in class does not independently verify every generated rationale step. A correct final answer is a proxy for process quality. Students explicitly identify the danger of learning a plausible post-hoc explanation that contains invalid reasoning.

The historical experiments include commonsense reasoning, mathematics and synthetic arithmetic. The lecture describes useful gains but also mixed results from rationalization, comparable benefits from good direct fine-tuning in some cases and plateauing improvement across iterations.

Prompt style affects the rationales collected, so the data can inherit the demonstrations’ format and biases. Qualitative human preference for some explanations is evidence about those examples, not proof that every chain is valid.

A stronger teacher offers a different route: distill its outputs into a smaller model. The class distinguishes that practical option from studying how a model can bootstrap from its own behavior. Neither mechanism makes the quality of training examples irrelevant.

The DeepSeekMath discussion begins with the base model and the data. The lecturer describes starting from a code-oriented model and curating mathematical material from the web, rather than assuming that a reinforcement-learning algorithm can supply missing domain competence by itself.

The argument is about the reported setup: data coverage and prior coding ability helped establish a model that could produce useful mathematical responses. It should not become a universal claim that academic papers are useless training data or that any coding model is automatically a strong mathematician.

This preparation connects back to the bootstrapping requirement. A reward can favor better candidates only when the system generates enough behavior that the signal can distinguish. Data work and optimization are complementary parts of the story.

GRPO reduces dependence on a separately learned critic for its baseline. For a question, it samples a group of responses and uses their reward distribution to form relative advantages.

The lecture’s intuitive calculation subtracts the group mean from a response reward and scales by the group’s spread. A response is evaluated relative to its peers, rather than only through an isolated absolute label. This can reduce memory demands associated with critic-based training.

That scalar is an explanation of the baseline, not the full algorithm. Policy constraints, clipping and other implementation details still matter. The number of model copies in a particular training diagram should not be mistaken for a universal memory formula for every implementation.

If every response in a group is correct, or every response is wrong, the within-group comparison supplies little useful advantage signal. This is different from saying that easy tasks never matter or that an all-wrong problem is fundamentally impossible.

The discussion of regression raises another separate question: optimizing a new benchmark can change other behavior. Keeping the policy near a reference can constrain drift, but it is not evidence that no unrelated capability will regress.

The lecturer also contrasts more consistent success with expanded coverage of previously unsolved problems. Improving majority-vote performance can reflect a changed distribution of answers. It should not automatically be advertised as proof of a wholly new general capability.

DAPO is introduced through unstable long-reasoning training: entropy can collapse, a policy can become too confident and response lengths can become difficult to control. A short loss curve alone does not diagnose all of these behaviors.

One change uses asymmetric clipping to permit more upward movement for low-probability choices. The lecture connects this to maintaining exploration instead of prematurely narrowing the policy’s output distribution.

Entropy is still a proxy. Very low entropy can signal collapsed exploration, but more entropy is not always better and does not establish correctness. The training process must watch both the variety of behavior and the quality of the resulting solutions.

Dynamic sampling oversamples and excludes groups in which all responses have the same success outcome. It retains groups with informative differences, aiming to preserve an effective batch for the relative-advantage update.

56:10View full image

It does not discard every individual response with reward zero or one. Those are precisely the outcomes that can create a useful contrast within a mixed group. The unit of filtering is important to understanding the method.

Sample counts in the lecture are illustrative or tied to the paper’s setup. Filtering an all-wrong group does not teach the model how to solve it by itself. It makes the current update more informative while leaving base capability and data selection as separate constraints.

Long responses make loss aggregation consequential. A token-level aggregation weights variable-length responses differently from giving every completed sample the same overall contribution. That is not identical to a rule that punishes all long reasoning.

The lecture separately discusses overlong and truncated responses. An unfinished trace can introduce a noisy signal if treated carelessly, so filtering and gradual overlength penalties address another part of the training problem.

The reported ablations combine several changes. Their historical score improvements are not universal additive gains that can be promised on a new run. The broader lesson is to monitor response length, entropy and reward variation together, then connect those observations to the actual update being applied.

The lecture contrasts the accessibility of rationale bootstrapping, the memory benefits of group-relative optimization and the engineering demands of long-reasoning RL. Good supervised data can be a practical route; RL requires infrastructure and a reward signal worth optimizing.

Its closing questions remain open: how much improvement reflects more frequent existing behavior, how to learn from failures and how to stop a strong policy exploiting a weak reward. The speakers’ interpretation of particular results should not become a universal theorem that RL can never add capability.

A useful assessment asks what improved, on which held-out tasks, at what cost and under which verification conditions. More verifiers cannot automatically repair data leakage or missing training signal. The loop earns confidence through those checks, rather than through the label “self-improving.”