CS329A Self-Improving AI Agents, Part 3: Robust Verification

A language model can generate a correct solution somewhere among many attempts and still return the wrong one. CS329A’s third lecture follows four papers that try to close this generation-verification gap: a learned answer verifier, human-supervised process rewards, automatically collected process labels and an ensemble of imperfect evaluators.

The progression changes where supervision comes from and where computation is spent. It does not turn a score into ground truth. A verifier can misrank candidates, a process label can reflect an inadequate rollout and an ensemble can share the same blind spot.

The recurring question is therefore operational: what evidence supports the score, how is that score used and does the final system actually deliver more correct answers? That is a stricter standard than merely adding another model called a judge.

The first paper introduces GSM8K and a verifier for multi-step grade-school mathematics. A generator produces multiple solutions for each question, and known final answers provide a way to label whether a generated response reaches the expected result.

A language-model-based verifier learns from those examples. The lecture describes a binary correctness objective together with a language-modeling objective, then a final score used to rank complete responses. At test time, the system samples candidates and returns the one with the highest verifier score.

The distinction from ordinary generator fine-tuning matters: the generator can remain general while the verifier learns a task-specific selection signal. The historical training recipe illustrates that division of work; it is not a requirement that every modern verifier use the same number of epochs or samples.

The lecturer discusses experiments with generator and verifier sizes, training-set size and candidate count. A larger generator paired with a smaller verifier worked well in the reported comparison, but identifying a good cost-quality frontier remains a research question.

Candidate count is especially revealing. The learned verifier continues helping beyond the range where majority voting had plateaued, yet its delivered accuracy eventually stops improving and can fall. The plot measures the selected answer, not merely whether a correct answer exists somewhere in the set.

With more candidates, an incorrect response may receive a deceptively high score. Increasing generation therefore changes the selection problem as well as the opportunity for success. There is no contradiction between rising coverage and declining final accuracy when the selector is imperfect.

The second paper, Let’s Verify Step by Step, distinguishes outcome supervision from process supervision. An outcome model labels the whole response; a process model receives labels on intermediate reasoning steps. Human annotations can mark a step positive, negative or neutral.

The motivation is a false positive that outcome labels can miss: invalid reasoning may happen to arrive at the expected final answer. Step-level supervision gives a more precise place to identify that error and assign credit. PRM800K refers to roughly 800,000 step labels, not that many separate problems.

The lecture illustrates the difference with a fraction problem. A sequence of correct steps can end in a simple arithmetic error. Judging each step helps locate the failure instead of treating the entire path as one undifferentiated success or failure.

The reported process model performs well on rare correct solutions and distribution shifts in the studied tasks. But comparing label counts is not automatically comparing annotation effort: a solution-level label and several step-level labels require different work.

Students also ask whether a step can look good without contributing to a valid solution, or whether a generator could skip reasoning and exploit the evaluator. The lecturer distinguishes using a fixed generator for selection from training a generator against a reward it may learn to exploit.

Combining outcome and process signals may help, but introduces thresholds and further tuning. Human annotations can express what a sensible transition looks like; they do not make every learned score infallible. The design must still examine which errors its feedback rewards or overlooks.

Math-Shepherd asks whether process labels can be collected without asking humans to annotate every step. From an intermediate state, it samples several continuations and checks their final answers against known outcomes.

A hard label is positive if any continuation reaches the correct answer. A soft label records the fraction that succeed. With three continuations and two successes, the illustration gives a positive hard label and a soft value of two-thirds.

40:40View full image

This estimates the potential to reach a correct ending under the sampled continuations. It is not a logical proof that the current step is valid. The approach reduces step-annotation work while still depending on known final answers and on what the rollout policy can discover.

The classroom discussion identifies two opposite errors. A promising but unusual path may receive zero because a small rollout budget never discovers its successful continuation. A wrong intermediate step may receive credit if later reasoning recovers or reaches the right answer by accident.

More samples can change those estimates, but do not guarantee the errors vanish. For very hard problems, every sampled continuation may fail, leaving little useful signal. These are limitations of the label-construction method, not problems solved merely by calling the labels automatic.

The resulting process reward model can be used to select answers at inference time or to train a generator with reinforcement learning. Both uses depend on its quality. Improvements reported in the lecture are experimental results, and the speakers also note plateauing gains and unresolved comparisons under equal compute.

Weaver takes another route: combine existing verifiers, including outcome models, process models and language-model judges. “Weak” means imperfect, not that the researchers intentionally chose bad evaluators.

The procedure scores candidates, normalizes outputs, filters low-quality verifiers using limited labeled evidence and estimates weights for aggregation. A simple average can help, but adding more evaluators is not guaranteed to improve monotonically.

56:20View full image

The lecture’s weak-supervision discussion relies on an independence assumption about verifier signals. Identical judges that agree on everything add little new information. Diversity is useful only to the extent that the signals contribute information about correctness, not because a larger head count constitutes independent proof.

The Weaver results discussed near the end measure the selected solution’s accuracy, not merely candidate coverage. That makes them a different kind of evidence from a graph showing that some correct candidate exists under an oracle selector.

Some plots use an “unsupervised” label while the lecture describes a variant using a small labeled subset for filtering. A faithful account should state that dependency rather than turn the method into a claim of zero labeled information.

Likewise, comparisons between smaller open models and larger or proprietary models belong to the specific historical benchmarks and budgets. They do not establish complete equivalence between model classes on every task, and this article does not claim to have reproduced the experiments.

An ensemble requires many verifier evaluations for every candidate. The lecture therefore discusses distilling its signal into a smaller model, reducing online computation after a separate training investment.

Reported retention of a large fraction of the teacher’s accuracy is relative to that teacher and experiment. It is not the same as saying every distilled answer is correct with that absolute probability, nor a universal compute-saving guarantee for other workloads.

The final questions return to a broader goal: make the first answer better without discarding all diversity. Sharpening a model toward one response may improve efficiency while reducing alternative paths worth exploring. Robust verification helps manage that tension, but does not make the future balance a settled fact.