CS329A Self-Improving AI Agents, Part 2: Test-Time Compute Scaling

The second CS329A lecture asks how to improve a model’s useful output while leaving its parameters unchanged. Pretraining and fine-tuning are separate investments; inference-time scaling changes what happens when the trained model is used.

Repeated generation is the first example, but the lecture develops a wider design space: revise a candidate, score an intermediate step, explore several branches, combine responses or search for a whole architecture of calls. The engineering objective is quality under a budget, not the largest possible token count.

This also changes how a result should be described. A system can deliver one final answer after many internal calls. Its final-answer accuracy is meaningful, but does not mean the computation was a single inexpensive model invocation.

Large Language Monkeys repeatedly presents the same problem to a model and asks for different answers. In the studies discussed, candidate coverage improves as more samples are generated, including on software-engineering tasks. Coverage means at least one candidate solves the problem.

The system still needs a verifier or selector to return that candidate. A small model’s large-sample coverage beating another model’s single attempt is therefore a specific comparison, not proof that the smaller model is uniformly superior as a deployed service.

The lecture keeps returning to this distinction because the rarest successes are especially important. Among thousands of candidates for a hard problem, only a few may be correct. A method that simply picks the common answer can miss exactly the unusual solution that made coverage increase.

For a problem with fixed success probability p and independent samples, the chance of at least one success after k attempts is 1 − (1 − p)^k. This describes a single problem under those assumptions.

A benchmark combines problems with different success probabilities. The lecture explains its observed aggregate scaling behavior through a long tail of difficult problems: many are solved readily, while others have very small single-attempt success probabilities and need many more samples.

The empirical curve is useful for estimating resources in the studied setting. It is not a promise that every model, domain or selector follows one universal fitted equation. Nor does a coverage fit remove the need to measure how well the actual selection mechanism works.

The lecture compares oracle-like selection with majority voting and learned reward-based ranking. Majority voting can plateau while candidate coverage continues improving, because frequency is not the same as correctness. Learned rankers can narrow the gap, yet still miss difficult cases.

16:50View full image

Formal proof checking and code tests provide stronger signals in suitable domains. For a CUDA implementation of a PyTorch operation, comparing outputs offers a concrete correctness check on tested inputs. It should not be inflated into a proof of equivalence for every possible input.

A student explicitly asks whether passing tests can conceal an incorrect patch. The response acknowledges that verifier quality and test coverage matter. Calling a signal a verifier does not make its blind spots disappear.

Parallel sampling starts several candidates independently. Sequential revision instead lets a model revisit an initial approach, improve it or consider the problem from another angle. A system can also combine the two, revising multiple candidates before selecting among them.

Neither pattern is automatically best. Revision may help when an initial approach contains useful material; exploration may matter when the first direction is wrong. The lecture presents these as choices to evaluate against a task and budget, not as a universal ratio of sequential to parallel tokens.

Reasoning models may acquire revision behavior through training, while a surrounding workflow can also ask for it through prompts. In either case, a model’s confidence that it is finished is not itself proof that the revised answer is correct.

An outcome reward model scores the completed response. A process reward model scores intermediate steps, which may be meaningful reasoning chunks rather than individual tokens. Both are learned evaluators, with quality that can change across domains.

The beam-search illustration generates several continuations, keeps the higher-scored branches and expands them again. A process score therefore changes where computation is spent, rather than only choosing among fully completed answers at the end.

This can concentrate a budget on promising paths, but it also makes the evaluator influential early. A poor score may discard a useful branch before it is finished. In-domain training helps in the studies discussed; generalization to a new task still needs examination.

The discussed study groups problems by single-attempt success under a given model. Difficulty is therefore operational and model-relative, not an intrinsic label that is identical for all systems.

For easier and medium problems, additional inference compute can look attractive compared with increasing pretraining in the presented experiments. For the hardest problems, stronger models can still retain an advantage within the investigated budgets. The lecture does not establish that unlimited sampling must close every capability gap.

There is also an accounting distinction: pretraining is a shared investment reused across requests, whereas inference spending recurs on each problem. A plotted trade-off is not a simple one-to-one token-price comparison. Feasibility matters too, since many teams can buy extra inference but cannot run a new frontier pretraining job.

Archon takes target benchmarks, an inference-call budget, available models and a collection of operations as inputs. Architecture search then explores how to connect those operations to improve quality at a given cost.

The operations include generation, criticism, ranking, verification and fusion. Fusion asks a model to synthesize a new response while seeing several candidates. These are prompted operations in the described framework; searching their arrangement is different from updating model weights.

The search itself costs computation. The researchers restrict its space, for example beginning with generation, placing criticism before ranking or fusion and pairing test generation with evaluation. Bayesian optimization then explores configurations more efficiently than an unrestricted enumeration. Those constraints reflect experiments in that project, not rules every workflow must follow.

One experiment compares random selection, model ranking, oracle selection and fusion. The lecturer reports that fusion can outperform choosing the best existing candidate. That is possible because fusion generates a new answer, which may incorporate useful parts of several responses.

An oracle selector is limited to the candidates it was given. A synthesizer can move outside that set. The comparison therefore does not mean a fallible model has somehow become better than perfect verification of the same fixed choices.

The lecturer does not identify the exact benchmark while describing this plot, so it should not be supplied by guesswork. The appropriate conclusion is the mechanism and the reported experimental observation, not a universal claim that mixing more answers always improves accuracy.

The final examples include generating unit tests and asking a model to assess answers against them. That assessment is not necessarily execution. For balanced brackets, a stated property can help evaluate a solution, but a necessary condition such as even length is not a complete correctness test.

The lecture reports improvements from deeper combinations of critics and fusers in its evaluated tasks, including architectures built from multiple open models. Those are historical experimental comparisons, not a current leaderboard or a guarantee that adding another layer is worth its cost.

The useful evaluation unit is the whole procedure: what it generates, what it discards, how it combines evidence, what it returns and how much it spends. A single final answer can be better than a base model’s first attempt while still depending on an expensive and imperfect chain of internal judgments.