CS329A Self-Improving AI Agents, Part 7: Self-Improvement and Deep Research Agents

CS329A’s seventh lecture connects two meanings of search. AlphaCode searches a large space of generated programs; Search-o1 retrieves external knowledge while a reasoning model works through a question. In both cases, finding more material is only part of the problem.

A coding system must decide which candidates deserve its limited submissions. A research system must decide which passages actually help the current reasoning step. Poor selection can waste a good generator or drown useful evidence in noise.

The shared lesson is therefore curation under constraints. The signals and failure modes remain different: executable code tests check particular behavior, while a retrieved document still needs interpretation and source evaluation.

Competitive programming asks a system to understand a problem description, choose an algorithm and produce a program satisfying an input-output contract. That is a different task from completing a line or filling a short function whose intended approach is already clear.

The historical AlphaCode pipeline trains a code model, samples many candidate programs and filters them with examples from the problem statement. It then groups candidates by behavior on generated inputs before choosing representatives for evaluation.

Behavioral grouping is useful because many syntactically different programs may behave alike on the probes. It is not a proof of full semantic equivalence. Passing a few example tests is also only an initial filter; the hidden evaluation remains the stronger check on the submitted program.

The lecture distinguishes pass-at-k coverage from a limited-submission measure such as 10-at-k. The first asks whether the k generated candidates contain a success; the second also tests whether filtering and clustering can choose useful submissions from that pool.

A system generating a million programs cannot simply submit all of them when the evaluation allows ten attempts. Repeated near-duplicates waste those slots, making diversity among selected candidates important.

This explains why a selection bottleneck can persist even as generation coverage rises. The final result belongs to the whole pipeline, including what it rejects and which representative it keeps, rather than to the sampler alone.

Students ask whether a favorable trend could be extended to vastly more samples. The lecturer treats that as conditional: additional samples must continue producing useful diversity, and their cost must remain acceptable.

If more computation generates the same approaches again, the pool may grow without adding much capability. A stronger base model can sometimes reach the same result with fewer candidates, changing the economics more directly than an enormous sample-count increase.

The lecture also notes that training loss is an imperfect proxy for solving competitive problems and that some algorithmic categories remain difficult. An observed scaling curve is a result to study, not a license to promise unlimited performance by extrapolation.

AlphaCode 2 starts from an existing Gemini Pro model and fine-tunes it for the task. The lecture describes a family of variants intended to produce diverse solutions, improved data and a learned scoring model for candidate correctness.

The sampling pipeline compiles and tests candidates, filters failures, clusters survivors and uses scores to choose representatives. A learned ranker adds information beyond a heuristic grouping, but remains an approximation rather than an oracle.

29:20View full image

The reported gains therefore involve a stronger base, diversity, data and selection together. They cannot be attributed to only one component from the aggregate comparison. The high rejection rate also makes clear that a strong result can still require substantial wasted generation and evaluation work.

The lecture describes AlphaCode 2 as competitive with strong human contestants in the evaluated contests. That is a meaningful historical result, but it is not equivalent to broad superiority at all software engineering or every programming task.

Per-contest outcomes vary, and standout results should not be substituted for the overall comparison. The class considers adaptive budgets, refinement and decomposition as possible extensions; those discussion ideas are distinct from what the cited system already demonstrated.

A useful transfer question is which parts of the pipeline apply elsewhere: diversity, limited submission selection, execution evidence or a learned scorer. The answer depends on the new task’s feedback, not on importing a contest percentile as a general intelligence score.

Search-o1 addresses knowledge gaps during complex reasoning. A model may reason at length while lacking a current fact, an unfamiliar term or information needed only after an earlier step has been completed.

A one-time retrieval baseline fetches documents at the beginning. That can help, but later subquestions may require different evidence. The lecture contrasts that baseline with a system that generates new search queries as the reasoning unfolds.

This is a distinction between the methods compared, not a definition that all retrieval-augmented systems must search only once. The important design choice is whether the system can notice an information need and obtain relevant evidence at the point where it matters.

Simply appending many raw documents can make the prompt longer without making the needed fact easier to use. The lecture’s chemistry example contrasts guessing, inserting lengthy retrieval and extracting relevant information before continuing.

Search-o1’s reason-in-documents module considers the query and current reasoning, analyzes retrieved material and returns a more focused piece of knowledge. That creates a nested process inside the retrieval tool rather than treating retrieval as a raw text dump.

56:20View full image

The extraction itself can omit or distort information, so relevance and source fidelity remain evaluation questions. The mechanism aims to reduce noise; it does not establish that every summary is correct or every retrieval result authoritative.

The lecture describes search-triggering tokens and a buffer that keeps the current question and reasoning context while retrieval happens. Returned information then becomes part of the next step rather than replacing the task with an unrelated document summary.

Iterative queries allow another search when the first result leaves a gap. That flexibility is useful in multi-hop questions, where the next entity or relation may only become apparent after earlier evidence arrives.

A visible uncertainty word is not a calibrated detector of every missing fact. The final discussion returns to overconfidence: a model can sound certain and still be wrong. The workflow needs evidence checks rather than relying solely on how assured the reasoning sounds.

The reported experiments show benefits from more documents when the method can extract relevant information. That condition matters. Retrieving additional noise or missing the needed source does not become useful merely because a document counter increased.

The lecturer also narrows comparisons with human specialists to the evaluated class of questions, rather than claiming broad superiority over experts. Retrieval precision and recall are raised in discussion but not supplied as a complete analysis in the lecture.

Historical benchmark gains therefore support studying the mechanism, not a current state-of-the-art claim or a guarantee about every scientific domain. The article preserves that scope instead of converting limited comparisons into a general research credential.

Across the coding and research examples, the system’s useful output depends on selecting and organizing information. Program clustering avoids wasting submission slots; document analysis avoids wasting context on irrelevant text.

Those operations also introduce failure points. A selector can discard the rare correct program, and an extractor can omit the decisive passage. Their quality must be measured with the generator and retriever, not assumed from the strength of the base model.

The lecture briefly contrasts prompting-based Search-o1 with reinforcement-learning-based search work without covering the latter in detail. Its durable question is broader than any name: after the system finds more possibilities or references, what justifies the answer it finally chooses?