CS329A Self-Improving AI Agents, Part 9: Future Research Areas

The final lecture returns to the course’s definition of an agent: a goal-directed system that interacts with an environment, receives feedback and changes what it does next. Its research agenda asks where that loop still depends on fragile data, weak verification or expensive computation.

The speakers organize the discussion around diverse reasoning, checking the checker, generating a useful curriculum and improving inference efficiency. They also ask how experience might become learning more continuously than an offline data-collection and fine-tuning cycle.

These are research directions with examples of promising work, not a declaration that general self-improvement has been solved. The lecture repeatedly relies on particular environments and evaluation signals, especially in code and mathematics.

A single model generating its own fine-tuning data can produce similar responses repeatedly. The lecture discusses specialized generation agents and critic agents as one way to broaden the initial solutions and refine them through debate.

Each round shares or summarizes other responses, critiques the updated answers and generates another attempt. The described training filters outputs using a majority-based signal and fine-tunes the participating models. Reported experiments track both performance and response dissimilarity.

Neither majority agreement nor embedding distance establishes truth. Several variants of the same base model can share errors, and extra generations cost computation. The proposal’s value is the attempt to maintain useful diversity across iterations, not the assumption that more agents make an answer independently verified.

The proof-verification discussion distinguishes matching a final answer from checking whether a derivation is rigorous. A language-model judge may accept an invalid proof, but it can also criticize a valid step for an issue that is not really there.

The described DeepSeekMath-V2 architecture adds meta-verification: review the verifier’s analysis, ask whether the identified issue exists and whether the assigned score follows from it. Expert annotations of critique quality help establish that signal.

18:20View full image

This checks a different object from the original proof. It can improve the incentive to give a justified evaluation, but remains learned verification rather than a formal proof checker. Adding a meta layer does not end all uncertainty or prove that an infinite chain of judges is unnecessary in every domain.

The task-generation discussion, identified later as the Absolute Zero work, asks whether one model can both propose tasks and solve them. Its coding environment gives programs, inputs and outputs a concrete relationship that can be checked by execution.

The proposer is rewarded for challenges near the solver’s current reach: not trivial, but not so hard that every attempt fails. A buffer of previous tasks helps condition future proposals and encourage diversity. As the solver improves, the curriculum can change.

“No human-curated prompts” has a specific scope. It does not mean no pretraining, no human-designed environment or no prior knowledge. The environment’s validation and reward construction remain central to whether the generated curriculum teaches something useful.

The lecture describes integrity, execution and repeatability checks before a proposed coding task enters training. These help reject malformed tasks and make the resulting reward more dependable.

Yet an environment can be internally consistent without being a good proxy for the real task one ultimately cares about. The closing questions explicitly return to that distinction: creating a simulation is different from establishing that it represents real-world interaction usefully.

Reported transfer from self-proposed code tasks to mathematical benchmarks is promising evidence in the studied setting. It does not establish transfer to every subjective, physical or professional domain, where feedback may be slower or correctness less sharply defined.

The speakers use chip simulation and laboratory experiments to illustrate costly feedback. An outcome may be observable, yet take days to obtain, making thousands of RL iterations impractical.

A surrogate reward model can predict the outcome of expensive work from previously collected data. That moves cost offline and speeds the loop, but adds prediction error and the possibility that optimization exploits the surrogate outside its reliable region.

Subjective work creates a related but different challenge: a reward may only imperfectly represent what people value. Breaking a large problem into verifiable pieces can help, while leaving the composition and final objective to be checked. None of these strategies makes a difficult feedback problem disappear.

The efficiency section considers whether some demand currently served in the cloud can move to local models and hardware. The motivation combines improving smaller models, growing device memory and many queries that may not require the largest available system.

The relevant word is some. A workload category such as writing or practical guidance contains tasks of very different difficulty. The reported study evaluates selected query sets and reasoning tasks; it does not prove that every request in those categories is easy.

Likewise, fitting a model in memory is not the whole serving problem. Quantization, latency, power and answer quality affect whether a local deployment is useful. The lecture presents a direction to measure, not a blanket recommendation that all inference belongs on a laptop.

The speakers define intelligence per watt as average task accuracy divided by average power draw in their evaluation. It deliberately combines a capability measure with a hardware-efficiency measure.

47:40View full image

That is not a universal unit of intelligence. It also differs from energy per completed query, because power and total energy are not identical and time matters. The lecture includes latency and other serving measurements alongside the ratio.

The study’s local-model definition, hardware comparisons and query mix set the scope of its findings. A lower-power device is not automatically more efficient on the combined metric, nor does a reported improvement across a particular two-year period establish the same future rate for every model or accelerator.

A possible future combines local and cloud inference, routing according to task complexity and resource needs. Model architectures, kernels and serving engines can then be optimized for the hardware and workloads they actually encounter.

That route needs evidence about which tasks a local model can handle and when a larger or remote system is worth using. The lecture frames hybrid serving as a research direction, not an already delivered mechanism that solves routing errors.

Test-time scaling makes the systems problem richer: parallel candidates, revisions, tool calls and shared context create different throughput and latency requirements from one isolated chatbot response. Efficiency has to be measured over that complete procedure.

The continual-learning discussion questions today’s separation between collecting experiences and later fine-tuning. The speakers ask whether successful and failed attempts could improve a model more naturally and continuously.

A memory database is one way to preserve information without changing weights. A model still needs to retrieve and reason over that information; storing a record does not ensure it has learned the skill required to use it in a new domain.

The lecture imagines perfect access to an unlimited context, then explicitly says that capability does not exist. It also mentions activation or cache-based approaches alongside weight updates. These are alternative research routes, not proof that either a larger memory store or continual fine-tuning is always sufficient.

The lecture closes with a reminder that methods evolve quickly while the underlying questions persist. What produces useful diversity? What verifies an evaluation? Which tasks create a learnable curriculum? What does a result cost to serve?

Each proposal should be assessed at the boundary where it was tested. A code environment can provide precise execution feedback; an informal proof critic provides a learned judgment; an efficiency ratio depends on its workload and measurement procedure.

The course’s final discussion opens these questions rather than certifying that the next system will improve without limits. A productive next experiment states the assumption it wants to test and the evidence that would show the proposed loop has failed.