CS329A Self-Improving AI Agents, Part 1: Course Overview

The opening lecture of Stanford’s CS329A connects three developments that are easy to conflate: larger pretrained language models, more capable reasoning at inference time and agents that act in an environment. Its central question is how a system obtains useful feedback and turns that feedback into better behavior.

A model that produces a good answer once is not automatically an agent. An agent needs a goal, actions, observations, a way to correct course and a decision about when to stop or report that it cannot finish. The lecture treats these as capabilities to study rather than a label that makes any workflow autonomous.

The overview is valuable precisely because it leaves difficult questions open. Generating candidates can be easier than recognizing the right one; a learned judge can be imperfect; and a promising improvement loop does not prove unlimited progress. The course builds its research agenda around those gaps.

The lecturers begin with the historical scaling picture: increasing compute, training data and parameter count reduced test loss in the presented language-model studies. Larger models also showed useful behaviors across tasks, making it easier to prototype without a separate fine-tuning run for every task.

Zero-shot prompting supplies a task instruction; few-shot prompting adds examples of the desired input-output pattern. The lecture uses translation as an accessible illustration. Those examples live in the prompt: they do not themselves update model weights.

The historical charts should be read as the lecture’s context. Some model sizes are estimates and the speakers themselves describe a slide as dated. The argument does not require treating an estimated proprietary parameter count as an official disclosure or a past size threshold as a universal rule for current models.

The chain-of-thought example asks how many tennis balls Roger has after starting with five and buying two cans of three. An answer-only demonstration gives eleven. A worked demonstration also supplies the intermediate relation: two times three is six, then five plus six is eleven.

The point is that a model can use the demonstrated process when solving another problem. The lecturers discuss older experiments in which the benefit depended on model scale, then connect that history to reasoning models trained to analyze, decompose and revisit a problem.

They later clarify that discovery and deliberate training coexist. Early prompting exposed useful reasoning behavior, while reasoning models are subsequently trained to use such behavior more effectively. This is not evidence that every visible explanation faithfully reveals an internal computation, or that a phrase asking for more thought guarantees correctness.

The lecture separates next-token pretraining from later stages that shape instruction following and preferences. Curated fine-tuning data and instruction-answer pairs teach useful response patterns; the quality and breadth of those examples affect what the resulting system can do.

RLHF introduces a different feedback construction. People compare or rate responses, a reward model learns from those preferences, and optimization steers the language model toward responses that receive better rewards. Correctness, helpfulness, specificity and harmlessness can represent different objectives.

That mechanism also exposes a limit: a reward is a chosen signal, not a declaration that all human values have been solved. The lecturers explicitly call alignment an ongoing problem. A system’s behavior depends on what feedback was collected, which dimensions were weighted and how well the learned reward represents the intended goal.

In the Large Language Monkeys example, a fixed model receives the same problem many times and generates different candidates. A verifier or selection mechanism is then needed to decide which candidate to return. Code tests are one illustrative source of feedback.

The plotted coverage is the fraction of problems for which at least one sampled answer is correct. It can rise as the number of samples increases even when a single sample performs poorly. The lecturers note that a difficult problem may have only a few correct candidates among thousands.

That distinction is essential. Knowing that a correct answer exists somewhere in a large set is different from deploying a system that reliably finds it. An oracle-like coverage result describes potential available to a good selector; it cannot be silently substituted for the accuracy of the answer actually delivered to a user.

Parallel candidates can be generated simultaneously, but parallelism does not erase compute cost. The lecture separates latency from total resources and notes that the trade-off changes with problem type and difficulty.

Diversity matters as well. Temperature can encourage different generations, yet increasing it without limit eventually produces unusable output. A numerical setting mentioned in discussion is an example from that context, not a universal tuning rule.

A student asks whether sample count can depend on difficulty. The speakers discuss reward-guided continuation as a research direction. The question is not simply “how many more calls can we afford?” but whether additional computation has a useful chance of producing and identifying a better solution.

Inference-time scaling holds model parameters fixed while spending more computation on generating or improving a response. The next step discussed in the lecture is to collect useful outputs, filter them with feedback and use them as synthetic data for later fine-tuning.

This connects search to learning without pretending they are the same operation. Generating candidates does not itself update weights. A separate training process may help the model produce better answers more often on a later attempt.

The lecturers contrast repeated-sampling coverage with the pass-at-one gains shown for reasoning models. They also acknowledge that how much reinforcement learning elicits existing ability versus adds capability remains debated. The feedback loop is a research opportunity, not a settled proof of limitless recursive improvement.

The transition from chatbot to agent adds a goal-directed cycle. A system plans actions, interacts with tools or an environment, observes feedback and adjusts its next step. Memory can help it retain what it is trying to accomplish across those steps.

43:10View full image

The lecture contrasts this with static agentic workflows. A human may draw a graph of calls, judges and aggregators in advance; an open-ended agent chooses more of its path as it proceeds. Both can be useful, but a hand-built graph should not be mistaken for unrestricted autonomy.

Stopping is part of the design. The system may achieve the goal, or return that it cannot. Clarifying user intent and identifying what would count as completion are therefore not optional niceties: they influence which files to inspect, which actions to take and what feedback could establish success.

The overview names several recurring patterns. Prompt chaining connects subtasks; routing chooses a path according to the input; parallel work divides or repeats effort before aggregation; an orchestrator plans further calls. A judge or critic can provide feedback at several of those points.

A verifier has a more specific role when the domain supports checks. Running code tests gives feedback from execution, whereas asking another language model for a judgment supplies a learned assessment. These signals should not be treated as interchangeable guarantees.

Even code tests cover the behaviors they actually exercise. Better generated tests may strengthen a coding loop, but weak tests can leave the generator-verifier gap unresolved. The lecture repeatedly returns to robust verification as a bottleneck, especially where human judgment is costly and no simple rule defines correctness.

The lecture uses code migrations, data cleanup, support assistance and research reports as examples of work agents may help perform. Each has a different completion signal: tests, relevant records, source coverage or human evaluation. A capability in one setting does not automatically validate another.

Scientific brainstorming is a particularly clear boundary. Models may suggest ideas outside a researcher’s usual path, yet can also hallucinate. Producing more ideas is not the same as validating an experiment or establishing a scientific result.

The course-project discussion reflects that standard. Students are asked for a question or hypothesis, experiments and an analysis of improvement or reliability, rather than only a survey or an assembled app. The archived logistics describe that recording’s course, not current enrollment advice. The durable lesson is to make the proposed improvement measurable enough to discover when it fails.