CS329A Self-Improving AI Agents, Part 4: Learning from Feedback with Tools and Code

CS329A’s fourth lecture compares three ways of improving model behavior: ReAct interleaves reasoning with environmental actions, RLEF learns from code execution, and Constitutional AI uses critiques guided by human-written principles. All form feedback loops, but the signals mean different things.

A search result is an observation, a failing test is execution evidence and a principle-based judgment is an evaluation of desired behavior. None should be silently substituted for another. The distinction determines what the system can check and which errors may persist.

The lecture also separates a better attempt within one interaction from learning that updates a model. A frozen model can use new observations in context; a training loop can later change its parameters. Calling both self-improvement does not make them the same mechanism.

The basic ReAct pattern alternates a thought, an action and an observation. The next thought can use what the action returned rather than blindly execute a complete plan written before any evidence arrived.

In the historical few-shot setup, a frozen model sees demonstrations containing these elements. Reasoning text itself does not change the environment; a tool call does. Giving the model a defined set of valid actions constrains its next choice, though it does not guarantee that the chosen action accomplishes the goal.

This is a useful abstraction for grounding a response. It is not a complete agent implementation with every memory or session mechanism included, and an interpretable trace should not be mistaken for proof that every explanation is faithful or every action safe.

The lecture walks through a HotpotQA example involving Apple Remote and Front Row. A search produces a clue, a follow-up query needs reformulation, and the next step depends on that result. Interleaving reasoning with retrieval helps the model use the relationship between pieces of evidence.

A student asks how a model knows when to search. The lecturer points to unreliable confidence and an unresolved calibration problem. An application should focus on providing appropriate tools and checking their outputs, rather than assume the model always knows the limits of its knowledge.

Another question raises contradictory search results. The response calls for validation appropriate to the application. Retrieval can reduce dependence on unsupported internal guesses, but a returned page is still something to evaluate; search does not turn disagreement into truth.

The reported ReAct results are not uniformly better than every reasoning baseline. In the discussed tasks, combinations that fall back between ReAct and chain-of-thought self-consistency can perform better than either simple pattern alone.

WebShop adds sequential decisions in a simulated shopping environment. Its success measure concerns completing that benchmark task, not proof that an agent has made an authorized real purchase. The lecture also shows a gap from human experts and notes how errors can cascade through multiple steps.

Larger action spaces require more demonstrations and can strain context; more reasoning steps add inference cost. Class suggestions such as backtracking, memory and better task decomposition identify directions to explore, not experimentally established cures for noisy feedback.

RLEF treats generated code as an action and test execution as an observation. A model receives a programming problem, proposes code and sees whether tests pass or fail. A failed attempt can be appended with its feedback before the next attempt, up to a turn limit.

The example begins with a palindrome solution that times out. After execution feedback, the model changes the implementation and passes the public tests. This is a concrete kind of evidence: an actual test ran, rather than another model merely stating that the code looks correct.

The feedback still has a scope. Passing the available tests establishes behavior under those checks, not all possible program properties. The lecture’s central result is that training models to use execution feedback can improve subsequent repair behavior in the studied coding tasks.

The two-tier test strategy gives a small public test subset to the inner iteration loop. Hidden tests then supply reward information for the outer reinforcement-learning update. “Private” here means hidden from candidate generation, not private user data and not an untouched final evaluation set.

30:10View full image

The separation aims to avoid reducing the entire learning signal to the same visible checks used for immediate repair. But hidden tests used for training reward are still part of training. The lecture’s classroom debate about the precise ablation and terminology remains unresolved on screen; it should not be rewritten as a completed proof against all overfitting.

The policy generates tokens while value estimation is organized at the response or turn level. That hybrid design assigns a broader reward across a generated sequence. The important conceptual distinction is the inner loop using a fixed policy and the outer loop updating that policy from execution outcomes.

The lecture reports better solve rates and fewer wrong outputs across repair turns after RLEF training. The tasks are competitive-programming-style problems, often short enough to inspect as a single solution. The lecturer explicitly leaves room for richer feedback on harder tasks.

A binary reward can identify success or failure, but an error trace or other metadata may help diagnose why a large system failed. Questions about process rewards, outcome rewards, supervised fine-tuning and fixed-compute comparisons remain domain-dependent.

When a repository exceeds context, students propose retrieval, summaries and representations to locate relevant code before patching and testing. These suggestions connect the execution loop to a larger information-gathering problem; they are not proof that a small-task training result already solves arbitrary production migrations.

Constitutional AI changes the feedback source again. Humans write principles describing desired behavior, and the model uses them to critique and revise responses. The revised examples then support a supervised fine-tuning stage.

A further stage uses AI-generated preferences to train a preference model, which supplies a reinforcement-learning signal. This reduces the need to collect a human ranking for every generated comparison, but human choices remain in the principles and in how the evaluator is checked.

The examples concern helpfulness and harmlessness. A model must be able to recognize the behavior named in a principle and follow a revision request. Neither ability is automatic merely because the rule is written clearly, which is why the lecture later returns to human-consistency checks.

A student asks what happens when the constitution changes and whether old rules can be completely removed. The lecturer calls continual learning and reliable forgetting open research problems. Updating a document is not equivalent to demonstrating that a trained model has forgotten a previous behavior.

The helpfulness-harmlessness plots also show a trade-off rather than one unconditional score. More restrictive behavior can reduce helpfulness in the presented comparisons. Elo-style preference measures describe relative judgments in those evaluations, not a guarantee of safety in every deployment.

The lecture therefore motivates seeking a better frontier between goals. It does not establish that one extra critique pass monotonically improves every property, or that a historical set of principles is the complete and current specification for all models.

Near the end, the lecturer says the preference model should be checked against human judgments on validation examples. Large amounts of AI feedback can reduce annotation work without eliminating the need to ask whether that feedback tracks the intended behavior.

The same question links all three papers. ReAct needs meaningful observations, RLEF needs tests that reveal relevant errors and principle-based training needs judgments that reflect the chosen goals. Repeating a loop can also repeat its blind spots.

The closing discussion leaves domain-specific workflow design important even as reinforcement learning improves models. A well-defined game has a clearer action space than an open-ended professional task. Better models may automate more search, but the source, scope and quality of feedback still determine what that search can learn.