Engineering Reliable Agentic AI Systems
The primitive cycle is still Reason and Act. The product is the stateful control system that decides what the model may do, how the result is checked, what the system remembers, and when the work stops.
Prompt-only systems depend on the model to follow instructions, remember constraints, judge completion, and stop at the right time. That design combines action and control in one probabilistic component. A reliable system separates those concerns. An orchestrator owns budget and exits. A doer acts inside an enforced scope. A judge scores evidence without a write path. Durable files carry state across attempts. Deterministic gates make the final decision. This architecture does not remove model uncertainty. It contains uncertainty inside a system that can measure and explain it.The quality boundary must become software.
A single agent session often looks effective. A person describes a task, watches the model act, reads the result, and sends corrective feedback. The loop appears autonomous because the model performs the visible work. The person still owns the hidden control system. The person supplies the quality bar, remembers the original intent, detects drift, and decides when the work is complete. This arrangement fails under volume. Ten tasks create ten opportunities for the quality bar to drift. One hundred tasks create a review queue. The bottleneck is not the model response. The bottleneck is the person who must inspect every diff, brief, or branch. The unit of work is a controlled loop, not a single generation. A generator with a repeat statement is not a control system. It can repeat an action, but it does not prove that the action was safe, useful, or complete. It also does not know whether a later attempt improved the result. Repetition without evidence creates cost without convergence. AlphaCodium reported a pass@5 increase from 19 to 44 with the same model and a different test-driven flow. This is evidence for flow design, not a universal performance guarantee.Prompting fails under volume because the human remains the quality system
Reason and Act describes the inner cycle. A production system needs another layer around that cycle. The outer layer receives a trigger, grants a bounded capability, observes evidence, stores state, and selects an exit. The workshop calls this discipline Loop Engineering. Each control surface answers a different question. Trigger asks when work may start. Scope asks what the current role may change. Verification asks what evidence proves completion. State asks what must survive the current context window. Exit policy asks whether the system passes, retries, or escalates.The product is the outer control system around the model
A trigger starts one attempt. The doer acts inside a declared scope. Independent verification evaluates the artifact, not the model's confidence. Durable state records the attempt before the decision runs. The decision has three outputs. Pass publishes the verified result. Retry returns to scoped action with specific failed checks. Escalate transfers control to a person when the system spends its budget or stops converging.
The decision must live outside the model. A model can recommend an exit, but code must enforce the budget and state transition. This prevents the model from extending its own budget, accepting its own unsupported claim, or declaring success because the response sounds complete.
A context window is a scratch pad. It is not a source of record. Large tool output goes to a file. A short summary returns to the model. The next attempt reads the relevant state, not the entire history of every prior attempt.
Many agent designs assign names such as planner, implementer, and reviewer. Names do not create separation. Two prompts that describe different roles can still call the same write tool. The workshop repository treats scope as a type. A role definition declares allowed and denied paths. The runtime checks the path before a tool performs the write. The orchestrator owns budget, state, and exits. The doer changes the controlled object inside a declared scope. The judge reads evidence and returns a verdict. The judge receives no write path. This absence is stronger than an instruction that asks the judge not to write.Roles matter only when capability boundaries enforce them
The doer reaches the target repository only through the tool boundary. The judge receives repository evidence and returns a verdict to the orchestrator. The important relationship is the one that does not exist. The judge has no write arrow to the repository. The doer cannot expand its own scope because the boundary evaluates paths before the write occurs.
Verification must measure the artifact, not the story about the artifact
An agent can produce a fluent explanation of incomplete work. It can also run tests that measure the wrong behavior. The harness therefore treats every check as a contract. The check must produce evidence that another component can evaluate.
A green unit test result is only one row in a larger rubric. Other rows can check whether a new test first failed, whether coverage stayed above a floor, whether end-to-end behavior passed, whether the diff stayed inside scope, and whether the receipt matches the current repository tree.
False completeness needs independent verification. Runaway iteration needs a budget and exit policy. Context rot needs durable state. Stagnation needs a normalized failure signature. Each failure that looks like a model problem can often be reframed as a missing control surface. The model can still fail after the control exists, but the failure becomes bounded, visible, and explainable.
Tests first create evidence that the implementation changed the outcome
The red gate compares test state before and after the test implementer acts. It does not ask whether any test is red. It asks whether a newly introduced test failed. That difference proves the new test could detect the missing behavior.
Stable failure is a first-class exit
The repository normalizes failed checks into a failure signature. If two consecutive attempts have the same signature, the loop escalates. The system now has evidence that it is not converging.
A green rubric reaches Pass. A gap that can change reaches Retry, then Act, and returns to Observe for a new evaluation. Budget spent and stable failure both reach Escalate. Pass and Escalate have no outgoing transition. A person can start a new loop with a revised contract, but the current loop is complete.
Research loops add a tool boundary and an evidence contract
The same graph can control a question instead of a code change. A researcher retrieves evidence. A writer assembles a brief. A deterministic judge checks citation coverage and style. The orchestrator owns the cost and iteration budget.
Model Context Protocol standardizes how an agent discovers and calls external tools. Standardization increases reach. It does not make every tool safe. The system must still choose which servers, tools, and arguments the role may use.
The research loop can search and write inside its own work directory. It cannot merge a branch, deploy a service, or edit the target repository. This is a capability boundary, not a prompt suggestion.
The researcher can reach only approved search tools through the MCP boundary. Retrieved evidence feeds the writer. The cited brief moves to a deterministic judge, which returns a result to the orchestrator. Research does not need merge or deploy authority. Removing those tools reduces the effect of a bad decision, malicious tool output, or prompt injection.
Production begins when the chair is empty
An interactive session can rely on a person to notice a dirty repository, a stale report, or an implausible result. An unattended loop cannot. The production architecture must make these conditions explicit.
Durable state identifies the current attempt, previous failure signature, spent budget, last verified result, and source revision. Every run writes a trace and machine-readable artifacts. A receipt binds verification to a specific repository tree and time.
Local and remote gates must agree before a workflow reports success. A local run can pass while remote policy fails. A remote workflow can report green against an old revision. Receipts and revision checks align the two views.
The verified artifact satisfies the contract.
The loop stopped safely and produced a reason.
The runtime failed before it could make a controlled decision.
The controlled object changes, but the architecture remains recognizable. Each loop has an orchestrator, scoped action, independent judgment, durable evidence, and three exits.
Start with the gate, then add autonomy
- Define a contract that can fail.Turn intent into checks that code can evaluate.
- Separate action from judgment.Give the judge evidence and no write path.
- Enforce scope at the tool boundary.Make deny rules stronger than prompt instructions.
- Put retry and cost budgets in code.Do not ask the model to count its own attempts.
- Persist state and receipts.Bind claims to a specific artifact and revision.
- Add unattended triggers last.Automate only after local and remote gates agree.
The loop is the product. The prompt is not.
Engineering Reliable Agentic AI Systems workshop
Glossary