A team wants an assistant to prepare a weekly customer-feedback report. The tempting specification is “read everything and tell us what to do.” A more useful specification names the inputs, the deliverable, and the decisions that remain with a person. This guide proposes a planning exercise for turning that request into a system you can inspect.
The architectural distinction
Anthropic’s Building effective agents distinguishes predefined workflows from agents that dynamically direct their process and tool use. Its advice favors starting with simpler systems. This is a foundational December 2024 source, and its own notice says the tooling landscape has changed. Use the architectural distinction here rather than treating its product references as a current SDK inventory. Source: https://www.anthropic.com/research/building-effective-agents
Step 1: Write the output contract
For our hypothetical report, require five themes, a count of supporting feedback items for each theme, links to the original items, and a separate section for uncertain interpretations. Define the reporting window and deduplication rule. “Improve customer satisfaction” is a business goal; it is too broad to be the acceptance test for a single run. A reviewer should be able to inspect the report without reconstructing the system’s intentions.
Step 2: Draw a fixed first version
Use this proposed sequence: select records from the date window, remove duplicate record IDs, group observations, draft themes with supporting IDs, validate that those IDs exist, and send the report for review. Let ordinary application code enforce the window and ID checks. Ask the model to interpret the selected text. This allocation is a design recommendation, not a claim that any particular model will classify feedback correctly.
Step 3: Locate the uncertain decision
Write down the point where a fixed path fails. Perhaps a reviewer sometimes needs additional context from a product manual. That may justify a bounded retrieval step. It does not automatically justify permission to update the manual or message customers. Add one capability at a time and give it an explicit purpose. If a proposed tool cannot be connected to an observable requirement, leave it out of the first version.
Step 4: Specify the operating envelope
For this example, propose a maximum of three retrieval attempts per theme and a run deadline selected by the team. Treat these as initial engineering limits, not universal thresholds. On exhaustion, return the partial report and name the unresolved question. Define who can authorize external writes, where failures are recorded, and how a reviewer can reproduce the source selection. Keep source documents as evidence; text inside them should not redefine the application’s permissions.
Step 5: Compare against a baseline
Run the fixed workflow and the more autonomous variant on the same saved feedback sets. Compare missing themes, unsupported statements, reviewer corrections, elapsed time, and recorded service usage. Keep a small set of difficult examples separate while editing the workflow. A system that produces a longer report has not necessarily improved. Choose the variant that meets the team’s requirements with acceptable review effort.
A usable handoff document
Before implementation, produce one page containing the output contract, permitted data sources, allowed actions, stop conditions, failure response, and evaluation examples. Mark unresolved choices explicitly. This document makes framework selection a concrete engineering decision: can the chosen implementation express and inspect the behavior you already defined? The proposed exercise is original editorial guidance; the linked Anthropic source supports the workflow-versus-agent distinction.
Worked example: a weekly feedback brief
Consider an invented team reviewing feedback about a scheduling application. Its weekly input is a fixed export of support messages, with customer identifiers removed and each message assigned a local evidence ID. The requested deliverable is three candidate themes, evidence IDs for each theme, and a separate list of unanswered questions. No customer reply or issue-tracker update is authorized.
The first implementation can be a fixed sequence: validate the export, group messages using a documented rule, draft theme summaries, and ask a reviewer to approve the brief. Record which messages were available to each step. A reviewer should be able to distinguish an omitted message from a message that was considered and assigned to another theme.
Now introduce one uncertain decision: a message uses an unfamiliar term that might describe an existing feature. A bounded assistant could search the approved documentation for that term. It should return the discovered evidence or an unresolved question, rather than inventing a feature definition. That decision is the candidate for autonomy; importing files and calculating counts can remain ordinary application code.
Define a small acceptance worksheet
Write expected behavior before comparing implementations. Use a message that clearly matches a known theme, one that fits two themes, and one that contradicts a proposed summary. Include an empty export and a document containing a request to contact someone. These are test fixtures, not reports of observed production behavior.
For every fixture, name the permitted information, the expected deliverable, and the condition that should stop further work. For an empty export, the correct result may be an explicit no-data report. For conflicting evidence, the result should retain the conflict rather than choose the more convenient message. For the contact request, the read-and-draft workflow must leave external communication unavailable.
A simple scorecard can record unsupported claims, omitted evidence, invalid output fields, and reviewer corrections. Keep those observations separate from execution time and service usage. A variant that finishes faster but requires substantially more correction may not serve the team's goal. Choose acceptance thresholds from the actual review process rather than borrowing an impressive-looking benchmark.
Plan recovery and human review
Decide what happens if retrieval succeeds but drafting fails. One practical design preserves the selected evidence and marks the draft incomplete, allowing the operator to resume from a known stage. Another reruns the full workflow using the same input snapshot. Whichever approach you choose, explain how duplicated work and stale evidence are recognized.
A review screen should show the draft beside its evidence, highlight unresolved questions, and record approval as an explicit event. Avoid treating an assistant's statement that a report is ready as proof that a person approved it. Similarly, a successful API response does not establish that the report's claims are supported.
Keep a small decision log for changes. Record what was changed, which fixtures were evaluated, which errors improved, and which new errors appeared. That log helps the next maintainer understand why a retrieval limit or escalation rule exists. It also makes reverting an unsuccessful change less dependent on memory.
Decide whether to expand autonomy
At the end of the pilot, compare the fixed workflow with the bounded variant on the same saved inputs. If the uncertain decision rarely occurs, a manual exception queue may be simpler than an autonomous loop. If it occurs often and the bounded variant resolves it reliably under your acceptance criteria, consider extending that specific decision while keeping the rest of the contract stable.
Use the classifier evaluation worksheet to structure the comparison. If external context or actions are needed, the MCP planning guide explains how to inventory them. These are complementary design exercises: define the workflow first, then choose integrations that can express its actual requirements.
Sources
- Anthropic: Building effective agents, for the workflow-versus-agent distinction. The worksheets and fictional feedback example above are original editorial guidance.