THE MOVING FRONTIER · MARCH 2026 RETROSPECTIVE
The PRD’s hidden assumption
Traditional product requirements often assume that a defined input should produce a defined output. The team writes acceptance criteria, engineering implements them, and QA verifies that the behavior is correct.
An AI system may produce several plausible outputs for the same request. Performance can vary by context, source quality, model changes, or the way a user phrases the task. A document that describes only the happy path leaves the most important product decisions implicit.
The answer is not to abandon product requirements. It is to write a better kind.
Specify the shape of success
I would begin with the job, not the model. What result is the customer trying to achieve? How would an expert recognize a good one? What is the smallest useful unit of work?
Then specify quality dimensions: factual grounding, completeness, consistency, timeliness, and the ability to recover from incomplete information. Different tasks prioritize them differently. A creative brainstorming assistant may tolerate variation that a compliance workflow cannot.
Next comes a failure budget. What errors are acceptable at low consequence? Which errors must be impossible by design or caught before action? If the failure would be serious, the system needs a control that does not depend on the model merely promising to behave.
The evaluation set is a product artifact
A good brief should contain representative cases: routine examples, edge cases, ambiguous inputs, adversarial conditions, and examples where the correct response is to abstain. The examples should come from the workflow, not just from model benchmarking conventions.
The product team should work with domain experts to define what a reviewer would look for. A numeric score is useful, but so is a record of disagreement: why did one expert accept the work while another rejected it?
Evaluation cases are not an engineering appendix. They are one of the most concrete ways a product manager expresses customer value.
State who is in control
Requirements also need to describe the interaction between human and system. Does the product present a recommendation, prepare an action for approval, or execute within predefined limits? What evidence is visible? Can the user correct a result without starting over?
These choices shape both usability and risk. ‘Human in the loop’ is too vague to be a requirement. ‘A user must confirm any external communication and can inspect the referenced source documents before sending’ is actionable.
Finally, specify what happens after launch: monitoring, feedback capture, regression evaluations, and the threshold for pausing the behavior. Probabilistic products are not done merely because a release passes once.
A more useful definition of done
The product is ready when it solves an important job, its failure modes are understood, the relevant quality bar is demonstrated, and the customer has an appropriate degree of control.
That definition is harder than a traditional feature checklist. It is also far more useful for building AI products that can be trusted with real work.
Editorial note: This is a retrospective essay written in November 2026. March 2026 describes the period explored, not its original publication date.
