Skip to content
Leone Intelligence Systems

Insights

The Production AI Brief.

Practical notes from production-agent work: evaluation, GraphRAG, approvals, and the handover problems prototypes hide.

Evaluation

Name the failure owner before you approve an AI workflow

A decision-ready evaluation states what counts as failure, who reviews it, what evidence settles it, and what happens next.

6 min

Approval should follow a named failure path, not a single quality score.

Define the failure condition

Write the operational case where the system must stop, ask for review, or route work back to a person. Keep the condition tied to a decision the team can observe.

  • Name the decision the system is allowed to support
  • List unsafe, incomplete, and out-of-scope outcomes
  • State the evidence required to classify each outcome

Assign the review path

A reviewer needs a clear queue, the source context behind the output, and an escalation route. Ownership is part of the control, not a note added after evaluation.

  • Name the accountable reviewer and backup
  • Show sources, model output, and policy state together
  • Record approve, reject, escalate, and defer outcomes

Record the release decision

The acceptance record should make the remaining limits visible to operators and future maintainers. A release can be bounded without pretending every case is solved.

  • Link evaluated cases to the acceptance decision
  • List unresolved limitations and their owner
  • Define the rollback or pause action for a failed case