Compare AI changes on the same tasks: building an operational evaluation set

FROM PRIMARY SOURCES

Official guidance distinguishes execution records from final state

Anthropic's January 2026 evaluation article distinguishes the record of a trial from the final state left in its environment.[1] A response declaring completion does not by itself establish that the required change was saved. We use this existing explanation as a reference for operational design.

Its November 2025 article on long-running agents describes working on one feature at a time and recording progress.[2] That structure alone does not guarantee quality. Below, we propose an original practical workflow for connecting each change to its evaluation results.

Sources: [1][2]

EDITORIAL PRACTICE

Collect inputs at the boundaries of business decisions

Consider an AI that routes enquiries to the right department. This is a design example, not a reported deployment. A set containing only obvious classifications misses difficult boundaries. Include enquiries with two concerns, missing information and requests outside existing categories, then record the handling expected by the responsible person.

When using real records, confirm their permitted use and remove identifying information. If anonymisation also removes conditions needed for the decision, replace the record with a fictional input that explicitly preserves those conditions. Give each task an identifier and a selection reason so the team can explain the business boundaries covered, rather than only the size of the set.

EDITORIAL PRACTICE

Separate machine-checkable criteria from human review

Checks against expected values may work for allowed department codes, required fields and the correct storage destination. Whether an explanation is misleading or a clarification question is appropriate needs review against written criteria. Before compressing everything into one overall score, record operational errors separately from improvements in expression.

If AI assists grading, first compare its judgments with human judgments and identify where they disagree. Record the permitted range when several answers are valid. Grading that rejects harmless differences in phrasing while praising a result saved to the wrong department cannot support operational decisions.

EDITORIAL PRACTICE

Limit the conditions changed in one comparison

Record the versions of the input set and instructions, model settings, connected tools and execution time. If both instructions and tools change, a result difference cannot be attributed to just one cause. Several changes may be necessary in practice; label them as a combined change instead of claiming a separate improvement effect.

For tasks with variable outputs, repeat the same input and compare all results rather than selecting only successful runs. Choose the repetition count according to failure impact and execution cost. Alongside overall scores, compare failures by category, duration and billable usage. This helps expose changes that improve accuracy while increasing the operational burden.

EDITORIAL PRACTICE

Request scores, differences and unresolved issues from the team

An offshore delivery request should identify the comparison, evaluation set, environment and deliverables together. Ask for before-and-after summaries, identifiers of tasks that worsened, grading reasons and settings needed to reproduce them. Records that let a reviewer select relevant tasks support more concrete decisions across time zones than pasting every raw response into chat.

When adding a corrected failure to the set, check other tasks to avoid an improvement tailored only to that example. Retain important previously successful tasks and rerun them so the adoption owner can see regressions. Add newly observed production failures to the next comparison after confirming their reproduction conditions.