Measure the work before you automate it

A model benchmark does not measure your organization’s complete process. Start by observing representative cases from intake through review and completion. Record the work people perform, the time a case waits, and the exceptions that send it back. Keep active effort separate from elapsed time: a faster response and fewer staff minutes can be valuable, but they are different outcomes.

Here is an invented invoice-review example to make the arithmetic concrete. Twenty cases at eight minutes of manual effort each total 160 minutes. Suppose an assisted version takes one minute to prepare and three minutes to review each case, plus 40 minutes of corrections across the batch. That totals 120 minutes, leaving a 40-minute difference before setup, provider fees and maintenance. These are illustrative numbers, not an Ashlr result or a savings forecast.

Quality needs its own acceptance criteria. For this example, require the vendor, invoice number, amount and currency to match the source; require a reviewer to resolve duplicates or ambiguity before approval. Score those checks separately from speed. A draft completed quickly but assigned to the wrong vendor should fail, even if the average processing time improves. Preserve the source reference so the reviewer can inspect the evidence rather than trust a confident explanation.

Anthropic’s engineering guide on evaluations emphasizes clear success criteria, inspection of execution records and maintaining evaluations as systems change. Apply that discipline to the whole handoff. Include difficult cases and previously observed failures, rather than comparing routine automated cases with unusually hard manual ones. Keep a set of cases out of development so repeated tuning does not become the only reason the score looks better.