This lesson is being recorded
Every video of this course is being uploaded to my YouTube channel. Subscribe so you do not miss it.
Bruno Ashford @brunoashfordSubscribeModule 12 · Lesson 12.3 · 11 min
Quality and Error Rates
Measure rework and errors avoided so speed never hides a quality problem.
Use ← → to move. The marker highlights text and the pencil draws on the slide; navigation locks while you annotate. Undo removes the last mark, Esc exits.
1Define what an error is, per agent
Write one sentence per agent: what counts as a failure. For a proposal agent, a wrong price or missing scope item. For a follow-up routine, a lead contacted twice or not at all. Without a definition, every review becomes an argument.
2Sample and score outputs weekly
Pick ten outputs per agent per week, or all of them if volume is low. Score each one: accepted as is, minor edit, major rework, or rejected. This is the verification habit from Module 11 turned into a number.
Here are the last 10 outputs from my proposal skill, with the final versions I actually sent. Compare each pair and classify it as: accepted as is, minor edit, major rework, or rejected. For every edit, name the cause (missing context, wrong data, tone, instruction unclear). Give me the counts and the top two causes to fix in the skill.3Calculate the rework rate
Rework rate equals major rework plus rejected, divided by total sampled. Under 10 percent is healthy for most business tasks. Above 25 percent means the skill needs better context or instructions before it deserves more volume.
4Count errors avoided
Look at your old error log, complaints or credit notes. If the agent removes a known failure (late invoices, missed renewals, forgotten follow-ups), estimate how often it used to happen and what each one cost. Count it as a gain, labelled clearly as an estimate.
5Feed the causes back into the skill
Every rework cause is a missing line in your CLAUDE.md or SKILL.md. Fix the instruction, then measure again next week. Quality that improves month on month is the strongest ROI story you can tell.
Here is a truth that hurts: an agent that is twice as fast and wrong one time in ten can cost you more than it saves. One wrong invoice, one email to the wrong client, one missed deadline in a law firm, and the hours saved all year are gone.
Quality is also where AI often wins quietly. A checklist that never skips a step, a follow-up that never drops. Those avoided errors are real money, and most owners never count them.
Key points
- Define failure per agent before you measure.
- Sample weekly and score outputs on a four-point scale.
- Rework above 25 percent means fix the skill before scaling it.
- Errors avoided are a real gain, label them as estimates.
Avoid
- Measuring only speed and calling it ROI.
- Letting the agent grade its own work without a human-approved final version.
- Sampling only the easy cases.
- Blaming the model when the real cause is missing context.
An online homeware store used an agent to draft replies to customer emails. It saved hours, but sampling showed 30 percent of replies needed major edits because the agent did not know the returns policy had changed. One updated line in the project CLAUDE.md dropped rework to 8 percent. The same sampling also showed chargebacks from "no reply" complaints had almost disappeared, which the owner counted as an avoided cost.