This lesson is being recorded
Every video of this course is being uploaded to my YouTube channel. Subscribe so you do not miss it.
Bruno Ashford @brunoashfordSubscribeModule 5 · Lesson 5.5 · 14 min
Testing and Fixing Skills
Test every skill with several different inputs and fix it until it produces dependable results.
Use ← → to move. The marker highlights text and the pencil draws on the slide; navigation locks while you annotate. Undo removes the last mark, Esc exits.
1Prepare three test inputs
One normal case, one difficult case and one edge case with missing or strange data. Save them in a tests folder inside the skill so you can rerun them after every change.
Create three test inputs for the complaint-reply skill in .claude/skills/complaint-reply/tests/: a normal complaint about a late delivery, an angry complaint that includes a false claim, and a very short vague complaint with no order number. Use made-up names.2Run each test in a fresh session
Use /clear between tests so earlier answers do not influence later ones. Call the skill directly first, then test once with a plain-English request to confirm it triggers.
/complaint-reply .claude/skills/complaint-reply/tests/test-2-angry-false-claim.md3Score against the checklist
For each output, check: did it follow every step, respect every red line, match the voice, handle missing data correctly and stay within length? Write down what failed, not just that it failed.
4Fix the cause, not the symptom
If it failed, find the instruction that allowed it. Usually a step is ambiguous, an example contradicts a rule, or the edge case is not covered. Ask Claude to diagnose it with you.
The complaint-reply skill agreed with the customer's false claim in test 2. Read SKILL.md and the examples, tell me which instruction or example caused this, and propose the smallest change that fixes it without breaking tests 1 and 3.5Retest everything and version it
After any change, rerun all three tests, not only the one that failed. When all pass, add a short change note at the bottom of SKILL.md with the date. If your team shares the Company folder, everyone now gets the improved version.
6Keep improving from real use
Every time a real output needs a correction, ask whether the skill should change. Save the tricky real input as a new test case. Over a few weeks your skills become the most reliable people in the company.
That reply needed an edit because it offered a refund we do not give. Update the complaint-reply skill so this cannot happen again, and add this complaint as a new anonymised test case.A skill that works on the example you built it with proves almost nothing. The real test is the awkward case: the angry customer who is also wrong, the report week with missing data, the proposal for a job you rarely do. That is where skills break, and that is exactly where your reputation is at stake.
The method is simple: test with at least three different inputs, compare against a checklist, fix the instruction that caused the problem, and test again. The skill testing checklist in this module's materials walks you through it.
Key points
- One passing example proves almost nothing.
- Test normal, difficult and edge cases.
- Use /clear between tests.
- Fix the instruction that caused the failure, then rerun all tests.
- Turn every real correction into a new test case.
Avoid
- Testing only with the example the skill was built from.
- Running tests one after another in the same conversation.
- Patching one failure and breaking another case without noticing.
- Letting the team use a skill that has never seen an edge case.
Priya's complaint skill passed the easy test first time. On the angry test it apologised for a delivery delay that the tracking showed had not happened. The cause was one line: "always apologise first". She changed it to "acknowledge the frustration, then check the facts in the order data before accepting fault". All three tests passed. Two months later she has 11 test cases and a skill her part-time support staff trust without checking every reply.