Suppose you are building an AI assistant that assigns support tickets to billing, technical support, or account access. This tutorial develops a small evaluation process using invented examples. You can complete the worksheet without calling a model. The goal is to make an improvement claim inspectable, not to establish a production-quality benchmark from a handful of tickets.
Step 1: Define the labels
Write one sentence per category and a rule for ambiguous tickets. In our proposed policy, billing covers charges and invoices, technical support covers product failures, and account access covers sign-in problems. Tickets spanning categories are flagged for review. These are example business rules; replace them with your team’s routing policy. Have a second reviewer label several tickets independently and resolve disagreements before treating labels as reference answers.
Step 2: Create a versioned worksheet
Use columns for ticket ID, ticket text, reference label, predicted label, review flag, and error notes. Include straightforward cases and boundary cases. An invented input such as “I cannot sign in after changing my email” can receive account access under this policy. “My invoice download fails” needs a documented decision about whether the failure or invoice intent takes precedence. Do not silently adjust that decision to match the model’s output.
Step 3: Keep a development set and a held-out set
Use the development examples while editing instructions. Keep a separate set unchanged for a later comparison. For a first exercise, a few examples are enough to expose an unclear policy, but they cannot establish broad reliability. Save the label policy, prompt, model identifier where available, evaluation date, and raw predictions with each run. If the data changes, assign a new dataset version.
Step 4: Calculate an understandable score
Hugging Face Evaluate’s official quick tour distinguishes metrics, comparisons, and measurements, and documents passing predictions and reference labels to evaluation modules. Its accuracy description is the proportion of correct predictions. Source: https://huggingface.co/docs/evaluate/a_quick_tour
Here is an original miniature calculation. Let the reference labels for five invented tickets be billing, technical, access, access, billing. Suppose the predictions are billing, technical, billing, access, access. Three positions match, so accuracy is 3 divided by 5, or 60 percent. Count the two errors by reference category as well: one access ticket and one billing ticket were misrouted. This arithmetic example is not a measured model result.
Step 5: Review errors before optimizing
For each mismatch, ask whether the label policy is clear, whether the ticket has enough information, and whether the system produced an allowed label. Record invalid outputs separately rather than dropping them from the denominator without explanation. Review flagged cases and ordinary cases, since a system that flags everything may reduce automated errors while creating too much manual work. Choose that tradeoff according to your actual workflow.
Step 6: Change one thing and rerun
Try a clearer ambiguity rule or a different prompt while keeping the evaluation inputs fixed. Inspect which cases improved and which regressed. Then run the selected version on the held-out set once you are ready to assess it. Report the number of examples, the score definition, category-level errors, and any exclusions alongside the result. Avoid claiming that a small gain on this worksheet proves general superiority.
Your deliverable
Save the dataset version, labeling policy, baseline output, revised output, and a short error report. If you later adopt Evaluate, review the specific metric module’s input format and limitations before using it. This tutorial does not execute downloaded modules or require an installation. The workflow and examples are original; the linked official documentation supports the evaluation terminology and accuracy definition.
Worked example: build a confusion table
Create six fictional tickets before using any real customer data. Two describe duplicate charges, two describe a feature failing, and two describe account access problems. Assign one reference label to each after reviewing your written labeling rules. These examples are deliberately small so that a person can inspect every decision; they are not a representative benchmark.
Suppose a candidate classifier gets five labels correct and sends one account-access ticket to technical support. Its accuracy on this invented worksheet is five divided by six. The overall score hides an operational detail: one of the two access tickets was misrouted. Record that mistake in a confusion table with reference labels as rows and predicted labels as columns.
Now imagine a second candidate makes a billing mistake instead. The overall accuracy may be identical, while the effect on your support queue differs. Decide which errors matter most to the actual routing process before declaring either candidate preferable. Do not manufacture a single weighted score merely to make one candidate look better.
Define invalid outputs and abstention
Specify what happens when the output is not one of the permitted labels. Empty responses, extra explanations where only a label is allowed, and invented categories should have explicit handling. Keep their counts visible rather than removing difficult cases from the score calculation.
An abstention label can be useful when a ticket lacks enough information, but it changes the workflow. Measure how many cases need human review as well as the mistakes among automatically routed cases. An assistant that abstains on everything has avoided automated routing errors while providing no automatic routing. Whether that is acceptable depends on the purpose of the system.
Add a ticket that mentions both a payment failure and a login problem. Ask two reviewers to apply the policy independently, then compare their choices. If they disagree because the policy is ambiguous, fix the policy or mark the example as unresolved before using it to judge a model. The reference label itself can be a source of error.
Keep development and evaluation decisions separate
Save the dataset and policy before changing the prompt. Give each version a stable identifier. Log the prompt version, model identifier when known, output parser, and any preprocessing rule. These details help explain why the same visible question can produce a different routing result after a deployment change.
Inspect the development set while improving the workflow. Keep a separate held-out set for the chosen version's assessment, and record when it has been used. Once repeated inspection influences your decisions, that set no longer provides the same independent check. Create new cases from the workflow's observed failure patterns while keeping sensitive information out of the fixture store.
Do not infer broad reliability from a tiny worksheet. Report the sample count beside every fraction, retain the actual errors, and describe which kinds of ticket were absent. A dataset containing only clear single-topic tickets cannot establish how well the system handles mixed requests, unfamiliar languages, or incomplete descriptions.
Prepare a reviewable experiment note
A concise experiment note can contain the decision you are considering, the fixed test inputs, the candidate change, the score definition, and a table of changed predictions. Separate observed results from your interpretation. State what the exercise does not establish, including conditions you have not tested.
When using an evaluation library, review the chosen metric's input format and interpretation in its official documentation. A library can calculate a score, but it cannot decide whether your labels capture the business task. Keep an arithmetic check on a small invented example so that a formatting mistake does not silently invert predictions and references.
Finish with a concrete next action: clarify one ambiguous label, add cases for one missing category, or test the selected version against the held-out inputs. For the surrounding automation design, read the workflow guide. For context from external systems, use the MCP planning worksheet. The aim is a decision supported by inspectable evidence, rather than a score presented without its assumptions.
Sources
- Hugging Face Evaluate quick tour, for library usage and metric terminology. The ticket examples, arithmetic exercises and experiment worksheet here are original editorial guidance.