A Toy Evaluation: Keep the Errors Behind the Score

Two classifiers can receive the same accuracy score on a fixed set while producing different errors and different routing work. The fictional exercise below makes that distinction visible with ten invented requests. It is not a benchmark, a deployment report or evidence about any named model. Its purpose is to show what information to retain beside a score before using it to support a decision.

Define the task and the meaning of correct

Imagine a fictional support desk with two labels: routine and access. Routine requests go to a general queue; access requests go to a specialist queue. For this exercise, each request has exactly one reference label, and a prediction is correct only when it matches that label.

The official Hugging Face Evaluate quick tour describes accuracy through the fraction of predictions matching their references. It also documents module descriptions and expected input formats. Those points establish the metric’s basic meaning; the scenario and review workflow here are original.

Before collecting predictions, write down the label definitions. Decide how to handle mixed requests and insufficient information. If your real task needs an uncertain or human-review outcome, define it explicitly rather than forcing every ambiguous request into this example’s two-label arrangement.

Work through the ten fictional requests

The constructed set contains eight routine requests and two access requests. Classifier A predicts routine for every row. Classifier B predicts routine for the first six rows and access for the remaining four.

Classifier A gets the eight routine requests right and the two access requests wrong. Classifier B gets six routine requests and both access requests right. Each therefore has eight correct predictions out of ten: 80% accuracy on this invented set.

Now inspect the routing. Under the stated rule, A sends all ten requests to the general queue. B sends six to the general queue and four to the specialist queue. B’s specialist queue includes two correctly routed access requests and two incorrectly routed routine requests. A’s general queue includes both incorrectly routed access requests.

Neither routing pattern can be judged acceptable from accuracy alone. You would need to know the consequences of the two error types, the available review process and the requirements for the task. This exercise deliberately supplies none of those as real business facts.

The per-category counts expose the difference: A is correct on eight of eight routine cases and zero of two access cases; B is correct on six of eight routine cases and two of two access cases. Retain the denominators so readers can see how little evidence each category contains.

Save a row-level error ledger

Keep the evaluated rows alongside the aggregate result. A useful ledger includes a stable request identifier, reference label, prediction, correctness flag and a brief error note. Add the evaluation-set version and prediction-run identifier so the row can be traced to a particular comparison.

For R09, an error note for A could say, “Reference is access; prediction is routine; routed to the general queue under the exercise rule.” That records an observable mismatch and its defined routing consequence. It should not claim that a customer was harmed or that a specialist later corrected the request unless such events were actually observed.

Separate prediction errors from other findings. A malformed response, a missing prediction and a disputed reference label require different investigation. Record them without silently converting each into an ordinary wrong label. Explain how each category affects the reported denominator.

Keep sensitive request content out of broadly shared reports when it is unnecessary. Stable identifiers and concise, authorized excerpts can support review without reproducing an entire underlying conversation.

Add decision criteria without hiding them in the score

Suppose a fictional reviewer proposes an acceptance rule: every access case in this particular set must reach the specialist queue. Under that stated rule, A fails the gate and B passes it. This conclusion follows from the invented criterion and rows; it does not establish that B meets every operational requirement.

Another reviewer could require that no routine request enter the specialist queue. Under that different rule, B fails and A passes. The conflicting results reveal a policy question that the accuracy score cannot answer.

If a team uses weighted error penalties, document the weights and who approved them. For illustration only, assigning five penalty units to each missed access request and one to each misrouted routine request gives A ten units and B two. Reversing the priorities could reverse the preference. These invented units express a chosen policy; they are not measured monetary losses or universal error costs.

Present the raw counts beside any weighted result. Readers should be able to see which judgment transformed the same errors into a particular ranking.

Compare versions on the same inputs

To compare two prompts or classifier configurations, hold the evaluation inputs, reference labels and scoring rules fixed for that comparison. Record the exact change being investigated. If you change both the prompt and the routing policy, describe the result as a combined change rather than attributing the difference solely to prompt wording.

Build a changed-outcome list as well as a score table. Identify rows that became correct, became incorrect or remained wrong. A flat headline score can conceal improvements on some requests and regressions on others, as the constructed A-versus-B comparison demonstrates.

If the system can produce different outputs across runs, retain each run separately. Define the number of repetitions and the reporting method before inspecting the results. Avoid selecting a convenient run afterward and presenting it as the entire comparison.

Also record missing outputs and formatting failures. Excluding them without explanation changes the question the reported accuracy answers.

Treat label corrections as dataset changes

A reference label can be disputed. Perhaps a request contains both a routine question and an access problem, while the original labeling policy never defined which should take precedence. Review that disagreement against the policy rather than assuming the classifier or the original label must be right.

When the policy or label changes, create a revised evaluation-set version and document the reason. Recompute both candidates against the revised references. Otherwise, their scores describe different definitions of correctness and should not be presented as a controlled comparison.

Keep examples used for prompt editing distinct from a later assessment collection. Record which cases influenced development. This does not guarantee an unbiased assessment, but it gives reviewers a concrete account of what the developer had already seen.

Do not delete an awkward case simply because it reduces a preferred result. Decide whether it is invalid using the dataset’s stated inclusion rules.

Keep conclusions proportional to the example

On a ten-item set, changing one prediction changes accuracy by ten percentage points. That is arithmetic about this denominator, not proof of a stable advantage on future requests. Report “eight of ten” alongside “80%” so the scale remains visible.

The set’s composition is another boundary. Eight routine and two access examples were chosen to demonstrate an error pattern. Those counts do not establish the distribution of any real support queue. Likewise, two correct access predictions do not establish performance across languages, account types or unfamiliar request wording.

A small exercise can help check label definitions and report structure. It cannot by itself establish readiness for unattended routing. State which task conditions have been represented and which remain unexamined. Prefer a narrow conclusion supported by the rows over a broad claim attached to an attractive percentage.

Choose the next evaluation step

Start by reproducing the arithmetic on paper or in a review document. Ask a second reader to explain the difference between A and B using only the table, label policy and routing rule. If their interpretation differs, clarify the report before expanding the exercise.

Next, add deliberately selected examples for gaps you identified: mixed requests, missing context or a disputed boundary between labels. Explain why each new case belongs. For related planning, read building a small evaluation set for an AI ticket classifier and evaluating RAG systems with metrics and frameworks.

The resulting review packet should contain the score, category counts, row-level errors, evaluation version and unresolved decision criteria. Keep those artifacts together so a later reader can assess what the result actually supports.