Build a support-ticket summary you can check

A support summary can sound useful while getting the counts wrong. Build a smaller workflow first: classify twelve tickets, preserve their IDs, and calculate totals in code. Then write a summary from those checked records.

This walkthrough provides a synthetic dataset, an answer key, and a Python counting script. The script has been checked against the supplied answer key. No language-model output or vendor comparison is presented as a completed test.

1. Download the sample

Download twelve synthetic support tickets (CSV). All messages are invented for this exercise. They contain no customer records.

The file contains ticket_id, subject, and body. Inspect it in a spreadsheet or text editor before supplying it to a model. Do not mix the answer key into the classification input.

2. Define the categories

Use these definitions:

• billing: charges, invoices, payment failures, or subscription billing.
• access: sign-in, passwords, account locks, or authentication.
• delivery: physical shipment, dispatch, or tracking.
• feature: a request for a capability that is not described as already available.
• bug: an existing software action fails or returns an error.
• unknown: the ticket does not provide enough information for one of the above.

A category label is a decision rule. Writing the definitions first makes disagreements visible. In a real support system, overlapping categories may require multiple labels or a clear precedence rule.

3. Ask for evidence with each label

Use the following instruction with the ticket CSV:

Classify each ticket using only its subject and body.
Allowed categories: billing, access, delivery, feature, bug, unknown.
Use unknown when the ticket lacks enough information.
Return CSV with ticket_id,category,evidence.
Copy evidence from the ticket; do not invent a reason or customer detail.
Include each supplied ticket exactly once.
Do not calculate totals or write a narrative summary yet.

Add the category definitions above to the instruction. The evidence field helps a reviewer see why a label was selected. It does not automatically prove that the category is correct.

Save the unedited result and the prompt version. If the product wraps its CSV in a code block, extract the CSV into a separate file and record that manual step.

4. Check labels before counting

Download the exercise answer key (CSV). Compare labels by ticket ID rather than row position. Inspect disagreements and the quoted evidence.

The intended unknown case is T012: the message asks for help without identifying a problem. A confident billing or access label would add information that is absent from the ticket.

These twelve simple examples are a teaching fixture, not a representative support benchmark. They cannot establish production accuracy. Add realistic ambiguity and new held-out cases before assessing a real workflow.

5. Validate the records and calculate totals

Download the Python counting script. It uses the Python standard library. Save your classification result as classifications.csv in the same folder, then run:

python3 count-ticket-categories.py classifications.csv

The script rejects missing or duplicate IDs and unsupported categories. It requires all twelve sample IDs. It accepts an additional evidence column, but does not evaluate that evidence.

When run on the supplied answer key, the expected counts are:

access: 3
billing: 3
bug: 1
delivery: 2
feature: 2
unknown: 1
total: 12

Valid structure is not the same as correct classification. Swapping two valid labels can pass the structural checks. Keep the answer-key comparison and human review as separate steps.

6. Write the report from the checked records

Only after resolving labels should you ask for a short summary. Supply the verified counts and supporting ticket IDs. Ask for three parts: category totals, observed issues, and questions that need follow-up.

Do not ask the model to infer business impact or root causes from these messages. A delivery complaint does not establish that a courier lost a package. A failed payment does not establish a bank outage.

A useful report might point to the three billing tickets and describe their different issues. It should not combine them into a single unsupported explanation.

7. Add deliberate failure cases

Try removing a row or duplicating an ID in a copy of the classifications file. The counting script should reject it. Then try replacing a valid category with an unsupported value.

For the language-model part, create a ticket that mentions both billing and sign-in. Decide how your policy should handle it before scoring the response. If your definitions cannot resolve it, revise the category scheme rather than treating every disagreement as a model failure.

Use the evaluation worksheet to record actual outputs, corrections, and costs. Write a project case study only after you have evidence from your own run.

What this exercise teaches

The model handles interpretation. The script handles counting and completeness checks. A person reviews the category policy and the evidence. Keeping these responsibilities visible makes mistakes easier to locate and correct.

The next step is not more automation by default. First identify which errors remain and whether another tool would reduce them.