Somewhere in your operation, skilled people spend part of every day reading messages and deciding where they go. Support tickets get triaged to the right queue. Invoices, complaints, and contract requests get filed to the right team. Inbound email gets forwarded — twice, usually — before it lands with someone who can act on it. None of this is the work; it's the sorting that happens before the work. It's slow, it's inconsistent between people and across Mondays, and every misrouted item adds a day of latency and a frustrated customer. The volume grows; the sorters don't.
Text classification is the fix: a system that reads each message and routes it to the right category, instantly and consistently. It's also the rare AI project where trust is easy to establish, because every decision it makes can be checked against what a person would have done. But that trust has to be built deliberately — and the build follows a discipline that has more to do with your categories and your examples than with any model.
Walk the Build
The walkthrough below follows a support ticket through the six stages of building a production classifier. Step through each stage to see the business decision it represents — and the shortcut it protects you from.
Splendor · Intelligent systems
Teach a system to understand and route text.
Follow a support ticket from raw message to a reliable category. Each step shows the business decision and the model behind it.
A
01 · Problem and taxonomy
First, decide what the sorter should recognize
Choose clear categories and decide whether a message can have one label or several. Define what to do when no category fits.
A ticket says “I cannot sign in.” → Access. A refund request → Billing.
Taxonomy
Single-label vs. multi-label
Fallback class
1 / 6
✓
02 · Representative examples
The team shows the sorter real examples
Gather messages that reflect production, label them with written rules, and resolve disagreements. Reserve separate validation and test sets before tuning.
“Password reset link expired” → Access. Keep near-duplicates and related customer threads in the same split.
Annotation guide
Class balance
Leakage-safe split
2 / 6
T→
03 · Simple first model
Start with a simple sorter you can inspect
Turn text into TF-IDF features, then train a linear classifier. This is the benchmark a more complex model must beat.
“Invoice charged twice” contains strong signals for Billing, even before a large language model is involved.
TF-IDF
Logistic regression
Baseline
3 / 6
⚙
04 · Model improvement
Teach a stronger model only when needed
If the baseline misses paraphrases or domain language, compare embeddings or a fine-tuned transformer. Tune on validation data, including class weighting and thresholds.
“The portal keeps rejecting my credentials” should map to Access without the word “password.”
Embeddings
Fine-tuning
Optional LoRA / QLoRA
4 / 6
◎
05 · Independent evaluation
Give the sorter a fair exam
Use untouched test messages. Review errors by category and confidence; set thresholds for human review when the model is uncertain.
A wrong Billing → Access prediction appears in the confusion matrix and is investigated.
Precision / recall / F1
Confusion matrix
Calibration + abstain
5 / 6
↻
06 · Production and learning
Put the sorter to work with guardrails
Serve the versioned model through an API or workflow. Capture corrections, watch for changing language and class mix, and retrain only after measured degradation.
Low-confidence tickets go to a person. Their corrections become candidates for the next labeled set.
Versioned inference
Human review
Drift monitoring
6 / 6
The goal is dependable decisions, not a bigger model. Compare each change against the baseline and route uncertain cases to a person.
Each stage exists because skipping it produces a specific, expensive failure:
Define is where clarity lives. Categories that overlap, or that no two supervisors would apply the same way, produce a classifier that faithfully automates the confusion. Deciding up front what happens when nothing fits — a fallback class, a human queue — is the difference between a system that degrades gracefully and one that guesses.
Label is where quality lives. The model will learn exactly what your examples teach it, including their gaps and biases. Written labeling rules, resolved disagreements, and a test set nobody touches until the end are what make the final numbers believable.
Baseline is where honesty lives. A simple, inspectable model sets the bar. If the expensive model can't clear it, you've saved yourself a costly deployment; if it can, you know exactly what the added complexity is worth.
Train is where restraint lives. Embeddings, fine-tuning, and adapter techniques earn their place only when the baseline demonstrably misses paraphrases and domain language — not because they're newer.
Test is where proof lives. Untouched data, error review by category, and a confusion matrix turn "it seems to work" into numbers an executive can accept or reject.
Operate is where durability lives. Language drifts, products change, and last year's categories quietly stop fitting. Versioned models, human review for low-confidence cases, and retraining triggered by measured degradation keep the system trustworthy after launch day.
What to Demand Before You Trust It
Accept a classifier the way you'd accept any operational system: on evidence. Demand precision and recall per category on the held-out test set — not one blended accuracy number that a dominant category can hide behind. Demand the confusion matrix, and look at where the costly mistakes land: a misrouted password reset is an annoyance, a misrouted legal notice is a liability. Demand a calibrated confidence threshold with an abstain path, so the system routes what it's sure about and escalates the rest to a person. And demand an operating plan — who reviews the escalations, where corrections go, and what number triggers retraining. This is the routing half of workflow engineering: the handoff a classifier automates is only as good as the process it feeds.
As with every AI initiative, the model is rarely the bottleneck — the examples are. If your organization can't produce a few thousand representative, consistently labeled messages, that's not a blocker; it's the first deliverable, and it's the same readiness work that determines whether you're AI-ready at all. Our AI & intelligent automation practice builds classifiers this way: taxonomy and labels first, baseline before big models, and a human in the loop wherever the evidence is thin.
Could your operation support this today?
The free AI Readiness Assessment scores the four conditions projects like this depend on — including whether your data can carry the load.
20+ years in AI, enterprise architecture, and application development. Helping organizations modernize technology and drive measurable business outcomes.
Work with Splendor
Buried in messages a system could route?
Let's talk about where classification fits in your operation — which queues, which categories, and what reliable would look like.
