Support Conversation QA Review: Checklist and Scoring

An agent's CSAT is 90%. Does that mean everything is fine? Not necessarily. The customer may have been happy because their problem was solved, while the agent gave incorrect information, sounded cold, or ignored company rules. A satisfaction survey tells you how the customer felt; it doesn't tell you how the conversation itself went.

Quality assurance (QA) means someone reads conversations and scores them against specific, consistent criteria. It doesn't replace CSAT; it complements it. If you're not yet familiar with measuring CSAT, DSAT, and SLA, read that first.

What does QA show that CSAT doesn't?

• Accuracy of information. A customer may not realize the answer they got was wrong.
• Process compliance. For example, verifying identity before sharing account details.
• Tone and courtesy, even when the problem was solved.
• Cases where the customer never leaves feedback. Not every customer answers a survey.
• Training gaps. If several agents are weak on the same criterion, the problem is training, not people.

The scorecard: how to write the criteria

A scorecard is the list of criteria each conversation is measured against. A few principles:

1. Keep criteria few. Five to eight is usually enough. A twenty-criterion card gets tedious, and reviewers start scoring carelessly.
2. Make every criterion observable. "The agent was polite" is vague; "addressed the customer by name and asked at the end whether there was anything else" is checkable.
3. Weight them. Not all criteria matter equally. Give accuracy and process compliance more weight than the use of a particular phrase.
4. Set a passing score, and have one or two "critical" criteria where failing alone means failing the review (for example, disclosing account information).
5. Keep the scale simple. Yes/no or three levels beats ten levels nobody can tell apart.

A simple starting example (fully adjustable):

Sampling: which conversations to read

You can't read everything. Two practical rules:

• A random sample from each agent, so you get a fair picture and don't only look at bad conversations.
• Always include low-CSAT conversations; that's where you'll learn the most.

Hodhod has automatic sampling: it picks a percentage of conversations, and low-CSAT conversations are always included.

Reviewing in practice

1. The reviewer reads the whole conversation, not just the last message.
2. They score each criterion and write one sentence of explanation, especially when the score is low.
3. The result is shown to the agent.
4. The agent can acknowledge it or dispute it.

The last point matters. A QA program where the agent has no right to respond quickly turns into a tool of control and distrust. In Hodhod, agents see their own reviews and can acknowledge or dispute them.

How to give feedback

• Talk about behavior, not the person. "In this conversation you didn't ask for the order number" is better than "You're not careful."
• Give one specific example, and if possible suggest a better sentence.
• Start with strengths. It doesn't need to be flattery, but the agent should know what to keep doing.
• One or two corrections per session, not a long list.
• Look at root causes too. If several people are weak on a criterion, maybe a canned reply or an instruction is unclear. Review your canned responses and fix the onboarding for new agents.

Calibration: getting reviewers aligned

If two reviewers score the same conversation differently, agents start doubting the fairness of the system. Once a month, have all reviewers score the same few conversations separately, then discuss the differences together. Most disagreements show that a criterion's description is unclear and needs rewriting.

How to read the reports

Hodhod shows the review report by agent, criterion, and week, along with pass rate and coverage (what share of conversations were reviewed), and offers CSV export. A few notes on reading it:

• Don't forget coverage. An average score from three reviews doesn't mean much.
• Look at criteria, not just people. A criterion everyone is weak on is usually a system problem.
• The trend matters more than the number. Compare improvement across several weeks.
• Read QA alongside other metrics. Check whether it agrees with your support KPIs and CSAT.

Access to QA management can also be delegated through custom roles (the qa_review_manage permission); not every reviewer needs to be a full admin.

To see conversations and tickets in one panel, take a look at the ticketing system page.

Common mistakes

• Using QA to punish. Agents start hiding their mistakes.
• A scorecard that's too long. Nobody fills it in carefully to the end.
• Reading only the bad conversations. It creates an unfair picture.
• Reviews without feedback. A score that never reaches the agent changes nothing.
• Never updating the card. After a few months, revisit the criteria against what has actually become important.

Frequently asked questions