Measure how well it answers. Not just how often.
Callario scores every resolution your AI agent earns against a quality rubric, samples your team’s conversations the same way, and counts a resolution only when the customer confirms it and it stays solved. Test sets gate every agent release; CSAT, SLAs and every model call sit in the same reports.
- Rubric review on every verified AI resolution
- Resolutions counted only when the customer confirms
- CSAT split by AI, AI-then-person and human
Verified-resolution funnel
- AI conversations1,240
- Answered868
- Resolution proposed531
- Verified412
CSAT · who handled it
- AI-handled91%
- AI, then a person84%
- Human89%
Top failed intents
- Cancel a subscription14
- Warranty on refurbished items9
- Change a delivery slot6
Illustrative data
01AutoQA
Every conversation scored the same way.
Whoever handled it.
A model reviews finished conversations against one published rubric: did the problem get handled, does every claim match the knowledge that was cited, was the customer treated well, was every question answered. Every verified AI resolution is scored, and a steady sample of human-handled conversations is scored beside them.
- Four weighted categories and five named risks, versioned
- A 5% sample of human conversations — up to 25 a day
- Any category at 40 or below goes to the review queue
- Resolution completeness40%92
Address changed before dispatch; the customer confirmed.
- Accuracy vs cited knowledge25%88
Cut-off time matches “Shipping cut-off times”.
- Tone & empathy20%80
Clear and warm; one answer ran long.
- Question coverage15%70
Gift-wrap question deferred to the team, not answered.
No risks found: rudeness, unanswered questions, unkeepable promises, wrong information, exposed personal data.
In the review queue
Omar · human-handled sample · Question coverage 35
A customer question that was never answered.
Scores are a model’s opinion, not a measurement — open the conversation before acting.
02Verified resolution
A resolution counts when the customer says so.
And only if it holds.
After an answer, the agent asks one fixed question — “Did that solve your problem?” An LLM judge reads the reply together with the whole turn, deterministic guards stop an “ok” or a “thanks” from counting, and the verdict waits out a reopen window. If the customer comes back, it is revoked.
- Negation read in English and Arabic, Gulf and Egyptian included
- When the judge is unsure, the answer is “not verified”
- A reopen inside the window revokes the verdict
Done — your order now ships to the new address tomorrow. Did that solve your problem?
thanks
Yes — it shows the new address now.
03Evals
Test the agent before your customers do.
Every release.
Keep a golden set of questions with the answers you expect. Each case is asked as the agent itself — its instructions, knowledge and handoff rules — and an LLM judge scores the answer for correctness and for grounding in the sources it used. Publishing stops when a critical case or required phrase fails, or the average drops below 0.6.
- Runs automatically on every publish, or on demand
- A teammate’s flagged answer becomes a test case in one step
- Grounding is scored against the sources the answer actually used
- How do I change my delivery address?0.920.95
- Can I return an opened item?Critical0.880.90
- When is your team available?Required phrase0.900.86
- هل يمكنني تغيير عنوان التوصيل؟0.860.91
- Do you ship to Oman?0.520.38
One non-critical miss is listed for review; version 5 may publish.
Flagged by Sara · Incorrect
“Do you ship to Oman?” — the answer missed the regional delivery article.
Added to the golden set for the next eval run
04Everything in the box
The whole picture of your support.
AI and team, side by side.
- AI quality reportVerified and unverified resolutions, reopen rate, handoffs and failed intents.
- Agent analyticsConversations, verified resolutions and their rate, answered rate and handoff rate.
- AutoQA rubricResolution, accuracy against cited knowledge, tone and question coverage.
- Review queueLow scores and named risks, with a link straight into the conversation.
- LLM-judge evalsGolden cases asked as the agent, scored for correctness and grounding.
- Verified resolutionsCustomer-confirmed, guarded against bare acknowledgments, revoked on reopen.
- CSATScores and comments, split by AI-handled, AI then a person, and human.
- Team performance & SLAsMedian first response and resolution, first response within SLA, breaching now.
- Tool & guardrail healthCalls, errors and denials per tool; checks and blocks per guardrail.
- AI observabilityEvery model call by feature and model, with token counts.
- Run tracesFor any AI conversation: sources retrieved, checks run, tools called, the decision.
- CSV exportTake the report for any window into a spreadsheet.
Not built yet
- AI-written insights
- Reports through the public API
05Straight answers
What quality leads
ask us first.
Who decides that a resolution is verified?
Are the QA scores objective?
Does it review our human team as well as the AI?
Does Callario write insights or recommendations for us?
Can we see why the AI answered the way it did?
Can we get the data out?
Put it in front of
real customers.
We’re onboarding a small group of teams ready to run their support on Callario’s AI — and to tell us where it breaks. Bring your real conversations; we’ll bring the platform and the people who build it.
- No training on your data
- English and Arabic
- Human takeover, any time
What design partners get
- The whole platform, set up with youAI agent, copilot, workflows, inbox and knowledge — configured around your real conversations.
- A direct line to the buildersTalk to the people who write the code, not a support queue.
- A say in what ships nextYour feedback shapes the roadmap, week by week.