A practical guide
How to check whether AI call scoring matches your standards
Calibration compares AI assessments with reviewer-approved expectations on the same calls. The aim is to identify where the rubric, evidence or model interpretation needs attention before scores inform decisions.
Start With A Decision, Not An Overall Score
Write down what the assessment will inform: coaching, review prioritization or escalation. Decide who checks uncertain evidence and which decisions must remain with a human reviewer. Do not treat an AI quality score as proof of a completed sale, activation or legal compliance.
Build A Reference Set You Have Permission To Use
Choose calls that reflect your campaigns and recording conditions. Include successful conversations, unsuccessful conversations, early exits, voicemail, poor audio and examples that reviewers find difficult. Keep an untouched evaluation set separate from the calls used to edit prompts or rules. Preserve failed processing and unavailable evidence in the report rather than selecting only successful examples. Recording rights, access and retention must be agreed before transfer.
Write An Observable Criterion
For every criterion, record: the behaviour required; when it becomes applicable; what evidence is sufficient; what counts as met, partly met or failed; and how to record uncertainty or an unreached stage. Keep “not reached,” “not applicable” and “cannot assess” distinct in the evaluation record even if a product's native labels differ. A missing sentence is not automatically a failure when the call ended before that stage.
Establish Human Reference Decisions
Have reviewers label the calls independently before they see the AI findings. Record supporting evidence, compare decisions and adjudicate differences. Retain unresolved cases as unresolved, with a reason; do not silently convert them into agreement. Define who approved the reference and whether labels came from people, prior model outputs or a mixture. Rerunning a previous model answer is not independent ground truth.
Compare Decisions And Evidence Separately
For each assessable criterion, compare the AI decision with the approved reference. Also check whether its quoted evidence occurs in the source and supports the decision. Report false alarms and missed findings separately for important criteria. Track wrongly excluded criteria as applicability errors so removing difficult cases cannot inflate agreement.
Report the denominator and exclusions
Illustrative arithmetic only, not an EchoAudit result: suppose there are 40 planned criterion comparisons. Reviewers approve 30 as assessable, mark 6 not applicable and leave 4 unresolved because the audio is unclear. If the AI matches 24 of the 30 assessable reference decisions, observed agreement is 24/30, or 80%, with 6 disagreements. Report all 40 planned comparisons and each exclusion category. This result does not describe the entire cohort, imply 80% call-level accuracy or establish performance on future calls. Human reference categories should determine this denominator before reviewing AI outputs; separately report AI failures and abstentions.
Record What Changed
Keep the rubric, prompt, model and assertion versions with the source-set identifier and review date. After a change, rerun the agreed benchmark and examine the separate evaluation set. Define acceptable differences before seeing results. A regression pass means the configured checks passed; disclose tolerances and do not rename it “accuracy.” Record time and cost for failures, retries and human adjudication as well as successful runs.
Decide Whether The Workflow Fits
Ask the vendor to demonstrate your input format, role permissions, review process, failure handling and output export. Confirm deployment location, subprocessors, retention/deletion, recording access and contractual terms through verified documentation. If these details are unavailable, mark them unknown rather than assume them. Choose a product based on the evidence you can inspect and the risks you can manage, not a promised overall score.
How EchoAudit supports this
EchoAudit provides configurable QA frameworks and a test suite with benchmark expectations. These support comparison and investigation; the quality of the benchmark still depends on the examples and reviewer decisions you supply.
Use the blank evaluation worksheet
Download the CSV and field guide below to record one planned case-and-criterion comparison per row. The file contains column headings only, with no customer data or measured results. It is a manual evaluation worksheet; it does not imply an EchoAudit import feature, automated reviewer consensus or automatic metric calculations.