Recorded-call evaluation guide
How should you evaluate an AI voice-agent call?
Start with what the agent was expected to do in the conversation, not a general impression of how natural it sounded. Use recordings, an explicit rubric and reviewer-approved examples to check what happened, where the evidence is uncertain and what needs another review.
AI-powered QA and QA of an AI agent are different questions
AI-powered QA describes how an assessment is produced. QA of an AI voice agent describes whose behaviour you are assessing. A recorded-call evaluation should make both clear: what evidence the assessor receives and which conversation rules the automated caller is expected to follow.
This guide concerns recorded conversations. It does not establish how a particular provider monitors live calls, connects to your agent platform or verifies actions in other systems. Ask for a demonstration of the workflow you need rather than infer those capabilities from the label AI QA.
Turn conversation expectations into review questions
Choose criteria that match your campaign and the stage the call reached. Define what would count as meeting a criterion, an applicable failure, an unreached stage or evidence that cannot be assessed. These are questions to agree and test, not a promise that every check is automatically available or accurate.
Purpose and opening
Did the agent explain why it was calling and answer a request to clarify the purpose?
Use of earlier answers
Did it use information already given, rather than re-ask a settled question? Distinguish repetition from a necessary clarification.
Ambiguity and question structure
Did it clarify an ambiguous answer before relying on it, and ask questions in a sequence the customer could follow?
Recommendation and recap
Did the explanation use the stated needs and match the approved offer facts supplied for review? Did the recap preserve any conditions?
Response to the customer
What did the agent do after a refusal, request for time, correction or interruption? Compare the observable response with your agreed conversation rules.
The next step
Was a callback or application step proposed, explicitly agreed or still conditional? Keep what was said separate from whether an external system completed the action.
Know what a recording can establish
A recording can support a review of what was said, the sequence of the exchange and, when clear enough, audible overlap, cut-off speech or silence. Check speaker attribution, timing and the full ending before drawing a conclusion.
A pause, tone or quiet voice does not establish the customer's willingness, comprehension or state of mind. A spoken promise does not verify a booking, order, CRM update or successful tool call. Those questions need the relevant system records and a separate verification process.
If response timing matters, agree the source timestamps, turn boundaries and exclusions before measuring it. Estimates from listening are not instrumented latency measurements. Access to audio, supplying audio to a model and correctly using an audible cue are separate things to verify; none alone proves an advantage over an accurate full-context transcript.
Build a test your reviewers can check
Report the full planned sample, processing failures and any diagnostic exclusions. A criterion-level comparison is not the same as a call-level acceptance test. EchoAudit's published calibration guarantee uses every call in the agreed sample; failed or unassessable calls do not count as agreement.
- Agree recording rights and mandatory access, processing, retention and deletion arrangements before transferring customer calls. Use permitted synthetic examples until those controls are satisfied.
- Freeze a representative recording set and the rubric version. Include ordinary exchanges, early endings, ambiguous answers and difficult audio, not just successful examples.
- Have reviewers record expected decisions and supporting evidence before seeing automated results. Agree how reviewer disagreements will be resolved.
- Define the comparison unit, matching rule, tolerances and treatment of unavailable evidence in advance. Check decision agreement and evidence correctness separately.
- Account for each recording in the input manifest with one current processing state. Keep completed, failed, held and other unresolved states distinct; retain rejection reasons and failed-attempt history.
- Record the specific behaviour to change or question to investigate. If the agent is changed, preserve the version and re-test against the agreed references and a separate evaluation set. A review finding is not proof that a change was made or that performance improved.
Ask a provider to show the actual workflow
Which recording formats and review criteria are supported? Does each assessment stage receive audio, a transcript or a summary? What happens when a recording cannot be assessed? Which results can you export, and how are identifiers and export failures handled?
Ask the provider to separate demonstrated product features, human-assisted pilot work and proposed custom integration. A general ingestion endpoint is not, by itself, a supported developer API; a recording review is not proof of live monitoring or verified backend actions.
Discuss whether your recordings fit the pilot
Bring your recorded sales conversations and rubric. We agree the review scope with you, including whether an AI voice-agent recording set is suitable for the pilot.
Before processing, confirm the supported input, criteria, reviewer responsibilities and delivery scope. Read the published pilot offer and guarantee terms; suitability for an AI voice-agent recording set must be agreed rather than assumed.