AI call QA evaluation worksheet
Version 1.0 — 20 October 2026
Guide: https://echoaudit.co/guides/ai-call-scoring-calibration

Purpose
Use this blank CSV for manual evaluation. It contains headings only, no
customer data, executed benchmark or product performance results. It is not
an EchoAudit import format and includes no formulas or automatic metrics.

Create one row per planned case-and-criterion comparison per evaluation run.
Freeze the planned set and human reference categories before inspecting AI
outputs. Keep failures, abstentions and unresolved references in the record.
Use new evaluation_id values for reruns; do not overwrite earlier outcomes.
Keep the rubric's observable behaviour, applicability rule and decision
definitions in a separate versioned rubric identified by rubric_version.

Field definitions
evaluation_id: Your versioned evaluation/run identifier and source-set reference.
case_ref: A pseudonymous or synthetic case identifier, such as synthetic-001.
criterion_id: Identifier of the criterion in the versioned rubric.
campaign_or_call_type: Descriptive category used for sampling and subgroup checks.
source_permission_status: approved, pending, denied or unknown. Do not transfer
  recordings without the required permission; document its scope separately.
rubric_version: Exact version of rules, applicability and decision definitions.
prompt_version: Prompt version or identifier used for this run; unknown if absent.
model_version: Provider/model/version identifier used, with date if unversioned.
reference_origin: independent_human, model or mixed. Only independently reviewed
  and approved human labels support an independent human-reference comparison.
reference_applicability: assessable, not_reached, not_applicable or cannot_assess.
reference_decision: Approved rubric decision, such as met, partly_met or failed.
  Leave blank if no approved decision exists; explain why in notes.
reference_evidence_locator: Timestamp or line reference within your controlled
  source set, not a public recording URL or customer excerpt.
reviewer_a: Pseudonymous first reviewer identifier; retain their independent
  labels and evidence in a separate controlled reference record.
reviewer_b: Pseudonymous second reviewer identifier; retain their labels likewise.
adjudication_status: agreed, adjudicated, unresolved or pending. Record approval
  and unresolved reasons in the controlled reference record and notes.
ai_applicability: assessable, not_reached, not_applicable, cannot_assess or unknown;
  map native product labels explicitly and preserve the mapping with this run.
ai_decision: AI's rubric decision, or abstained; blank if processing produced none.
ai_evidence_locator: Timestamp or line reference claimed by the AI, without
  publishing customer excerpts or private source links.
evidence_verified: yes, no, pending or not_applicable. Yes requires checking that
  the evidence exists in the source AND supports the decision.
assertion_kind: exact_decision, numeric_tolerance, applicability, evidence or none.
  If several checks apply, record their definitions/results in a linked controlled
  record; do not duplicate the comparison row and inflate its denominator.
assertion_tolerance: Predefined tolerance with units; use exact or not_applicable
  where appropriate. Keep the assertion version in evaluation_id or notes.
comparison_result: match, mismatch, abstained, processing_failed, pending or
  excluded_reference. A regression check outcome belongs in notes, separately
  from decision agreement, unless the check is an exact decision comparison.
error_category: none, false_alarm, missed_finding, applicability_error,
  unsupported_evidence, processing_error, other or unknown. Record additional
  categories in notes when more than one applies; define positive findings first.
processing_status: not_run, completed, failed or partial. Completed does not mean
  correct. Preserve retry/failure details in the run record.
review_seconds: Human review/adjudication seconds allocated to this row.
processing_cost: Processing cost allocated to this row. Declare one currency in
  the run record and notes. Include failures/retries; do not repeat a call's full
  cost on every criterion row. Leave blank if no defensible allocation exists.
review_date: Review date in YYYY-MM-DD format; blank until reviewed.
notes: Unresolved reasons, mappings, approval/check versions and limitations.
  Use pseudonymous references; do not include personal data or secrets.

Categories and missing values
Not_reached means the call ended before the stage. Not_applicable means the
criterion did not apply. Cannot_assess means evidence is insufficient. These
categories are distinct from an assessable failed criterion. An unclear or
unadjudicated reference remains unresolved; do not count it as agreement.
Blank means unrecorded, not zero or not applicable. Use 0 only for a known zero
time/cost. If a field is unavailable, use unknown where its category allows it,
otherwise leave it blank and explain in notes. CSV has no validation controls.

Manual reporting
Report the planned comparison count, unique call count, reference categories,
unresolved/pending counts, processing failures, AI abstentions and versions.
Agreement numerator: exact AI/reference decision matches on the predefined
human-approved assessable comparisons. Denominator: ALL those comparisons,
including AI processing failures, abstentions and wrongful AI exclusions.
Do not let AI applicability decide which human-assessable rows count. Report
each failure/abstention separately so agreement is not confused with coverage.
Use n.a. if the denominator is zero. Report counts beside every rate.

Check evidence support independently from decision agreement. For a defined
positive finding, a false alarm flags a finding the reference rejects; a miss
omits one the reference confirms. Define the criterion, positive class and
rate denominator before reporting these measures. No metrics are calculated
by this file. A regression pass means only the specified checks/tolerances
passed, and is not an empirical accuracy result.

Illustrative arithmetic only: 40 planned comparisons comprise 30 approved
assessable, 6 not applicable and 4 unresolved. With 24 exact matches, agreement
is 24/30 = 80%. Report all 40 and their categories. This is not an EchoAudit
result, call-level accuracy estimate or guarantee of future performance.

Keep names, phone numbers, customer excerpts, raw recording URLs, access tokens
and private organisation IDs out of shared copies. Use a controlled source set
with agreed access and retention. Record limitations rather than invent data.