Skip to content

Auto QA operations

Auto QA scores conversations against scorecards automatically. From the operator's perspective it is not a separate service: Auto QA is an AI task whose destination is Auto QA, executed by the AI Assistant job like every other task in the pipeline. This page covers how it runs and how to verify it per tenant; the tenant-side setup is in the Administration Guide.

How Auto QA runs

  • An Auto QA task links a prompt to exactly one evaluation form (scorecard). The scorecard's questions and grading guidance are supplied to the AI automatically; the result is stored as a scored evaluation on the conversation.
  • The task runs when the AI Assistant job processes the conversation — after transcription, and after the job's other destinations (Auto QA is deliberately last in the fixed destination order, so summaries, sentiment, and custom fields are already in place).
  • Licensing: a conversation is auto-scored only if its assigned user holds the Speech analytics license (target name: Conversation Analytics license). The evaluation license gates manual evaluation work, not Auto QA.
  • Filters: the task's filtering criteria (and the job's) decide which conversations are scored; conversations that already have an evaluation for the scorecard are skipped.
  • Auto QA tasks can be global with per-tenant overrides, like any AI task: a shared global scoring task can be enabled, disabled, or re-prompted per tenant from the Overrides tab.

Auto QA consumes LLM tokens against the tenant's token limits — heavy scorecards on high volumes are usually the largest token consumer on the platform. Watch the Usage tab of the AI Tasks page (see AI tasks).

Verifying results per tenant

After enabling an Auto QA task for a tenant:

  1. Confirm the task state — on Administration › Conversation Analytics › AI Tasks, check the tenant's rollup and the Overrides tab: is the task enabled for that tenant, and is the prompt Default or Overridden?
  2. Run one conversation through the playground — the Auto QA form experiment in the AI Playground scores a real conversation of that tenant against the scorecard without persisting anything.
  3. Check the job — the Jobs tab shows the AI Assistant job's runs, processing records, and logs; license errors and engine failures surface here (same workflow as transcription monitoring).
  4. Confirm evaluations appear — in the tenant's scope, auto-scored evaluations show up under QA › Evaluations and on each conversation's QA tab as new conversations are processed. What supervisors see and how scores are calculated is documented in the User Guide QA chapter and Understanding Auto QA.

The legacy auto-score job

The jobs catalog still lists a standalone job type named Auto QA (legacy). It is retired: it predates the AI-task pipeline and is not how Auto QA runs today. Do not create new instances of it — configure Auto QA as an AI task with the Auto QA destination, as described above.