By Question Metric¶

The By Question metric is a key feature of the Quality AI module, managed within the Evaluation Forms section. It enables supervisors to define, customize, and track performance indicators that measure the quality of agent–customer interactions, specifically at the level of individual questions. This metric helps you to evaluate how well agents answer specific questions during interactions. You can apply it universally across all conversations or use it selectively in trigger-based scenarios.

By verifying the accuracy of responses, it supports focused feedback, targeted coaching, and continuous improvement.

What It Offers for Supervisors¶

Standardized quality assessment framework.
Customizable evaluation criteria based on business needs.
Multi-language support for global operations.
Automated evaluation suggestions through AI.

When to Use This Metric¶

Primary Use Cases:

Quality Assurance: To systematically evaluate agent adherence to protocols, scripts, or expected responses.
Training Assessment: To measure how well agents follow prescribed interaction patterns during customer conversations.
Compliance Monitoring: To verify, how agents deliver critical information, such as disclaimers, privacy policies, or regulatory statements (privacy policies, disclaimers).
Performance Standardization: To apply consistent evaluation criteria across agents and interactions.

How It Works¶

The metric operates through a question-driven evaluation process with two main approaches:

Static Adherence¶

Evaluates agent responses across all conversations without conditional requirements.
Best for universal standards like greetings or standard procedures.
No trigger conditions needed.

Dynamic Adherence¶

Evaluates agent responses only when specific triggers occur.
Trigger-based detection using customer or agent utterances.
Ideal for conditional scenarios where specific responses are required only in certain contexts.

Configure by Question Metric¶

Navigate to Quality AI > Configure > Evaluation Forms > Evaluation Metrics.
Select + New Evaluation Metric.
From the Evaluation Metrics Measurement Type dropdown, select By Question.
Enter a descriptive Name for future reference of the metrics. (for example, agent’s warm greeting).
Select a preferred language from the dropdown to evaluate the form and configure the required metric.
Enter an evaluation Question to help supervisors assess agent adherence as a reference for the audits and interaction reviews.
Select the agent Adherence Type (Static or Dynamic).
Note
- For Static, you must configure at least one agent answer utterance for the Adherence Type.
- For Dynamic, you must configure at least one trigger and one agent answer utterance for the adherence type.

Adherence Type Configuration¶

The following are the key trigger components that you can configure:

Customer Utterance: Evaluation initiated by customer statements.
Agent Utterance: Evaluation triggered by agent responses.
Multiple Triggers: Support for complex conditional scenarios.
GenAI Suggestions: Automated variation generation.

Static Adherence¶

Static Adherence measures whether agents say the required phrase, regardless of what triggers the conversation. It applies to all calls, without any condition or contextual trigger.

Use this adherence when you need a consistent, universal check (for example, mandatory greeting scripts, regulatory disclaimers).
You must define acceptable utterances for a queue.
You set a similarity threshold to evaluate whether the agent’s actual response matches the pre‑defined acceptable utterance.
No triggers or contextual conditions needed, which is ideal for fixed and non‑situational compliance items.
You must configure at least one agent utterance template for this adherence type.

Dynamic Adherence¶

Dynamic Adherence evaluates agent performance based on specific triggers (context-sensitive method) instead of reviewing every interaction. It assesses agent behavior when the configured conditions occur.

The system checks adherence only when it detects a configured trigger, such as an agent or customer utterance.
You must define at least one trigger (either customer or agent utterances) and one acceptable agent response that activate the adherence check.
Evaluates agent behavior only in the context of the detected trigger.
Determines how closely an agent's response must match a predefined response.
Adjustable by criticality of the adherence similarity use case:
- Lower Threshold: Must close to 60% (Yellow) for casual interactions, greetings.
- Higher Threshold: Must close to 100% (Green) for critical topics, such as legal disclaimers or privacy policies.
- When the system detects a trigger, it evaluates whether the agent responded with one of the predefined acceptable utterances.
  
  Note
  
  For Dynamic, configure at least one Trigger and one agent Answer utterance for the adherence type.
Choose utterance source for Trigger (who initiates the trigger).

Trigger Configuration¶

Provides the following two options to select based on the trigger created by Agent Utterance or Customer Utterance for evaluation.

* **Trigger**: Select and evaluate responses based on triggers created from agent or customer utterances.

    * You can add multiple trigger utterances for conditional checks (for example, AND/OR conditions) to serve as the trigger condition or refine when the trigger is activated.

    * You can define a trigger for either a customer or agent utterance and configure the corresponding responses for that scenario.      
    <img src="../images/by-question-trigger.png" alt="By Question Trigger" title="By Question Trigger" style="border: 1px solid gray; zoom:70%;">

    * **Customer Utterance**: Configure the Customer Utterance that triggers the adherence check. You can enter more than one utterance using Generative AI Assistants that are similar to utterances with the same meaning. For example, customer says “I need a refund”.

    * **Agent Utterance**: Configure the Agent Utterance for triggers initiated by the agent. Enter the utterances using generative AI Assistants’ suggestions that have similar utterances with the same meaning. You can add or delete multiple utterances for the Customer and the Agent. For example, agent says “let me transfer you to support”.

Choose a Trigger Detection Method.
GenAI-Based Adherence: Enter a descriptive prompt to define intent. This allows contextual detection of whether the agent’s response aligns with the intended goal, without relying on predefined samples. This uses LLMs to understand context and intent and do not require sample utterances or thresholds.
Deterministic Adherence: This relies on predefined sample utterances and detects adherence based on semantic similarity.
Utterance: Enter sample training utterances to detect adherence and define similarity thresholds accordingly.

Enablement of GenAI-Based Features (Pre-requisite)¶

The GenAI-based features are activated only when the following conditions are enabled:

a. Navigate to **Manage**> **Generative AI**> **GenAI Features**.

b. [Enable](../../../../generative-ai-tools/genai-features.md){:target="_blank"} and [Publish](../../../../deploy/publishing-bot.md/#publishing-components){:target="_blank"} the following two features:

    * **GenAI-based agent answer adherence**

    * **GenAI-based customer trigger detection**  
    <img src="../images/gen-ai-based-agent-answer-adherence-with-trigger.png" alt="GenAI-based Features" title="GenAI-based features" style="border: 1px solid gray; zoom:70%;">

Choose an Agent Answer adherence type.

Agent Answer¶

GenAI-Based Adherence: Use AI and natural language understanding (LLM) to detect meaning, context, and intent. This evaluates whether the agent's answer fulfills the intents, even if phrased differently.
Deterministic Adherence: This relies on an ML-based method using the semantic similarity.
Description: Enter a prompt Description to check the agent's intent accuracy, phrasing response, and adherence.

GenAI-Based Adherence¶

Uses Generative AI to automatically evaluate agent responses by understanding natural language, including intent and context, even when phrased differently.

Agent Answer

a. Select an **Answer Detection Method** to evaluate whether agents respond according to the prompt’s intent. This uses a probabilistic LLM-based method and requires no model training.

b. Enable the **GenAI-Based Adherence**. Use a **Large Language Model** (LLM) to detect trigger phrases and evaluate adherence using contextual understanding.

c. Enter a prompt **Description** explaining the metric’s intent or details behind the adherence metric. This applies to all selected languages.   
<img src="../images/by-question-agent-answer.png" alt="Agent Answer" title="Agent Answer" style="border: 1px solid gray; zoom:70%;">

!!! Note

    * Before assessing GenAI-based adherence responses, ensure that the [Supported models](../../../../generative-ai-tools/genai-features-qualityai.md){:target="_blank"} and [GenAI features](../../../../generative-ai-tools/genai-features.md){:target="_blank"}, including GenAI-based agent answer adherence and customer trigger detection, are enabled for the respective products in the GenAI features section.

    * No example utterances or similarity thresholds are required; LLMs evaluate adherence contextually using zero-shot prompts.

    * For effective prompts and LLM-based adherence detection, refer to
    [AutoQA - Prompting Guide](autoqa-prompting-guide.md){:target="_blank"}.

Deterministic Adherence¶

Evaluates agent responses based on semantic similarity to predefined reference utterances or answers.

Agent Answer Configuration¶

a. Select **Deterministic Adherence** to assess responses based on similarity to reference answers. Encoder-based mode lets you define expected replies.

b. Define an **Answer** as a set of acceptable utterances for each queue, using **Generative AI** to generate the following automated response variations:

c. Set a similarity threshold to determine how must the user input must match the expected utterances to return agent answers.

d. Add language-specific, prompt-based evaluation parameters.

e. Find relevant answers by suggesting different ways to ask the same question.

f. Get the expected answers that match the meaning of your question, even though asked in different ways. Provides expected answers that match the meaning of your question, even when you phrase it in different ways.

g. Delete AI-suggested answers that are not required.

h. Set the **Similarity** percentage for the metric based on the defined use case and attribute.

Similarity Thresholds¶

Evaluates agent responses based on semantic similarity to predefined sample utterances or answers.

You can set the Similarity percentage for the desired evaluation metrics. Whether it's Static or Dynamic, you can configure the expected Similarity threshold.
You must set a lower adherence similarity threshold (for example, 60%) for soft skills like greetings and etiquette, and a higher adherence similarity threshold (for example, 100%) for compliance-critical (Policy Privacy or Disclaimer) statements, because it's critical for the user to follow the adherence depending on the use cases.

Note

The Similarity threshold option is available only when GenAI-Based Adherence is enabled. You can configure thresholds for both Static and Dynamic evaluations.
Choose a Count Type based on the selected Adherence Type.

Count Type Configuration¶

The following are the key count types that you can configure:

Entire Conversation: Full interaction evaluation
Time Bound: Specific time ranges or message counts

Entire Conversation¶

Evaluates adherence throughout the complete interaction. This lets you to check for adherence at different points of conversation. It doesn't matter where the agent wants to check adherence throughout the conversation.

Time Bound¶

Focuses on specific timeframes (first or last X seconds or messages). This evaluates adherence within a specific time range or number of messages in the interaction. It can occur at the start or end of the conversation, either for a defined number of seconds or a set number of chat messages.

Parameter: Choose between the First Part of Conversation or the Last Part of Conversation, and configure the relevant criteria to evaluate the metric.
Voice: Enter the number of seconds from the start or end of the interaction to evaluate this metric.
Chat: Enter the number of messages from the start or end of the interaction to evaluate this metric.
Select Create to save and activate the By Question configured adherence metric.

Edit or Delete By Question Metric¶

Steps to edit or delete any existing By Question evaluation metrics:

Select a required evaluation metric name given in the By Question category.
Choose an option:
- Select Edit to modify the selected metric details.
- Select Delete to remove the selected metric.
Select Update to save the changes.

Language Dependency Warnings¶

This section outlines the limitations and dependencies associated with modifying language settings in evaluation metrics.

Modification Warnings¶

You must keep a language if any evaluation form or attribute uses it.
Remove the language from all associated evaluation forms and attributes before modifying their language settings.
Remove the languages that are not linked to any forms or metrics.

Delete Warnings¶

This section describes the warnings and prerequisites you must address before deleting a metric.

If any evaluation form uses the metric, the system displays a warning message.
Remove the metric from all associated evaluation forms before you delete it.
If any attributes link to the metric, assign a different metric to them before deleting the original.
The system lets you delete the metric only after resolving all dependencies.

Send Feedback