By AI Agent Evaluation Metric

The By AI Agent metric type enables you to configure evaluation metrics powered by AI agents that can understand complex requests, perform multi-step reasoning, and make autonomous decisions through natural language processing. This metric type is designed for sophisticated evaluation scenarios that require domain expertise and advanced analytical capabilities beyond standard Generative AI metrics.

When to Use By AI Agent Metric

Use By AI Agent metrics for evaluation scenarios that require:

• Multi-step reasoning and complex analysis: Evaluation scenarios require connecting multiple pieces of information across a conversation.

• Domain-specific expertise: Evaluations that need specialized knowledge in areas like compliance, technical support, or industry-specific protocols.

• Knowledge-based verification: Scenarios where the AI agent must search through knowledge bases, documentation, or external sources to verify factual accuracy, compliance adherence, or policy conformance against established information repositories.

• Ground truth validation using tools: Cases where specialized tools are required to cross-reference information against authoritative sources, validate the correctness of responses or recommendations, and ensure adherence to specific standards or protocols.

• Comprehensive contextual understanding: Situations where the full conversation context and nuanced understanding are critical for accurate evaluation.

• Advanced decision-making capabilities: Complex judgment calls that require sophisticated reasoning beyond simple pattern matching.

AI Agent vs Gen AI Metrics Comparison

AI Agent Metrics	Gen AI Metrics
High - handles multi-step reasoning and domain expertise.	Low to Medium - pattern matching and similarity detection.
Requires external AI agent configuration and API integration.	Built-in configuration with predefined parameters.
Autonomous decision-making with custom logic.	Rule-based or similarity-based evaluation.
Nuanced compliance checks, complex quality assessments.	Adherence verification, standard response validation.

Prerequisites

To configure By AI Agent metrics, ensure the following prerequisites are met:

• The By AI Agent metric type is in private beta.

• You have access to Kore.ai's AI Agent platform and a deployed agent.

• You have the endpoint URL and API key for your AI agent service.

• Your AI agent is configured to provide Quality AI-compatible responses.

  Note

  The By AI Agent metric type is currently in private beta. To enable this feature:

  • Contact Kore.ai support through the support portal.

  • Provide your workspace account ID in the request.

Configuring By AI Agent Metric

1. Navigate to Contact Center AI > Quality AI > Configure > Evaluation Forms> Evaluation Metrics.

2. Click + New Evaluation Metric.

3. From the Evaluation Metrics Measurement Type dropdown, select By AI Agent.
   

4. Enter a descriptive Name for future reference of the metrics.

5. Select the Language from the dropdown to provide language context for the AI agent's evaluation process.
   

6. Enter the evaluation Question that defines what the AI agent should assess.

7. Configure the AI Agent Connection:

   1. AI Agent Endpoint: Enter the API endpoint URL for your external AI agent service.

   2. API Key: Provide the authentication key for accessing the AI agent endpoint.

Note

For detailed instructions on retrieving your AI Agent Endpoint and API Key, see AI Agent Endpoint Documentation.

1. Test Connection: Click the Test Request button to validate your configuration. The system verifies the API endpoint accessibility, authentication, and response format compatibility.

2. Click Create to save the new metric for AI Agent evaluation.

Response format

When configuring your AI Agent in the Agent Platform, you must define the response format in the Description field to ensure proper communication with Quality AI. This format specification tells the AI Agent how to structure its evaluation responses.

Setting up response format

1. Navigate to your AI Agent configuration in the Agent Platform.

2. Locate the Description field.

3. Enter the response format specification as shown in the template below.
   

4. Save your configuration.

   Note

   For detailed instructions on setting up an AI Agent, see the AI Agent creation guide.

Response Format Template

The AI Agent must return responses in the following JSON structure:

Field descriptions

{
  "messageId": "msg-{uuid-here}",
  "output": [
    {
      "type": "text",
      "content": "{
        \"source\": \"{customer|agent}\",
        \"isQualified\": \"{yes|no|NA}\",
        \"messageIds\": [
          \"{message_id_1}\",
          \"{message_id_2}\"
        ],
        \"sourceIds\": [
          \"{source_id_1}\",
          \"{source_id_2}\"
        ],
        \"msgTimestamps\": [
          \"{timestamp_1_milliseconds}\",
          \"{timestamp_2_milliseconds}\"
        ],
        \"justification\": [
          {
            \"aspect\": \"{Evaluation_Aspect_Name}\",
            \"status\": \"{Identified|Not Identified}\",
            \"justification\": \"{Detailed explanation of finding or empty string}\",
            \"message_id\": [
              \"{supporting_message_id_1}\",
              \"{supporting_message_id_2}\"
            ],
            \"timestamp\": [
              \"{supporting_timestamp_1}\",
              \"{supporting_timestamp_2}\"
            ]
          }
        ]
      }"
    }
  ]
}

Field	Type	Description
messageId	String	Unique identifier for the response message.
output	Array	Contains the evaluation results.
type	String	Always set to text.
content	String	JSON string containing the evaluation details.
source	String	Indicates whether evaluation focuses on customer or agent.
isQualified	String	Overall evaluation result: yes, no, or NA.
messageIds	Array	List of message IDs that were evaluated.
sourceIds	Array	List of source IDs corresponding to the messages.
msgTimestamps	Array	Timestamps in milliseconds for evaluated messages.
justification	Array	Detailed breakdown of evaluation aspects.

Justification Object Structure

Each item in the justification array contains:

Field	Type	Description
aspect	String	Name of the specific evaluation aspect.
status	String	Identified or Not Identified.
justification	String	Detailed explanation or empty string if not identified.
message_id	Array	Supporting message IDs for this aspect.
timestamp	Array	Corresponding timestamps for supporting messages.

Configuration Example

When setting up your AI Agent, include this specification in the Description field:

Response Format Requirements:

The response must be a JSON object with the following structure:

- isQualified: String representing overall compliance outcome ("Yes", "No", or "Partial")
- compliance_evaluation_details: Array of JSON objects detailing specific compliance aspects

Each compliance evaluation object should include:
- aspect: Descriptive name of the compliance aspect (e.g., "Data Security Measures")
- status: "Identified" or "Not Identified"
- justification: Detailed explanation of findings
- supporting_evidence: References to specific messages or timestamps

Sample response

An example of a properly formatted response:

{
  "messageId": "msg-b0a1b156-928f-4024-b337-805c193d1472",
  "output": [
    {
      "type": "text",
      "content": "{
        \"source\": \"customer\",
        \"isQualified\": \"yes\",
        \"messageIds\": [\"1\", \"9\"],
        \"sourceIds\": [\"1\", \"9\"],
        \"msgTimestamps\": [\"1746792625000\", \"1746792681000\"],
        \"justification\": [
          {
            \"aspect\": \"Agent Greetings\",
            \"status\": \"Identified\",
            \"justification\": \"The agent starts the conversation with a greeting: 'Thank you for calling Premier Banking Services. This is Michael. How may I assist you today?'\",
            \"message_id\": [\"1\"],
            \"timestamp\": [\"1746792625000\"]
          },
          {
            \"aspect\": \"Customer Satisfaction\",
            \"status\": \"Identified\",
            \"justification\": \"The customer expressed understanding and satisfaction: 'I think you've covered everything. The annual fee is $550, but with the $300 travel credit and lounge access, it seems worth it for me.'\",
            \"message_id\": [\"38\"],
            \"timestamp\": [\"1746792884000\"]
          }
        ]
      }"
    }
  ]
}

Managing Evaluation Metrics

Edit or Delete Evaluation Metrics

Steps to edit or delete existing Evaluation Metrics:

1. Right-click to select any of the existing Evaluation Metrics.
   

2. Click Edit to update the required Edit By Question Metrics dialog box fields.

3. Click Delete to remove the selected By Question evaluation metric.

4. Click Update to save the changes.

Send Feedback