Conversation Evaluation REST API

The Conversation Evaluation REST API provides comprehensive tools for assessing conversation quality, persona performance, and skill effectiveness. Use these endpoints to create evaluation frameworks, run quality assessments, and optimize your conversational AI.

Base URL & Authentication

Base URL: https://your-api-name.mercury.ratiomachina.com
Authentication: x-mercury-api-key: YOUR_API_KEY

Persona Evaluation Jobs

🎯 Persona Evaluation Jobs: Define comprehensive evaluation frameworks for assessing persona performance across multiple dimensions including accuracy, empathy, consistency, and business goal alignment.

List All Persona Evaluation Jobs

GET/persona-eval-jobs

Query Parameters

limit - Number of results (default: 50)
offset - Skip results for pagination
personaId - Filter by specific persona
status - Filter by evaluation status
evaluationType - Filter by evaluation type

Response Example

{
  "personaEvalJobs": [
    {
      "id": "uuid-eval-job-123",
      "personaId": "uuid-persona-456",
      "name": "Q1 2024 Sales Performance Evaluation",
      "evaluationType": "comprehensive_assessment",
      "status": "active",
      "configuration": {
        "dimensions": [
          "accuracy", 
          "empathy", 
          "goal_alignment",
          "consistency"
        ],
        "benchmarks": ["industry_standard", "internal_baseline"]
      },
      "metrics": {
        "totalRuns": 47,
        "avgScore": 8.2,
        "lastRun": "2024-01-15T14:30:00Z"
      },
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ],
  "pagination": {
    "total": 15,
    "limit": 50,
    "offset": 0
  }
}

Create Persona Evaluation Job

POST/persona-eval-jobs

Request Body

{
  "personaId": "uuid-persona-456",
  "name": "Customer Support Quality Assessment",
  "description": "Comprehensive evaluation of customer support persona performance",
  "evaluationType": "quality_assurance",
  "configuration": {
    "dimensions": [
      {
        "name": "accuracy",
        "weight": 0.3,
        "criteria": [
          "factual_correctness",
          "solution_relevance", 
          "information_completeness"
        ]
      },
      {
        "name": "empathy",
        "weight": 0.25,
        "criteria": [
          "emotional_understanding",
          "tone_appropriateness",
          "customer_acknowledgment"
        ]
      },
      {
        "name": "efficiency",
        "weight": 0.25,
        "criteria": [
          "response_time",
          "solution_directness",
          "followup_necessity"
        ]
      },
      {
        "name": "compliance",
        "weight": 0.2,
        "criteria": [
          "policy_adherence",
          "privacy_protection",
          "regulatory_compliance"
        ]
      }
    ],
    "scoringModel": {
      "type": "weighted_average",
      "scale": "1_to_10",
      "passingThreshold": 7.0
    },
    "benchmarks": {
      "compareToBaseline": true,
      "baselinePeriod": "30d",
      "industryComparison": true
    },
    "samplingStrategy": {
      "method": "stratified",
      "conversationTypes": ["support", "billing", "technical"],
      "sampleSize": 100,
      "timeRange": "7d"
    }
  },
  "schedule": {
    "enabled": true,
    "frequency": "weekly",
    "day": "monday",
    "time": "09:00"
  },
  "notifications": {
    "onCompletion": true,
    "onFailure": true,
    "recipients": ["qa-team@company.com"],
    "thresholdAlerts": {
      "scoreDropThreshold": 1.0,
      "recipients": ["manager@company.com"]
    }
  }
}

Response Example

{
  "personaEvalJob": {
    "id": "uuid-eval-job-789",
    "personaId": "uuid-persona-456",
    "name": "Customer Support Quality Assessment",
    "evaluationType": "quality_assurance",
    "status": "active",
    "configuration": {
      "dimensions": [
        {
          "name": "accuracy",
          "weight": 0.3,
          "criteria": [
            "factual_correctness",
            "solution_relevance",
            "information_completeness"
          ]
        }
      ],
      "scoringModel": {
        "type": "weighted_average",
        "scale": "1_to_10",
        "passingThreshold": 7.0
      }
    },
    "schedule": {
      "enabled": true,
      "frequency": "weekly",
      "day": "monday",
      "time": "09:00",
      "nextRun": "2024-01-22T09:00:00Z"
    },
    "createdAt": "2024-01-15T10:30:00Z",
    "updatedAt": "2024-01-15T10:30:00Z"
  }
}

Get Persona Evaluation Job by ID

GET/persona-eval-jobs/{id}

Path Parameters

id - Persona Evaluation Job UUID

Query Parameters

includeRuns - Include recent evaluation runs
includeMetrics - Include performance metrics
includeTrends - Include trend analysis

Response Example

{
  "personaEvalJob": {
    "id": "uuid-eval-job-789",
    "personaId": "uuid-persona-456",
    "name": "Customer Support Quality Assessment",
    "configuration": { ... },
    "metrics": {
      "totalRuns": 23,
      "avgScore": 8.4,
      "trend": "improving",
      "trendChange": "+0.3",
      "lastRunScore": 8.7,
      "bestScore": 9.1,
      "worstScore": 7.2
    },
    "recentRuns": [
      {
        "id": "uuid-run-001",
        "score": 8.7,
        "status": "completed",
        "executedAt": "2024-01-15T09:00:00Z"
      }
    ],
    "trends": {
      "scoreHistory": [
        {"date": "2024-01-08", "score": 8.1},
        {"date": "2024-01-15", "score": 8.7}
      ],
      "dimensionTrends": {
        "accuracy": "stable",
        "empathy": "improving", 
        "efficiency": "declining",
        "compliance": "stable"
      }
    }
  }
}

Update Persona Evaluation Job

PUT/persona-eval-jobs/{id}

Updates evaluation job configuration. Changes take effect on the next scheduled run or manual execution.

Request Body

{
  "name": "Enhanced Customer Support Assessment",
  "configuration": {
    "dimensions": [
      {
        "name": "accuracy",
        "weight": 0.35,
        "criteria": [
          "factual_correctness",
          "solution_relevance",
          "information_completeness",
          "technical_accuracy"
        ]
      },
      {
        "name": "customer_satisfaction",
        "weight": 0.3,
        "criteria": [
          "resolution_effectiveness",
          "customer_effort_score",
          "satisfaction_prediction"
        ]
      }
    ],
    "scoringModel": {
      "passingThreshold": 7.5
    }
  },
  "schedule": {
    "frequency": "daily"
  }
}

Response Example

{
  "personaEvalJob": {
    "id": "uuid-eval-job-789",
    "name": "Enhanced Customer Support Assessment",
    "configuration": {
      "dimensions": [
        {
          "name": "accuracy",
          "weight": 0.35,
          "criteria": [
            "factual_correctness",
            "solution_relevance", 
            "information_completeness",
            "technical_accuracy"
          ]
        },
        {
          "name": "customer_satisfaction",
          "weight": 0.3,
          "criteria": [
            "resolution_effectiveness",
            "customer_effort_score",
            "satisfaction_prediction"
          ]
        }
      ]
    },
    "schedule": {
      "frequency": "daily",
      "nextRun": "2024-01-16T09:00:00Z"
    },
    "updatedAt": "2024-01-15T11:30:00Z"
  }
}

Delete Persona Evaluation Job

DELETE/persona-eval-jobs/{id}

⚠️ Warning: Deleting an evaluation job will permanently remove it and all associated run history. Any scheduled evaluations will stop immediately.

Path Parameters

id - Persona Evaluation Job UUID

Response Example

{
  "message": "Persona evaluation job deleted successfully",
  "deletedJob": {
    "id": "uuid-eval-job-789",
    "name": "Customer Support Quality Assessment",
    "personaId": "uuid-persona-456",
    "totalRuns": 23
  }
}

Persona Evaluation Suites

📊 Evaluation Suites: Collections of related evaluation jobs that can be executed together, providing comprehensive assessment across multiple personas, time periods, or evaluation dimensions.

List All Persona Evaluation Suites

GET/persona-eval-suites

Query Parameters

limit - Number of results (default: 50)
offset - Skip results for pagination
status - Filter by suite status
tags - Filter by tags

Response Example

{
  "personaEvalSuites": [
    {
      "id": "uuid-suite-123",
      "name": "Q1 2024 Complete Assessment",
      "description": "Comprehensive evaluation of all active personas",
      "status": "active",
      "jobs": [
        {
          "id": "uuid-job-001",
          "personaId": "uuid-persona-sales",
          "name": "Sales Persona Assessment"
        },
        {
          "id": "uuid-job-002", 
          "personaId": "uuid-persona-support",
          "name": "Support Persona Assessment"
        }
      ],
      "schedule": {
        "frequency": "monthly",
        "nextRun": "2024-02-01T00:00:00Z"
      },
      "metrics": {
        "totalJobs": 3,
        "lastRunScore": 8.3,
        "completionRate": 100
      },
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ]
}

Create Persona Evaluation Suite

POST/persona-eval-suites

Request Body

{
  "name": "Weekly Quality Assurance Suite",
  "description": "Regular assessment of all customer-facing personas",
  "jobs": [
    {
      "personaEvalJobId": "uuid-job-001",
      "weight": 0.4,
      "critical": true
    },
    {
      "personaEvalJobId": "uuid-job-002", 
      "weight": 0.3,
      "critical": false
    },
    {
      "personaEvalJobId": "uuid-job-003",
      "weight": 0.3,
      "critical": false
    }
  ],
  "configuration": {
    "executionOrder": "parallel",
    "failureHandling": "continue_on_failure",
    "aggregation": {
      "method": "weighted_average",
      "includeIndividualScores": true
    }
  },
  "schedule": {
    "enabled": true,
    "frequency": "weekly",
    "day": "sunday",
    "time": "02:00"
  },
  "notifications": {
    "onCompletion": true,
    "onFailure": true,
    "recipients": ["qa-team@company.com"]
  },
  "tags": ["quality_assurance", "weekly", "production"]
}

Response Example

{
  "personaEvalSuite": {
    "id": "uuid-suite-456",
    "name": "Weekly Quality Assurance Suite",
    "description": "Regular assessment of all customer-facing personas",
    "status": "active",
    "jobs": [
      {
        "personaEvalJobId": "uuid-job-001",
        "weight": 0.4,
        "critical": true
      }
    ],
    "configuration": {
      "executionOrder": "parallel",
      "failureHandling": "continue_on_failure"
    },
    "schedule": {
      "enabled": true,
      "frequency": "weekly",
      "day": "sunday",
      "time": "02:00",
      "nextRun": "2024-01-21T02:00:00Z"
    },
    "createdAt": "2024-01-15T10:30:00Z"
  }
}

Persona Evaluation Runs

🏃 Evaluation Runs: Individual executions of evaluation jobs that produce specific scores, insights, and recommendations. Each run captures a snapshot of persona performance at a specific point in time.

List Runs for Evaluation Job

GET/persona-eval-jobs/{id}/runs

Query Parameters

limit - Number of results (default: 50)
offset - Skip results for pagination
status - Filter by run status
from - Runs created after date
to - Runs created before date

Response Example

{
  "personaEvalRuns": [
    {
      "id": "uuid-run-123",
      "personaEvalJobId": "uuid-job-456",
      "status": "completed",
      "overallScore": 8.4,
      "dimensionScores": {
        "accuracy": 8.7,
        "empathy": 8.2,
        "efficiency": 8.1,
        "compliance": 8.6
      },
      "insights": {
        "strengths": [
          "Excellent factual accuracy",
          "Strong compliance adherence"
        ],
        "improvements": [
          "Could improve response efficiency",
          "Enhance emotional intelligence"
        ],
        "recommendations": [
          "Focus training on response time optimization",
          "Add empathy-building scenarios"
        ]
      },
      "sampledConversations": 95,
      "executionTime": 247,
      "createdAt": "2024-01-15T09:00:00Z",
      "completedAt": "2024-01-15T09:04:07Z"
    }
  ],
  "pagination": {
    "total": 23,
    "limit": 50,
    "offset": 0
  }
}

Get Evaluation Run Details

GET/persona-eval-runs/{id}

Path Parameters

id - Evaluation Run UUID

Query Parameters

includeDetails - Include detailed scoring breakdown
includeSamples - Include conversation samples

Response Example

{
  "personaEvalRun": {
    "id": "uuid-run-123", 
    "personaEvalJobId": "uuid-job-456",
    "overallScore": 8.4,
    "dimensionScores": {
      "accuracy": 8.7,
      "empathy": 8.2,
      "efficiency": 8.1,
      "compliance": 8.6
    },
    "detailedScoring": {
      "accuracy": {
        "factual_correctness": 9.0,
        "solution_relevance": 8.5,
        "information_completeness": 8.6
      },
      "empathy": {
        "emotional_understanding": 8.0,
        "tone_appropriateness": 8.4,
        "customer_acknowledgment": 8.2
      }
    },
    "conversationSamples": [
      {
        "conversationId": "uuid-conv-001",
        "score": 9.1,
        "highlights": ["Excellent problem resolution"],
        "concerns": []
      },
      {
        "conversationId": "uuid-conv-002",
        "score": 6.8,
        "highlights": ["Good empathy"],
        "concerns": ["Slow response time"]
      }
    ],
    "trends": {
      "comparedToLastRun": "+0.3",
      "comparedToAverage": "+0.2", 
      "trend": "improving"
    }
  }
}

Skill Evaluation Jobs

🛠️ Skill Evaluation Jobs: Specialized evaluation frameworks for assessing individual skills like sentiment analysis, knowledge retrieval, JSON parsing, or custom API integrations. Focus on functional accuracy and performance.

Create Skill Evaluation Job

POST/skill-eval-jobs

Request Body

{
  "skillId": "uuid-skill-sentiment-001",
  "name": "Sentiment Analysis Accuracy Test",
  "description": "Evaluate sentiment analysis skill performance on diverse text samples",
  "evaluationType": "functional_accuracy",
  "configuration": {
    "testCases": [
      {
        "input": {
          "text": "I absolutely love this product! It's amazing!",
          "expectedSentiment": "positive",
          "expectedConfidence": ">0.8"
        },
        "weight": 1.0
      },
      {
        "input": {
          "text": "This is terrible. Worst experience ever.",
          "expectedSentiment": "negative", 
          "expectedConfidence": ">0.8"
        },
        "weight": 1.0
      },
      {
        "input": {
          "text": "The weather is okay today.",
          "expectedSentiment": "neutral",
          "expectedConfidence": ">0.6"
        },
        "weight": 0.5
      }
    ],
    "metrics": [
      "accuracy",
      "precision",
      "recall",
      "f1_score",
      "execution_time"
    ],
    "performance": {
      "maxExecutionTime": 5.0,
      "timeoutHandling": "fail"
    }
  },
  "schedule": {
    "enabled": true,
    "frequency": "daily",
    "time": "03:00"
  }
}

Response Example

{
  "skillEvalJob": {
    "id": "uuid-skill-eval-123",
    "skillId": "uuid-skill-sentiment-001",
    "name": "Sentiment Analysis Accuracy Test",
    "evaluationType": "functional_accuracy",
    "status": "active",
    "configuration": {
      "testCases": [
        {
          "input": {
            "text": "I absolutely love this product!",
            "expectedSentiment": "positive"
          },
          "weight": 1.0
        }
      ],
      "metrics": [
        "accuracy",
        "precision", 
        "recall",
        "f1_score",
        "execution_time"
      ]
    },
    "schedule": {
      "enabled": true,
      "frequency": "daily",
      "time": "03:00",
      "nextRun": "2024-01-16T03:00:00Z"
    },
    "createdAt": "2024-01-15T10:30:00Z"
  }
}

Skill Evaluation Types

Knowledge Base Skill

{
  "skillId": "uuid-skill-kb-lookup",
  "name": "Knowledge Base Retrieval Test",
  "evaluationType": "retrieval_accuracy",
  "configuration": {
    "testQueries": [
      {
        "query": "How do I reset my password?",
        "expectedResults": [
          "password_reset_procedure",
          "account_recovery_steps"
        ],
        "relevanceThreshold": 0.8
      }
    ],
    "metrics": [
      "retrieval_accuracy",
      "relevance_score",
      "response_time",
      "coverage"
    ]
  }
}

JSON Parser Skill

{
  "skillId": "uuid-skill-json-parser",
  "name": "JSON Parsing Validation Test",
  "evaluationType": "parsing_accuracy",
  "configuration": {
    "testCases": [
      {
        "input": {
          "jsonData": "{\"order\": {\"id\": 123}}",
          "extractFields": ["order.id"]
        },
        "expected": {
          "order.id": 123
        }
      }
    ],
    "metrics": [
      "parsing_success_rate",
      "field_extraction_accuracy",
      "error_handling"
    ]
  }
}

Evaluation Types & Models

Common Evaluation Dimensions

Accuracy Metrics

Factual Correctness: Information accuracy
Solution Relevance: Response appropriateness
Technical Accuracy: Technical detail precision
Context Understanding: Conversation context grasp

Empathy & Tone

Emotional Intelligence: Emotion recognition
Tone Appropriateness: Situational tone matching
Customer Acknowledgment: Issue validation
Professional Courtesy: Respectful communication

Efficiency & Performance

Response Time: Speed of response
Solution Directness: Path to resolution
Conversation Length: Interaction efficiency
Follow-up Necessity: Resolution completeness

Scoring Models

Weighted Average Model

{
  "scoringModel": {
    "type": "weighted_average",
    "scale": "1_to_10",
    "weights": {
      "accuracy": 0.4,
      "empathy": 0.3,
      "efficiency": 0.2,
      "compliance": 0.1
    },
    "passingThreshold": 7.0,
    "normalization": "z_score"
  }
}

Rubric-based Model

{
  "scoringModel": {
    "type": "rubric_based",
    "scale": "letter_grade",
    "rubric": {
      "excellent": {"min": 9.0, "grade": "A"},
      "good": {"min": 7.0, "grade": "B"},
      "satisfactory": {"min": 5.0, "grade": "C"},
      "needs_improvement": {"min": 3.0, "grade": "D"},
      "unsatisfactory": {"min": 0.0, "grade": "F"}
    }
  }
}

Evaluation Best Practices

Evaluation Design

Define clear, measurable criteria for each evaluation dimension
Use representative conversation samples for accurate assessment
Set realistic thresholds based on business requirements
Regularly review and update evaluation criteria

Performance Optimization

Schedule evaluations during low-traffic periods
Use appropriate sample sizes for statistical significance
Implement trend analysis to track performance over time
Create automated alerts for significant performance changes

Evaluation RPC API →Quickstart Guide →

MERCURY

Conversation Evaluation REST API

Base URL & Authentication

Persona Evaluation Jobs

List All Persona Evaluation Jobs

Query Parameters

Response Example

Create Persona Evaluation Job

Request Body

Response Example

Get Persona Evaluation Job by ID

Path Parameters

Query Parameters

Response Example

Update Persona Evaluation Job

Request Body

Response Example

Delete Persona Evaluation Job

Path Parameters

Response Example

Persona Evaluation Suites

List All Persona Evaluation Suites

Query Parameters

Response Example

Create Persona Evaluation Suite

Request Body

Response Example

Persona Evaluation Runs

List Runs for Evaluation Job

Query Parameters

Response Example

Get Evaluation Run Details

Path Parameters

Query Parameters

Response Example

Skill Evaluation Jobs

Create Skill Evaluation Job

Request Body

Response Example

Skill Evaluation Types

Knowledge Base Skill

JSON Parser Skill

Evaluation Types & Models

Common Evaluation Dimensions

Accuracy Metrics

Empathy & Tone

Efficiency & Performance

Scoring Models

Weighted Average Model

Rubric-based Model

Evaluation Best Practices

Evaluation Design

Performance Optimization