RATIO MACHINA STARTER logo

MERCURY

POST /persona-eval-suites/{id}/runs

The POST /persona-eval-suites/{id}/runs endpoint executes persona evaluation suites asynchronously using AWS Step Functions. Start evaluation suite runs, track execution progress via ARN, and monitor comprehensive persona performance across multiple evaluation criteria.

Base URL & Authentication

POST https://your-api-name.mercury.ratiomachina.com/persona-eval-suites/{id}/runs
Content-Type: application/json
Authentication: x-mercury-api-key: YOUR_API_KEY

Basic Suite Run Execution

Execute a persona evaluation suite asynchronously. The system creates a new Step Functions execution to run all evaluations in the suite and returns tracking information including the AWS execution ARN.

Request Body

{}
Path Parameters
  • id - PersonaEvalSuite UUID to execute
Request Body
  • Optional - Empty body or custom configuration

Response Example

{
  "personaEvalSuiteRun": {
    "id": "789e1234-e89b-12d3-a456-426614174000",
    "suiteId": "456e7890-e89b-12d3-a456-426614174000",
    "suiteName": "Customer Service Evaluation Suite",
    "status": "running",
    "totalEvals": 2,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-persona-eval-runner-staging:789e1234-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T10:35:00Z",
    "completedAt": null
  }
}
Response Fields
  • id - UUID of the newly created evaluation suite run
  • suiteId - UUID of the parent persona evaluation suite
  • suiteName - Name of the evaluation suite being run
  • status - Current status (running, completed, failed)
  • totalEvals - Total number of evaluations in the suite
  • executionArn - AWS Step Functions execution ARN for tracking

Customer Support Suite Execution

Execute a comprehensive customer support evaluation suite with multiple evaluation criteria and performance metrics.

Request Example

curl -X POST 'https://api.hermes.com/persona-eval-suites/support-suite-456/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{}'

Response Example

{
  "personaEvalSuiteRun": {
    "id": "abc12345-e89b-12d3-a456-426614174000",
    "suiteId": "support-suite-456",
    "suiteName": "Customer Support Excellence Evaluation",
    "status": "running",
    "totalEvals": 8,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-persona-eval-runner-staging:abc12345-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T14:20:00Z",
    "completedAt": null
  }
}

Sales Performance Suite Execution

Execute a sales performance evaluation suite to assess persona effectiveness in lead qualification, objection handling, and closing techniques.

Request Example

curl -X POST 'https://api.hermes.com/persona-eval-suites/def67890-e89b-12d3-a456-426614174000/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{
    "configuration": {
      "priority": "high",
      "timeoutMinutes": 60
    }
  }'

Response Example

{
  "personaEvalSuiteRun": {
    "id": "def67890-e89b-12d3-a456-426614174000",
    "suiteId": "def67890-e89b-12d3-a456-426614174000",
    "suiteName": "Sales Performance Comprehensive Suite",
    "status": "running",
    "totalEvals": 12,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-persona-eval-runner-staging:def67890-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T16:45:00Z",
    "completedAt": null
  }
}

Completed Suite Run

Example response for a completed evaluation suite run showing final results and completion timestamp.

Completed Status

{
  "personaEvalSuiteRun": {
    "id": "ghi98765-e89b-12d3-a456-426614174000",
    "suiteId": "quality-assurance-suite-789",
    "suiteName": "Quality Assurance Evaluation Suite",
    "status": "completed",
    "totalEvals": 5,
    "passedEvals": 4,
    "failedEvals": 1,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-persona-eval-runner-staging:ghi98765-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T09:00:00Z",
    "completedAt": "2024-01-15T09:25:00Z"
  }
}

Results Summary

  • Overall Performance: 80% pass rate (4/5 evaluations)
  • Execution Time: 25 minutes
  • Success Rate: High confidence in persona effectiveness
  • Areas for Improvement: 1 failed evaluation requires attention
  • Step Functions Status: Successfully completed
  • Monitoring: Track via executionArn in AWS Console

Error Handling

Error Response Example

{
  "error": "PersonaEvalSuite not found",
  "details": {
    "suiteId": "invalid-suite-id",
    "message": "No evaluation suite found with the provided ID"
  },
  "status": "error"
}

Common Error Types

  • Suite not found - Invalid persona evaluation suite ID
  • Already running - Suite has an active execution in progress
  • Invalid configuration - Malformed request parameters
  • Step Functions error - AWS execution service unavailable
  • Permission denied - Insufficient access to evaluation suite
  • Rate limit exceeded - Too many concurrent suite executions

AWS Step Functions Integration

This endpoint integrates with AWS Step Functions for asynchronous execution. Use the returned executionArn to track progress and monitor the evaluation suite run in the AWS Console.

Execution Tracking

  • Use executionArn to track progress in AWS Console
  • Monitor execution status and step-by-step progress
  • Review execution history and performance metrics
  • Debug failed executions with detailed error logs
  • Set up CloudWatch alarms for execution monitoring

Monitoring Best Practices

  • Poll for completion rather than long-running connections
  • Implement retry logic for failed suite executions
  • Set appropriate timeout values for large evaluation suites
  • Use AWS CloudWatch for execution metrics and alerting
  • Store execution results for historical analysis

cURL Examples

Basic Suite Execution

curl -X POST 'https://api.hermes.com/persona-eval-suites/456e7890-e89b-12d3-a456-426614174000/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{}'

With Custom Configuration

curl -X POST 'https://api.hermes.com/persona-eval-suites/support-excellence-suite/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{
    "configuration": {
      "priority": "high",
      "timeoutMinutes": 90,
      "retryOnFailure": true
    }
  }'

External ID Support

curl -X POST 'https://api.hermes.com/persona-eval-suites/external/cs-evaluation-suite-v2/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{}'

Persona Evaluation Suite Runs Best Practices

Execution Management

  • Monitor executionArn in AWS Step Functions console
  • Set appropriate timeout values for evaluation complexity
  • Use external IDs for consistent suite identification
  • Implement proper error handling and retry logic

Performance & Monitoring

  • Track completion rates and execution times
  • Set up CloudWatch alarms for failed executions
  • Avoid concurrent runs of the same evaluation suite
  • Store results for historical performance analysis
← Persona Eval SuitesSkill Eval Runs →
Transcend the hype with MERCURY by Ratio Machina