PersonaEvalSuites REST API

The PersonaEvalSuites REST API manages comprehensive evaluation collections that group multiple related evaluations. Create, manage, and execute evaluation suites for holistic persona assessment across different dimensions and scenarios.

Base URL & Authentication

Base URL: https://your-api-name.mercury.ratiomachina.com
Authentication: x-mercury-api-key: YOUR_API_KEY

List Persona Evaluation Suites

GET /persona-eval-suites

Query Parameters

limit - Number of results (default: 50, max: 100)
offset - Skip results for pagination
status - Filter by suite status (active, archived)
personaExternalId - Filter by persona external ID
tags - Filter by tags (comma-separated)

Response Example

{
  "personaEvalSuites": [
    {
      "id": "123e4567-e89b-12d3-a456-426614174000",
      "name": "Customer Service Excellence Suite",
      "description": "Comprehensive evaluation for customer service personas",
      "personaExternalId": "cs-agent-v1",
      "status": "active",
      "totalEvaluations": 12,
      "tags": ["customer-service", "quality-assurance"],
      "configuration": {
        "executionTimeout": 300,
        "parallelExecution": true,
        "failureThreshold": 20
      },
      "schedule": {
        "enabled": true,
        "frequency": "weekly",
        "nextRun": "2024-01-22T09:00:00Z"
      },
      "lastRun": {
        "executedAt": "2024-01-15T09:00:00Z",
        "status": "completed",
        "passRate": 91.7,
        "duration": 180
      },
      "createdAt": "2024-01-01T12:00:00Z",
      "updatedAt": "2024-01-15T14:30:00Z"
    }
  ],
  "pagination": {
    "total": 1,
    "limit": 50,
    "offset": 0,
    "hasMore": false
  }
}

Create Persona Evaluation Suite

POST /persona-eval-suites

Request Body

{
  "name": "Customer Service Excellence Suite",
  "description": "Comprehensive evaluation for customer service personas",
  "personaExternalId": "cs-agent-v1",
  "externalId": "cs-eval-suite-v2",
  "evaluationIds": [
    "eval-tone-check-123",
    "eval-accuracy-456",
    "eval-response-time-789"
  ],
  "configuration": {
    "executionTimeout": 300,
    "parallelExecution": true,
    "failureThreshold": 20,
    "retryFailedEvals": true,
    "generateReport": true
  },
  "schedule": {
    "enabled": true,
    "frequency": "weekly",
    "dayOfWeek": "monday",
    "hour": 9,
    "timezone": "UTC"
  },
  "notifications": {
    "onCompletion": true,
    "onFailure": true,
    "webhookUrl": "https://your-system.com/webhooks/eval-results",
    "emailRecipients": ["qa-team@company.com"]
  },
  "tags": ["customer-service", "quality-assurance", "automated"]
}

Response Example

{
  "personaEvalSuite": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "name": "Customer Service Excellence Suite",
    "description": "Comprehensive evaluation for customer service personas",
    "personaExternalId": "cs-agent-v1",
    "externalId": "cs-eval-suite-v2",
    "status": "active",
    "totalEvaluations": 3,
    "evaluationIds": [
      "eval-tone-check-123",
      "eval-accuracy-456", 
      "eval-response-time-789"
    ],
    "configuration": {
      "executionTimeout": 300,
      "parallelExecution": true,
      "failureThreshold": 20,
      "retryFailedEvals": true,
      "generateReport": true
    },
    "schedule": {
      "enabled": true,
      "frequency": "weekly",
      "nextRun": "2024-01-22T09:00:00Z"
    },
    "tags": ["customer-service", "quality-assurance", "automated"],
    "createdAt": "2024-01-15T16:30:00Z",
    "updatedAt": "2024-01-15T16:30:00Z"
  }
}

Get Persona Evaluation Suite

GET /persona-eval-suites/{id}

Path Parameters

id - PersonaEvalSuite UUID

Query Parameters

includeEvaluations - Include full evaluation details (default: false)
includeHistory - Include execution history (default: false)
includeMetrics - Include performance metrics (default: false)

Response Example

{
  "personaEvalSuite": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "name": "Customer Service Excellence Suite",
    "description": "Comprehensive evaluation for customer service personas",
    "personaExternalId": "cs-agent-v1",
    "externalId": "cs-eval-suite-v2",
    "status": "active",
    "totalEvaluations": 3,
    "evaluationIds": [
      "eval-tone-check-123",
      "eval-accuracy-456",
      "eval-response-time-789"
    ],
    "configuration": {
      "executionTimeout": 300,
      "parallelExecution": true,
      "failureThreshold": 20,
      "retryFailedEvals": true,
      "generateReport": true
    },
    "schedule": {
      "enabled": true,
      "frequency": "weekly",
      "nextRun": "2024-01-22T09:00:00Z"
    },
    "lastRun": {
      "executedAt": "2024-01-15T09:00:00Z",
      "status": "completed",
      "passRate": 91.7,
      "duration": 180,
      "executionArn": "arn:aws:states:us-west-2:123456789012:execution:persona-eval-runner:run-123"
    },
    "metrics": {
      "totalRuns": 24,
      "successRate": 95.8,
      "averageExecutionTime": 165,
      "averagePassRate": 89.2
    },
    "tags": ["customer-service", "quality-assurance", "automated"],
    "createdAt": "2024-01-01T12:00:00Z",
    "updatedAt": "2024-01-15T14:30:00Z"
  }
}

Update Persona Evaluation Suite

PUT /persona-eval-suites/{id}

Request Body

{
  "name": "Enhanced Customer Service Suite",
  "description": "Updated comprehensive evaluation suite with new metrics",
  "evaluationIds": [
    "eval-tone-check-123",
    "eval-accuracy-456",
    "eval-response-time-789",
    "eval-sentiment-analysis-101"
  ],
  "configuration": {
    "executionTimeout": 420,
    "parallelExecution": true,
    "failureThreshold": 15,
    "retryFailedEvals": true,
    "generateReport": true
  },
  "schedule": {
    "enabled": true,
    "frequency": "daily",
    "hour": 6,
    "timezone": "UTC"
  },
  "tags": ["customer-service", "quality-assurance", "automated", "enhanced"]
}

Response Example

{
  "personaEvalSuite": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "name": "Enhanced Customer Service Suite",
    "description": "Updated comprehensive evaluation suite with new metrics",
    "personaExternalId": "cs-agent-v1",
    "externalId": "cs-eval-suite-v2",
    "status": "active",
    "totalEvaluations": 4,
    "evaluationIds": [
      "eval-tone-check-123",
      "eval-accuracy-456",
      "eval-response-time-789",
      "eval-sentiment-analysis-101"
    ],
    "configuration": {
      "executionTimeout": 420,
      "parallelExecution": true,
      "failureThreshold": 15,
      "retryFailedEvals": true,
      "generateReport": true
    },
    "schedule": {
      "enabled": true,
      "frequency": "daily",
      "nextRun": "2024-01-16T06:00:00Z"
    },
    "tags": ["customer-service", "quality-assurance", "automated", "enhanced"],
    "createdAt": "2024-01-01T12:00:00Z",
    "updatedAt": "2024-01-15T16:45:00Z"
  }
}

Delete Persona Evaluation Suite

DELETE /persona-eval-suites/{id}

Permanently delete a persona evaluation suite. This will also cancel any scheduled runs and remove all associated execution history.

Path Parameters

id - PersonaEvalSuite UUID to delete

Query Parameters

force - Force deletion even if suite has active runs (default: false)

Response Example

{
  "message": "PersonaEvalSuite deleted successfully",
  "deletedSuite": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "name": "Customer Service Excellence Suite",
    "status": "deleted"
  },
  "deletedAt": "2024-01-15T17:30:00Z",
  "activeRunsCancelled": 0,
  "scheduleCancelled": true
}

Execute Evaluation Suite

POST /persona-eval-suites/{id}/runs

Execute a persona evaluation suite immediately using AWS Step Functions. Returns execution tracking information for monitoring progress.

Request Body

{
  "configuration": {
    "priority": "high",
    "timeoutMinutes": 60,
    "retryOnFailure": true
  }
}

Optional Parameters

configuration - Override suite configuration for this run
evaluationIds - Run only specific evaluations from the suite
tags - Add run-specific tags

Response Example

{
  "personaEvalSuiteRun": {
    "id": "789e1234-e89b-12d3-a456-426614174000",
    "suiteId": "123e4567-e89b-12d3-a456-426614174000",
    "suiteName": "Customer Service Excellence Suite",
    "status": "running",
    "totalEvals": 4,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-persona-eval-runner-staging:789e1234-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T18:00:00Z",
    "completedAt": null
  }
}

⚠️ Security Warning

Important: The persona-eval-suites endpoints do not validate the x-mercury-api-key header. These endpoints are currently accessible without proper authentication. Use caution when calling these endpoints and ensure proper access controls are in place at the network level.