RATIO MACHINA STARTER logo

MERCURY

POST /skill-eval-suites/{id}/runs

The POST /skill-eval-suites/{id}/runs endpoint executes skill evaluation suites asynchronously using AWS Step Functions. Start evaluation suite runs, track execution progress via ARN, and monitor comprehensive skill performance across multiple evaluation criteria and test scenarios.

Base URL & Authentication

POST https://your-api-name.mercury.ratiomachina.com/skill-eval-suites/{id}/runs
Content-Type: application/json
Authentication: x-mercury-api-key: YOUR_API_KEY

Basic Skill Suite Run Execution

Execute a skill evaluation suite asynchronously. The system creates a new Step Functions execution to run all skill evaluations in the suite and returns tracking information including the AWS execution ARN.

Request Body

{}
Path Parameters
  • id - SkillEvalSuite UUID to execute
Request Body
  • Optional - Empty body or custom configuration

Response Example

{
  "skillEvalSuiteRun": {
    "id": "789e1234-e89b-12d3-a456-426614174000",
    "suiteId": "456e7890-e89b-12d3-a456-426614174000",
    "suiteName": "JSON Parser Skills Evaluation Suite",
    "status": "running",
    "totalEvals": 5,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-skill-eval-runner-staging:789e1234-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T10:35:00Z",
    "completedAt": null
  }
}
Response Fields
  • id - UUID of the newly created skill evaluation suite run
  • suiteId - UUID of the parent skill evaluation suite
  • suiteName - Name of the skill evaluation suite being run
  • status - Current status (running, completed, failed)
  • totalEvals - Total number of skill evaluations in the suite
  • executionArn - AWS Step Functions execution ARN for tracking

Sentiment Analysis Skill Suite

Execute a comprehensive sentiment analysis skill evaluation suite with multiple test scenarios and accuracy metrics.

Request Example

curl -X POST 'https://api.hermes.com/skill-eval-suites/sentiment-analysis-suite/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{}'

Response Example

{
  "skillEvalSuiteRun": {
    "id": "abc12345-e89b-12d3-a456-426614174000",
    "suiteId": "sentiment-analysis-suite",
    "suiteName": "Sentiment Analysis Comprehensive Suite",
    "status": "running",
    "totalEvals": 15,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-skill-eval-runner-staging:abc12345-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T14:20:00Z",
    "completedAt": null
  }
}

Knowledge Base Skills Suite

Execute a knowledge base skill evaluation suite to assess search accuracy, relevance scoring, and information retrieval capabilities.

Request Example

curl -X POST 'https://api.hermes.com/skill-eval-suites/def67890-e89b-12d3-a456-426614174000/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{
    "configuration": {
      "priority": "high",
      "timeoutMinutes": 45
    }
  }'

Response Example

{
  "skillEvalSuiteRun": {
    "id": "def67890-e89b-12d3-a456-426614174000",
    "suiteId": "def67890-e89b-12d3-a456-426614174000",
    "suiteName": "Knowledge Base Skills Evaluation Suite",
    "status": "running",
    "totalEvals": 20,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-skill-eval-runner-staging:def67890-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T16:45:00Z",
    "completedAt": null
  }
}

Completed Skills Suite Run

Example response for a completed skill evaluation suite run showing final results and performance metrics.

Completed Status

{
  "skillEvalSuiteRun": {
    "id": "ghi98765-e89b-12d3-a456-426614174000",
    "suiteId": "json-parser-suite-789",
    "suiteName": "JSON Parser Skills Evaluation Suite",
    "status": "completed",
    "totalEvals": 10,
    "passedEvals": 9,
    "failedEvals": 1,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-skill-eval-runner-staging:ghi98765-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T11:00:00Z",
    "completedAt": "2024-01-15T11:18:00Z"
  }
}

Results Summary

  • Overall Performance: 90% pass rate (9/10 evaluations)
  • Execution Time: 18 minutes
  • Success Rate: Excellent skill performance
  • Areas for Improvement: 1 failed evaluation in edge case handling
  • Step Functions Status: Successfully completed
  • Monitoring: Track via executionArn in AWS Console

Custom Business Logic Suite

Execute custom business logic skill evaluation suites with specialized test cases and domain-specific validation criteria.

Request Example

curl -X POST 'https://api.hermes.com/skill-eval-suites/external/lead-scoring-suite-v3/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{
    "configuration": {
      "priority": "normal",
      "timeoutMinutes": 30,
      "parallelExecution": true
    }
  }'

Response Example

{
  "skillEvalSuiteRun": {
    "id": "jkl56789-e89b-12d3-a456-426614174000",
    "suiteId": "lead-scoring-suite-v3",
    "suiteName": "Lead Scoring Business Logic Suite",
    "status": "running",
    "totalEvals": 25,
    "passedEvals": 0,
    "failedEvals": 0,
    "executionArn": "arn:aws:states:us-west-2:123456789012:execution:hermes-skill-eval-runner-staging:jkl56789-e89b-12d3-a456-426614174000",
    "startedAt": "2024-01-15T13:30:00Z",
    "completedAt": null
  }
}

Error Handling

Error Response Example

{
  "error": "SkillEvalSuite not found",
  "details": {
    "suiteId": "invalid-skill-suite-id",
    "message": "No skill evaluation suite found with the provided ID"
  },
  "status": "error"
}

Common Error Types

  • Suite not found - Invalid skill evaluation suite ID
  • Already running - Suite has an active execution in progress
  • Invalid configuration - Malformed request parameters
  • Step Functions error - AWS execution service unavailable
  • Skill unavailable - Required skills not accessible
  • Rate limit exceeded - Too many concurrent skill suite executions

AWS Step Functions Integration

This endpoint integrates with AWS Step Functions for asynchronous skill evaluation execution. Use the returned executionArn to track progress and monitor the skill evaluation suite run in the AWS Console.

Execution Tracking

  • Use executionArn to track skill evaluation progress in AWS Console
  • Monitor execution status and individual skill test results
  • Review execution history and performance benchmarks
  • Debug failed skill evaluations with detailed error logs
  • Set up CloudWatch alarms for suite execution monitoring

Performance Monitoring

  • Track skill execution times and accuracy rates
  • Monitor resource usage for skill processing
  • Implement timeout handling for long-running evaluations
  • Use parallel execution for independent skill tests
  • Store evaluation results for skill performance analysis

cURL Examples

Basic Skill Suite Execution

curl -X POST 'https://api.hermes.com/skill-eval-suites/456e7890-e89b-12d3-a456-426614174000/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{}'

With Custom Configuration

curl -X POST 'https://api.hermes.com/skill-eval-suites/nlp-processing-suite/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{
    "configuration": {
      "priority": "high",
      "timeoutMinutes": 60,
      "retryOnFailure": true,
      "parallelExecution": true
    }
  }'

External ID Support

curl -X POST 'https://api.hermes.com/skill-eval-suites/external/data-processing-suite-v4/runs' \
  -H 'Content-Type: application/json' \
  -H 'x-hermes-api-key: YOUR_API_KEY' \
  -d '{}'

Skill Evaluation Suite Runs Best Practices

Execution Management

  • Monitor executionArn in AWS Step Functions console
  • Set appropriate timeout values for skill complexity
  • Use external IDs for consistent skill suite identification
  • Implement proper error handling for skill failures

Performance & Quality

  • Track skill accuracy and execution performance
  • Set up CloudWatch alarms for suite failures
  • Avoid concurrent runs of resource-intensive skill suites
  • Store evaluation results for skill optimization
← Persona Eval RunsSkill Evals REST →
Transcend the hype with MERCURY by Ratio Machina