Guided Decoding
Use theauto_evaluation.guided_decoding task type for configuring auto evaluation tasks where the set of potential results are well defined.
Task Types
Auto Evaluation
Leverage LLM-based tasks to produce evaluation results.
Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
Leverage LLM-based tasks to produce evaluation results.
auto_evaluation.guided_decoding task type for configuring auto evaluation tasks where the set of potential results are well defined.
Hide configuration properties
<model_provider>/<model_name> format) to generate the auto evaluation result. Note this model must support some form of guided decoding (e.g. OpenAI’s response formatting) in order for results to be computed.{ "temperature": 0 }).client.evaluations.create(
name="Example Correctness Evaluation",
data=[
{
"input": "What color is the sky?",
"expected_output": "Blue",
"generated_output": "The sky appears blue during ..."
},
...
],
tasks=[
{
"task_type": "auto_evaluation.guided_decoding",
"alias": "correctness",
"configuration": {
"model": "openai/o3-mini",
"prompt": """
Given the user's query: {{item.input}},
The agent's response was: {{item.generated_output}}
The expected response is: {{item.expected_output}}
Did the agent's response fully represent the expected response?
""",
"response_format":
{
"type":"object",
"properties":
{
"question-response":
{
"type":"string",
"enum":["yes","no"]
}
},
"required":["question-response"]
}
}
}
]
)
tasks = [
{
"task_type": "auto_evaluation.guided_decoding",
"alias": "multi_response_option_judge",
"configuration": {
"model": "openai/gpt-4o",
"prompt": "Evaluate this response...",
"response_format": {
"type": "object",
"properties": {
"is_helpful": {
"type": "boolean",
"description": "Whether the response is helpful"
},
"quality_score": {
"type": "integer",
"minimum": 1,
"maximum": 5,
"description": "Quality score from 1 to 5"
},
"accuracy_score": {
"type": "number",
"minimum": 0.0,
"maximum": 1.0,
"description": "Accuracy as a decimal"
},
"category": {
"type": "string",
"enum": ["excellent", "good", "fair", "poor"],
"description": "Quality category"
},
"reasoning": {
"type": "string",
"description": "Explanation of the evaluation"
}
},
"required": ["is_helpful", "quality_score", "accuracy_score", "category", "reasoning"]
}
}
}
]