Agent-as-a-Judge
Use theauto_evaluation.agent task type for configuring auto evaluation tasks with specialized agents.
Task Types
Agent Evaluation
Leverage specialized agents to produce evaluation results.
Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
Leverage specialized agents to produce evaluation results.
auto_evaluation.agent task type for configuring auto evaluation tasks with specialized agents.
Hide configuration properties
<model_provider>/<model_name> format) that the Agent will use to execute the task.output_type is inferred from output rules.APEAgent (Default Agent) you can configure a temperatureclient.evaluations.create(
name="Example Instruction Following Evaluation,
data=[
{
"prompt": "You must either describe the process of photosynthesis in exactly 20 words or explain why it’s important in a single sentence under 10 words — choose whichever option is more precise.",
"response": "Photosynthesis converts sunlight, water, and carbon dioxide into glucose and oxygen, sustaining plant life and fueling ecosystems.",
},
...
],
tasks=[
{
"task_type": "auto_evaluation.agent",
"alias": "instruction-following-agent",
"configuration": {
"name": "instruction-following-agent",
"definition": "Perform comprehensive analysis of instruction following",
"output_rules": [
"Score adherence on a 1-5 scale",
"List specific violations and well-followed instructions"
],
"output_type": "integer",
"designated_to": {
"agent_name": "IFAgent",
"config": {
"model": "openai/o3-mini"
}
}
}
}
]
)