Question Use the contributor_evaluation.question task type to ask contributors questions in the annotation UI.
Hide configuration properties
The ID of the question to include in the contributor annotation flow. Questions are most easily created through the UI here .
Controls the presentation of evaluation item data in the annotation UI.
A list of elements to be rendered within the layout container. Can be either an individual Component object, or a nested Container.
A pointer to the data in each evaluation item to be displayed within the component.
An optional display name to be used at the top of the component. Defaults to the field name from the ItemLocator used for data.
A pointer to the data in each evaluation item to be displayed within the component.
An optional display name to be used at the top of the component. Defaults to the field name from the ItemLocator used for data.
A list of elements to be rendered within the layout container. Can be either an individual Component object, or a nested Container.
direction
enum<'row' | 'column'>
default: "row"
The axis that children are placed in the container. Based on CSS flex-direction
direction
enum<'row' | 'column'>
default: "row"
The axis that children are placed in the container. Based on CSS flex-direction
An optional group identifier. Any contributor_evaluation tasks that share the same queue_id will be grouped together in the annotation UI.
Example Usage Assume we have a categorical question with ID a0b25d45-7c6c-4480-9871-61ae0fd0b819 configured as such:
We can create an annotation queue as part of our evaluation to ask our contributors to answer this question in order to produce the same correctness result as the guided decoding example :
client.evaluations.create( name = "Example Correctness Evaluation" , data = [ { "input" : "What color is the sky?" , "expected_output" : "Blue" , "generated_output" : "The sky appears blue during ..." }, ... ], tasks = [ { "task_type" : "contributor_evaluation.question" , "alias" : "correctness" , "configuration" : { "question_id" : "a0b25d45-7c6c-4480-9871-61ae0fd0b819" , "layout" : { "direction" : "column" , "children" : [ # Component: { "data" : "item.input" , "label" : "User Query" }, # Nested container: { "direction" : "row" , "children" : [ { "data" : "item.expected_output" , "label" : "Expected Output" }, { "data" : "item.generated_output" , "label" : "Generated Output" } ] } ] }, "queue_id" : "default" } } ] ) 
