> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Contributor Evaluation

> Defer evaluation tasks to human contributors.

## Question

Use the `contributor_evaluation.question` task type to ask contributors [questions](https://egp.dashboard.scale.com/evaluation/questions/questions) in the annotation UI.

<Expandable title="configuration properties" defaultOpen={true}>
  <ResponseField name="question_id" type="string" required>
    The ID of the question to include in the contributor annotation flow. Questions are most easily created through the UI [here](https://egp.dashboard.scale.com/evaluation/questions/questions/new).
  </ResponseField>

  <ResponseField name="layout" type="object" required>
    Controls the presentation of evaluation item data in the annotation UI.

    <Expandable title="child attributes" defaultOpen={true}>
      <ResponseField name="children" type="object[]" required>
        A list of elements to be rendered within the layout container. Can be either an individual `Component` object, or a nested `Container`.

        <Card title="">
          <Tabs>
            <Tab title="Component">
              <Expandable title="child attributes" defaultOpen={true}>
                <ResponseField name="data" type="ItemLocator" required>
                  A pointer to the data in each evaluation item to be displayed within the component.
                </ResponseField>

                <ResponseField name="label" type="string">
                  An optional display name to be used at the top of the component. Defaults to the field name from the `ItemLocator` used for `data`.
                </ResponseField>
              </Expandable>
            </Tab>

            <Tab title="Container">
              <Expandable title="child attributes" defaultOpen={true}>
                <ResponseField name="children" type="object[]" required>
                  A list of elements to be rendered within the layout container. Can be either an individual `Component` object, or a nested `Container`.
                </ResponseField>

                <ResponseField name="direction" type="enum<'row' | 'column'>" default="row">
                  The axis that children are placed in the container. Based on CSS [`flex-direction`](https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction).
                </ResponseField>
              </Expandable>
            </Tab>
          </Tabs>
        </Card>
      </ResponseField>

      <ResponseField name="direction" type="enum<'row' | 'column'>" default="row">
        The axis that children are placed in the container. Based on CSS [`flex-direction`](https://developer.mozilla.org/en-US/docs/Web/CSS/flex-direction).
      </ResponseField>
    </Expandable>
  </ResponseField>

  <ResponseField name="queue_id" type="string" default="default">
    An optional group identifier. Any `contributor_evaluation` tasks that share the same `queue_id` will be grouped together in the annotation UI.
  </ResponseField>
</Expandable>

### Example Usage

Assume we have a categorical question with ID `a0b25d45-7c6c-4480-9871-61ae0fd0b819` configured as such:

<img width="75%" className="mx-auto" noZoom src="https://mintcdn.com/scalegp/2sSrpizRElJqluR6/images/nextgen-evaluation/correctness-question.png?fit=max&auto=format&n=2sSrpizRElJqluR6&q=85&s=4697cc6e2832b1266578deed4f5d489f" data-path="images/nextgen-evaluation/correctness-question.png" />

We can create an annotation queue as part of our evaluation to ask our contributors to answer this question in order to produce the same `correctness` result as the [guided decoding example](/docs/v5/next-gen-evaluation/tasks/auto-evaluation#example-usage):

```python theme={null}
client.evaluations.create(
  name="Example Correctness Evaluation",
  data=[
    {
      "input": "What color is the sky?",
      "expected_output": "Blue",
      "generated_output": "The sky appears blue during ..."
    },
    ...
  ],
  tasks=[
    {
      "task_type": "contributor_evaluation.question",
      "alias": "correctness",
      "configuration": {
        "question_id": "a0b25d45-7c6c-4480-9871-61ae0fd0b819",
        "layout": {
          "direction": "column",
          "children": [
            # Component:
            { "data": "item.input", "label": "User Query" },
            # Nested container:
            {
              "direction": "row",
              "children": [
                { "data": "item.expected_output", "label": "Expected Output" },
                { "data": "item.generated_output", "label": "Generated Output" }
              ]
            }
          ]
        },
        "queue_id": "default"
      }
    }
  ]
)
```