Annotation Configuration

By default, the annotation UI which annotators see in SGP shows the test case input, expected output, and output. However, for complex evaluations may want to:

• display data from the trace
• select which parts of test case inputs and test case outputs to display
• modify the layout the annotation UI

The Annotation Configuration allows you to do all three.

Here’s what an example annotation configuration looks like:

from scale_gp.lib.types.data_locator import data_locator # this is a helper to produce

annotation_configuration = dict(
  annotation_config_type="flexible", # this is the default, so we could have omitted annotation_config_type entirely.
  direction="row", # or "col"
  components=[ # 2D array representing how things will be layed out in the UI
    [
      dict(data_loc=["test_case_output", "output", "string_output"], label="string output"),
      dict(data_loc=["test_case_data", "expected_output", "string_expected"]),
    ],
    [
      dict(data_loc=data_locator.test_case_output.output["messages_output"]), # The data_locator is an easier way of producing data_locs
    ],
    [
      dict(data_loc=data_locator.trace["tool_call"]).input["string_input"] # reference the "tool_call" node from the trace earlier
    ]
  ]
)

evaluation = sgp_client.evaluations.create(
    account_id=ACCOUNT_ID,
    name=f"example flexible evaluation",
    description="This is a test evaluation",
    type="builder",
    evaluation_config_id=evaluation_config.id, # You need to create an evaluation config, evaluation_dataset, and application spec/variant
    evaluation_dataset_id=flexible_evaluation_dataset.id,
    application_variant_id=application_variant.id,
    application_spec_id=application.id,
  annotation_config=annotation_configuration,
)

When a contributor annotates this evaluation in the UI, they will see an annotation UI that looks something like this:

Let’s break down how a custom annotation config is set up:

• annotation_config_type: by default this is “flexible”. The other types are “summarization” and “multiturn” which make it easier to work with specific use cases
• components: this is a 2D list of annotation items. Each annotation item points to somewhere in the test case data, test case output, or trace. When the annotator grades the test case output, they will see data pulled from each location
  • Each annotation item has a “data_loc” field and an optional “label” field. The “data_loc” is an array that points to where annotation data should be pulled from. The “label” is a name to be displayed to a user for the “data_loc”.

    ⚠️ if a “data_loc” points somewhere that doesn’t exist for one or more test cases, you will not be able to create the evaluation.

• direction: by default “row”. Decides whether components are laid out as rows or as columns

Here’s are some examples of how different arrangements of components produce different UIs:

data_locs can take any of these shapes:

It is highly recommended that you use the data_locator helper instead of manually creating the data_loc array.

​

Customizing the Annotation UI per question

Sometimes, you have certain questions in an evaluation rubric that are relevant only to a specific part of the test case, test case output or trace. For instance, you might ask a question specifically about the “completion” or “reranking” step in the trace.

In that case you can create a question_id_to_annotation_config mapping that lets you override the annotation config for a specific question ID:

question_id_to_annotation_config = {
    questions[1].id: dict(
        components=[
            [
                dict(
                    data_loc=data_locator.trace["completions"].input,
                    label="string output",
                ),
                dict(
                    data_loc=data_locator.trace["completions"].output
                )
            ],
            [
                dict(
                    data_loc=data_locator.trace["completions"].expected
                ),
            ],
        ],
    )
}

evaluation = sgp_client.evaluations.create(
  ... # specify all the usual evaluation fields,
  annotation_config=annotation_configuration,
  question_id_to_annotation_configuration=queston_id_to_annotation_config # where specified, overrides the annotation_config
)

In the annotation UI, the information rendered will now change for each respective evaluation question as mapped above:

​

Dev Mode

SGP also supports “Dev Mode” which allows an annotator to view all the inputs, outputs and the full trace all at once. You can toggle Dev Mode by clicking on the top right in the annotation UI: