A simplified and extendable update to SGP’s evaluation framework
input
field within the user defined data, or a human annotation task to produce the expected output. These tasks can be interdependent and composed to create as simple or complex of an evaluation that your use case requires.scale-gp-beta
package which runs exclusively on the V5 API.data
is made up of two items, each with an input
field. A single task is specified to generate a chat completion from each data item.
data
field can be thought of as a row in a table. We refer to these rows as items, and they can be retrieved like so:
task_type
specific parameters, such as the messages
field for a chat_completion
task.Referencing item data
"item.<field>"
, here on referred to as an ItemLocator
, can be used within a configuration to indicate that the value should be pulled from the evaluation item at task execution time.For example, in the first code snippet we defined a chat_completion
task that used an ItemLocator
as the content
field for a user message:content
field populated based on the item’s input
field.Data can also be referenced within a string by wrapping an ItemLocator
in double curly braces:task_type
.