Knowledge Bases
- GETList Knowledge Bases
- POSTCreate Knowledge Base
- GETGet Knowledge Base
- DELDelete Knowledge Base
- PATCHPatch Knowledge Base
- GETList Upload Jobs
- GETGet Chunks
- POSTQuery Relevant Chunks
- POSTDelete Knowledge Base Data Source Connection
- POSTSubmit Upload Job with local files
- GETList Upload Jobs
- POSTSubmit Upload Job
- GETGet Upload Job
- POSTCancel Upload Job
- GETList Tracked Artifacts
- GETGet Tracked Artifact Details
- DELDelete Locally Stored Artifact
- PATCHPatch Artifact Information
- GETList Chunks of Artifacts with Pagination
- POSTCreate chunk for local chunk artifacts
- GETGet Single Chunk Information and status
- PUTUpdate Single Chunk data for local artifact
- DELDelete Single Chunk from Local Artifact
- POSTTest Knowledge Base Data Source credentials
- POSTBatch Delete Locally Stored Artifacts
- GETList Upload Schedules
- POSTCreate Upload Schedule
- GETGet Upload Schedule
- DELDelete Upload Schedule
- PATCHUpdate Upload Schedule
Chunks
Agents
Completions
Chat Completions
Models
- POSTExecute Model Deployment
- POSTGenerate text embedding
- POSTGenerate reranking
- POSTGenerate completion
- POSTGenerate chat completion
- GETList Model Deployments of a Model
- POSTDeploy Model
- GETGet Model usage by model name
- GETList All Model Deployments
- GETGet Model usage for one deployment
- POSTAdd model to group
- GETGet Model usage for a group
- GETList Models
- POSTCreate Model
- GETGet Model
- DELDelete Model
- PATCHUpdate Model
- GETList Model Groups
- POSTCreate Model Group
- GETGet Model Group
- DELDelete Model Group
- PATCHUpdate Model Group
- GETGet Deployment
- DELDelete Deployment
- PATCHUpdate Deployment
Accounts
Organizations
Question Sets
Evaluations
- POSTClaim Evaluation Task
- PATCHBatch assign tasks to user
- PATCHUpdate Evaluation Task
- GETGet Contributor Metrics
- GETList Contributor Metrics
- GETGet Evaluation Metrics
- GETGet Hybrid Evaluation Metrics
- GETList Evaluations
- POSTCreate Evaluation
- GETGet Evaluation
- DELDelete Evaluation
- PATCHUpdate Evaluation
- GETList Test Case Results
- POSTCreate Test Case Result
- POSTBatch Create Test Case Results
- GETGet Test Case Result
- PATCHUpdate Test Case Result
- GETGet Historical Test Case Result
- GETGet All Versions of Test Case Result
- GETList Historical Test Case Results
Evaluation Configs
Evaluation Datasets
- GETList Evaluation Datasets
- POSTCreate Evaluation Dataset
- GETGet Evaluation Dataset
- DELDelete Evaluation Dataset
- PATCHUpdate Evaluation Dataset
- GETList Evaluation Dataset Versions
- POSTCreate Evaluation Dataset Version
- GETGet Evaluation Dataset Version
- GETList Test Cases
- POSTCreate Test Case
- POSTBatch Create Test Cases
- GETGet Test Case
- DELDelete Test Case
- PATCHUpdate Test Case
- GETGet Historical Test Case
- GETGet All Versions of Test Case
- DELWipe Full History of Test Cases
- GETList Historical Test Cases
- GETList Autogenerated Draft Test Cases
- POSTCreate Autogenerated Draft Test Case
- GETGet Autogenerated Draft Test Case
- DELDelete Autogenerated Draft Test Case
- PATCHUpdate Autogenerated Draft Test Case
- POSTApprove Autogenerated Draft Test Case
- POSTPublish Latest Evaluation Dataset Version
- POSTApprove Auto Generated Test Cases Batch
- GETGet Evaluation Dataset Generation Jobs
- POSTCreate Evaluation Dataset Generation Job
- GETGet Evaluation Dataset Generation Job
- POSTCancel Evaluation Dataset Generation Job
- POSTPublish Evaluation Dataset Version
Studio Projects
Application Specs
Knowledge Base Data Sources
Model Templates V3 (Beta)
Model server
API Reference
Fine Tuning Jobs V3 (Beta)
Training Datasets V3 (Beta)
package deployments
Beta
Applications
- GETList Application Variants
- POSTCreate Application Variant
- GETGet Application Variant
- DELDelete Application Variant
- PATCHUpdate Application Variant
- GETList Application Deployments
- POSTCreate Application Deployment
- GETGet Application Deployment
- PATCHUpdate Application Deployment
- GETList Application Variant Reports
- POSTCreate Application Variant Report
- GETGet Application Variant Report
- GETList Application Test Case Outputs
- POSTBatch Create Application Test Case Outputs
- GETGet Application Test Case Output
- POSTCreate Application With Variant
- GETGet Application Schemas
- POSTProcess Application
- POSTProcess Application By Id
- POSTProcess Deployed Application For Thread
- POSTProcess Application By Alias
- POSTValidate Application
- GETList Application Threads
- POSTCreate New Application Thread
- POSTUpload Application Files
- PATCHPatch Application Variant Guardrails
- POSTPublish Application Variant
ChatThreads
Interactions
MonitoringDashboard
Chat Themes
account groups
Get Test Case Result
Description
Gets the details of a test case result
Details
This API can be used to get information about a single test case result by ID. To use this API, pass in the id that was returned from your Create Test Case Result API call as a path parameter.
Review the response schema to see the fields that will be returned.
import SGPClient from 'sgp';
const client = new SGPClient({
apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});
async function main() {
const testCaseResultWithViews = await client.evaluations.testCaseResults.retrieve(
'evaluation_id',
'test_case_result_id',
);
console.log(testCaseResultWithViews);
}
main();{
"application_test_case_output": {
"application_variant_id": "<string>",
"evaluation_dataset_id": "<string>",
"test_case_version_id": "<string>",
"test_case_id": "<string>",
"output": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"schema_type": "GENERATION",
"application_interaction_id": "<string>",
"application_test_case_output_group_id": "<string>",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"interaction": {
"id": "<string>",
"input": {},
"output": {},
"aggregated": true,
"application_spec_id": "<string>",
"application_variant_id": "<string>",
"start_timestamp": "2023-11-07T05:31:56Z",
"duration_ms": 123,
"operation_status": "SUCCESS",
"operation_metadata": {},
"chat_thread_id": "<string>",
"interaction_source": "EXTERNAL_AI",
"created_at": "2023-11-07T05:31:56Z",
"trace_spans": [
{
"id": "<string>",
"application_interaction_id": "<string>",
"application_variant_id": "<string>",
"node_id": "<string>",
"operation_type": "TEXT_INPUT",
"operation_status": "SUCCESS",
"operation_input": {},
"operation_output": {},
"operation_expected": {},
"operation_metadata": {},
"start_timestamp": "2023-11-07T05:31:56Z",
"end_timestamp": "2023-11-07T05:31:56Z",
"duration_ms": 123,
"trace_id": "<string>",
"parent_id": "<string>"
}
]
}
},
"test_case_version": {
"evaluation_dataset_id": "<string>",
"schema_type": "GENERATION",
"test_case_data": {
"input": "<string>",
"expected_output": "<string>",
"expected_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
},
"artifact_ids_filter": [
"<string>"
]
},
"chat_history": {},
"autogenerated": true,
"test_case_metadata": {},
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z"
},
"task": {
"id": "<string>",
"task_type": "EVALUATION_ANNOTATION",
"task_entity_parent_id": "<string>",
"task_entity_id": "<string>",
"account_id": "<string>",
"assigned_to": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
},
"assignment_expires_at": "2023-11-07T05:31:56Z",
"status": "NOT_READY",
"priority": 123,
"task_entity": {
"label_status": "PENDING",
"audit_status": "UNAUDITED",
"audit_required": true,
"audit_comment": "<string>",
"application_spec_id": "<string>",
"evaluation_id": "<string>",
"evaluation_dataset_id": "<string>",
"evaluation_dataset_version_num": "<string>",
"test_case_id": "<string>",
"test_case_evaluation_data": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"test_case_evaluation_data_schema": "GENERATION",
"result": {},
"completed_at": "2023-11-07T05:31:56Z",
"time_spent_labeling_s": 123,
"application_test_case_output_id": "<string>",
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"annotated_by_user_id": "<string>",
"edited_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z",
"edited_by": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
}
}
},
"metrics": {},
"custom_metrics": [
{
"id": "<string>",
"type": "<string>",
"name": "<string>",
"output": 123
}
],
"audit_tasks": [
{
"id": "<string>",
"task_type": "EVALUATION_ANNOTATION",
"task_entity_parent_id": "<string>",
"task_entity_id": "<string>",
"account_id": "<string>",
"assigned_to": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
},
"assignment_expires_at": "2023-11-07T05:31:56Z",
"status": "NOT_READY",
"priority": 123,
"task_entity": {
"label_status": "PENDING",
"audit_status": "UNAUDITED",
"audit_required": true,
"audit_comment": "<string>",
"application_spec_id": "<string>",
"evaluation_id": "<string>",
"evaluation_dataset_id": "<string>",
"evaluation_dataset_version_num": "<string>",
"test_case_id": "<string>",
"test_case_evaluation_data": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"test_case_evaluation_data_schema": "GENERATION",
"result": {},
"completed_at": "2023-11-07T05:31:56Z",
"time_spent_labeling_s": 123,
"application_test_case_output_id": "<string>",
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"annotated_by_user_id": "<string>",
"edited_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z",
"edited_by": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
}
}
}
],
"annotation_results": [
{
"question_id": "<string>",
"annotation_type": "llm_auto",
"test_case_result_lineage_id": "<string>",
"selected_choice": {
"label": "<string>",
"value": "<string>",
"audit_required": true
},
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"llm_auto_eval_metadata": {
"cost": 123,
"prompt_tokens": 123,
"completion_tokens": 123,
"time_elapsed_s": 123,
"llm_reasoning": "<string>",
"annotation_result_id": "<string>"
}
}
],
"label_status": "PENDING",
"audit_status": "UNAUDITED",
"audit_required": true,
"audit_comment": "<string>",
"application_spec_id": "<string>",
"evaluation_id": "<string>",
"evaluation_dataset_id": "<string>",
"evaluation_dataset_version_num": "<string>",
"test_case_id": "<string>",
"test_case_evaluation_data": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"test_case_evaluation_data_schema": "GENERATION",
"result": {},
"completed_at": "2023-11-07T05:31:56Z",
"time_spent_labeling_s": 123,
"application_test_case_output_id": "<string>",
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"annotated_by_user_id": "<string>",
"edited_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z",
"edited_by": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
}
}Authorizations
Query Parameters
AnnotationResults, AuditTasks, CustomMetrics, Metrics, Task, TestCaseVersion, Trace Response
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED CHUNKS The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
CHUNKS GENERATION, FLEXIBLE The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
Boolean of whether this interaction has been uploaded to s3 bucket yet, default is false
How much time the step took in milliseconds(ms)
The outcome of the operation
SUCCESS, ERROR The date and time when the entity was created in ISO format.
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
EXTERNAL_AI, EVALUATION, SGP_CHAT, AGENTS_SERVICE Identifies the application step
The id of the node in the application_variant config that emitted this insight
Type of the operation, e.g. RERANKING
TEXT_INPUT, TEXT_OUTPUT, COMPLETION_INPUT, COMPLETION, KB_RETRIEVAL, KB_INPUT, RERANKING, EXTERNAL_ENDPOINT, PROMPT_ENGINEERING, DOCUMENT_INPUT, MAP_REDUCE, DOCUMENT_SEARCH, DOCUMENT_PROMPT, CUSTOM, INPUT_GUARDRAIL, OUTPUT_GUARDRAIL, CODE_EXECUTION, DATA_MANIPULATION, EVALUATION, FILE_RETRIEVAL, KB_ADD_CHUNK, KB_MANAGEMENT, TRACER, STANDALONE The outcome of the operation
SUCCESS, ERROR The JSON representation of the input that this step received.
The JSON representation of the output that this step emitted.
The start time of the step
How much time the step took in milliseconds(ms)
The interaction ID this span belongs to
The id of the application variant this span belongs to
The JSON representation of the expected output for this step
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
The end time of the step, nullable, since it can be set to done at a later point in time.
The root-level ID where this span belongs to
Who is the parent span of this current span, null if span is root parent.
The ID of the associated evaluation dataset.
CHUNKS Boolean to track whether or not the test case is autogenerated
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
GENERATION Used for tracking previous chat interactions for multi-chat test cases
Metadata for the test case
The date and time when the entity was overwritten or deleted in ISO format.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The type of annotation result.
llm_auto, human The selected choices(s) for the annotation result, in JSON form. For categorical questions, this is an object or list of objects (depending on if multiple selections are allowed). For free text questions, this is a string. For numeric or rating questions, this is a number.
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The time elapsed to generate this annotation in seconds.
The reasoning the LLM gave for the annotation it provided.
The ID of the associated annotation result.
The cost of the annotation in cents.
UNAUDITED, FIXED, APPROVED The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED CHUNKS The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
CHUNKS GENERATION, FLEXIBLE The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
Boolean of whether this interaction has been uploaded to s3 bucket yet, default is false
How much time the step took in milliseconds(ms)
The outcome of the operation
SUCCESS, ERROR The date and time when the entity was created in ISO format.
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
EXTERNAL_AI, EVALUATION, SGP_CHAT, AGENTS_SERVICE Identifies the application step
The id of the node in the application_variant config that emitted this insight
Type of the operation, e.g. RERANKING
TEXT_INPUT, TEXT_OUTPUT, COMPLETION_INPUT, COMPLETION, KB_RETRIEVAL, KB_INPUT, RERANKING, EXTERNAL_ENDPOINT, PROMPT_ENGINEERING, DOCUMENT_INPUT, MAP_REDUCE, DOCUMENT_SEARCH, DOCUMENT_PROMPT, CUSTOM, INPUT_GUARDRAIL, OUTPUT_GUARDRAIL, CODE_EXECUTION, DATA_MANIPULATION, EVALUATION, FILE_RETRIEVAL, KB_ADD_CHUNK, KB_MANAGEMENT, TRACER, STANDALONE The outcome of the operation
SUCCESS, ERROR The JSON representation of the input that this step received.
The JSON representation of the output that this step emitted.
The start time of the step
How much time the step took in milliseconds(ms)
The interaction ID this span belongs to
The id of the application variant this span belongs to
The JSON representation of the expected output for this step
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
The end time of the step, nullable, since it can be set to done at a later point in time.
The root-level ID where this span belongs to
Who is the parent span of this current span, null if span is root parent.
The ID of the associated evaluation dataset.
CHUNKS Boolean to track whether or not the test case is autogenerated
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
GENERATION Used for tracking previous chat interactions for multi-chat test cases
Metadata for the test case
The date and time when the entity was overwritten or deleted in ISO format.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The type of annotation result.
llm_auto, human The selected choices(s) for the annotation result, in JSON form. For categorical questions, this is an object or list of objects (depending on if multiple selections are allowed). For free text questions, this is a string. For numeric or rating questions, this is a number.
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The time elapsed to generate this annotation in seconds.
The reasoning the LLM gave for the annotation it provided.
The ID of the associated annotation result.
The cost of the annotation in cents.
UNAUDITED, FIXED, APPROVED The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED CHUNKS The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
CHUNKS GENERATION, FLEXIBLE The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
Boolean of whether this interaction has been uploaded to s3 bucket yet, default is false
How much time the step took in milliseconds(ms)
The outcome of the operation
SUCCESS, ERROR The date and time when the entity was created in ISO format.
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
EXTERNAL_AI, EVALUATION, SGP_CHAT, AGENTS_SERVICE Identifies the application step
The id of the node in the application_variant config that emitted this insight
Type of the operation, e.g. RERANKING
TEXT_INPUT, TEXT_OUTPUT, COMPLETION_INPUT, COMPLETION, KB_RETRIEVAL, KB_INPUT, RERANKING, EXTERNAL_ENDPOINT, PROMPT_ENGINEERING, DOCUMENT_INPUT, MAP_REDUCE, DOCUMENT_SEARCH, DOCUMENT_PROMPT, CUSTOM, INPUT_GUARDRAIL, OUTPUT_GUARDRAIL, CODE_EXECUTION, DATA_MANIPULATION, EVALUATION, FILE_RETRIEVAL, KB_ADD_CHUNK, KB_MANAGEMENT, TRACER, STANDALONE The outcome of the operation
SUCCESS, ERROR The JSON representation of the input that this step received.
The JSON representation of the output that this step emitted.
The start time of the step
How much time the step took in milliseconds(ms)
The interaction ID this span belongs to
The id of the application variant this span belongs to
The JSON representation of the expected output for this step
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
The end time of the step, nullable, since it can be set to done at a later point in time.
The root-level ID where this span belongs to
Who is the parent span of this current span, null if span is root parent.
The ID of the associated evaluation dataset.
CHUNKS Boolean to track whether or not the test case is autogenerated
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
GENERATION Used for tracking previous chat interactions for multi-chat test cases
Metadata for the test case
The date and time when the entity was overwritten or deleted in ISO format.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The type of annotation result.
llm_auto, human The selected choices(s) for the annotation result, in JSON form. For categorical questions, this is an object or list of objects (depending on if multiple selections are allowed). For free text questions, this is a string. For numeric or rating questions, this is a number.
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The time elapsed to generate this annotation in seconds.
The reasoning the LLM gave for the annotation it provided.
The ID of the associated annotation result.
The cost of the annotation in cents.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED CHUNKS The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
CHUNKS GENERATION, FLEXIBLE The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
Boolean of whether this interaction has been uploaded to s3 bucket yet, default is false
How much time the step took in milliseconds(ms)
The outcome of the operation
SUCCESS, ERROR The date and time when the entity was created in ISO format.
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
EXTERNAL_AI, EVALUATION, SGP_CHAT, AGENTS_SERVICE Identifies the application step
The id of the node in the application_variant config that emitted this insight
Type of the operation, e.g. RERANKING
TEXT_INPUT, TEXT_OUTPUT, COMPLETION_INPUT, COMPLETION, KB_RETRIEVAL, KB_INPUT, RERANKING, EXTERNAL_ENDPOINT, PROMPT_ENGINEERING, DOCUMENT_INPUT, MAP_REDUCE, DOCUMENT_SEARCH, DOCUMENT_PROMPT, CUSTOM, INPUT_GUARDRAIL, OUTPUT_GUARDRAIL, CODE_EXECUTION, DATA_MANIPULATION, EVALUATION, FILE_RETRIEVAL, KB_ADD_CHUNK, KB_MANAGEMENT, TRACER, STANDALONE The outcome of the operation
SUCCESS, ERROR The JSON representation of the input that this step received.
The JSON representation of the output that this step emitted.
The start time of the step
How much time the step took in milliseconds(ms)
The interaction ID this span belongs to
The id of the application variant this span belongs to
The JSON representation of the expected output for this step
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
The end time of the step, nullable, since it can be set to done at a later point in time.
The root-level ID where this span belongs to
Who is the parent span of this current span, null if span is root parent.
The ID of the associated evaluation dataset.
CHUNKS Boolean to track whether or not the test case is autogenerated
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
GENERATION Used for tracking previous chat interactions for multi-chat test cases
Metadata for the test case
The date and time when the entity was overwritten or deleted in ISO format.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The type of annotation result.
llm_auto, human The selected choices(s) for the annotation result, in JSON form. For categorical questions, this is an object or list of objects (depending on if multiple selections are allowed). For free text questions, this is a string. For numeric or rating questions, this is a number.
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The time elapsed to generate this annotation in seconds.
The reasoning the LLM gave for the annotation it provided.
The ID of the associated annotation result.
The cost of the annotation in cents.
UNAUDITED, FIXED, APPROVED FLEXIBLE The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED CHUNKS The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
CHUNKS GENERATION, FLEXIBLE The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
Boolean of whether this interaction has been uploaded to s3 bucket yet, default is false
How much time the step took in milliseconds(ms)
The outcome of the operation
SUCCESS, ERROR The date and time when the entity was created in ISO format.
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
EXTERNAL_AI, EVALUATION, SGP_CHAT, AGENTS_SERVICE Identifies the application step
The id of the node in the application_variant config that emitted this insight
Type of the operation, e.g. RERANKING
TEXT_INPUT, TEXT_OUTPUT, COMPLETION_INPUT, COMPLETION, KB_RETRIEVAL, KB_INPUT, RERANKING, EXTERNAL_ENDPOINT, PROMPT_ENGINEERING, DOCUMENT_INPUT, MAP_REDUCE, DOCUMENT_SEARCH, DOCUMENT_PROMPT, CUSTOM, INPUT_GUARDRAIL, OUTPUT_GUARDRAIL, CODE_EXECUTION, DATA_MANIPULATION, EVALUATION, FILE_RETRIEVAL, KB_ADD_CHUNK, KB_MANAGEMENT, TRACER, STANDALONE The outcome of the operation
SUCCESS, ERROR The JSON representation of the input that this step received.
The JSON representation of the output that this step emitted.
The start time of the step
How much time the step took in milliseconds(ms)
The interaction ID this span belongs to
The id of the application variant this span belongs to
The JSON representation of the expected output for this step
The JSON representation of the metadata insights emitted through the execution. This can differ based on different types of operations
The end time of the step, nullable, since it can be set to done at a later point in time.
The root-level ID where this span belongs to
Who is the parent span of this current span, null if span is root parent.
The ID of the associated evaluation dataset.
CHUNKS Boolean to track whether or not the test case is autogenerated
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
GENERATION Used for tracking previous chat interactions for multi-chat test cases
Metadata for the test case
The date and time when the entity was overwritten or deleted in ISO format.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
EVALUATION_ANNOTATION, EVALUATION_AUDIT The ID of the account that owns the given entity.
NOT_READY, PENDING, COMPLETED The date and time when the task assignment expires in ISO format.
The entity that the task is associated with.
The status of the test case result. This should not be explictly set when creating a test case result. When patching a test case result, this field can be set to 'PENDING' to prevent the test case result from being marked 'COMPLETED'.
PENDING, COMPLETED, FAILED The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The ID of the account that owns the given entity.
The user who originally created the entity.
UNAUDITED, FIXED, APPROVED GENERATION The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
The type of annotation result.
llm_auto, human The selected choices(s) for the annotation result, in JSON form. For categorical questions, this is an object or list of objects (depending on if multiple selections are allowed). For free text questions, this is a string. For numeric or rating questions, this is a number.
The unique identifier of the entity.
The date and time when the entity was created in ISO format.
The time elapsed to generate this annotation in seconds.
The reasoning the LLM gave for the annotation it provided.
The ID of the associated annotation result.
The cost of the annotation in cents.
UNAUDITED, FIXED, APPROVED The result of the test case evaluation, in JSON form where the key is the question ID and the value is the result.
The time spent labeling in seconds.
The date and time when the entity was overwritten or deleted in ISO format.
The user who annotated the task.
The user who edited the task.
The date and time when the entity was archived in ISO format.
import SGPClient from 'sgp';
const client = new SGPClient({
apiKey: process.env['SGP_API_KEY'], // This is the default and can be omitted
});
async function main() {
const testCaseResultWithViews = await client.evaluations.testCaseResults.retrieve(
'evaluation_id',
'test_case_result_id',
);
console.log(testCaseResultWithViews);
}
main();{
"application_test_case_output": {
"application_variant_id": "<string>",
"evaluation_dataset_id": "<string>",
"test_case_version_id": "<string>",
"test_case_id": "<string>",
"output": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"schema_type": "GENERATION",
"application_interaction_id": "<string>",
"application_test_case_output_group_id": "<string>",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"interaction": {
"id": "<string>",
"input": {},
"output": {},
"aggregated": true,
"application_spec_id": "<string>",
"application_variant_id": "<string>",
"start_timestamp": "2023-11-07T05:31:56Z",
"duration_ms": 123,
"operation_status": "SUCCESS",
"operation_metadata": {},
"chat_thread_id": "<string>",
"interaction_source": "EXTERNAL_AI",
"created_at": "2023-11-07T05:31:56Z",
"trace_spans": [
{
"id": "<string>",
"application_interaction_id": "<string>",
"application_variant_id": "<string>",
"node_id": "<string>",
"operation_type": "TEXT_INPUT",
"operation_status": "SUCCESS",
"operation_input": {},
"operation_output": {},
"operation_expected": {},
"operation_metadata": {},
"start_timestamp": "2023-11-07T05:31:56Z",
"end_timestamp": "2023-11-07T05:31:56Z",
"duration_ms": 123,
"trace_id": "<string>",
"parent_id": "<string>"
}
]
}
},
"test_case_version": {
"evaluation_dataset_id": "<string>",
"schema_type": "GENERATION",
"test_case_data": {
"input": "<string>",
"expected_output": "<string>",
"expected_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
},
"artifact_ids_filter": [
"<string>"
]
},
"chat_history": {},
"autogenerated": true,
"test_case_metadata": {},
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z"
},
"task": {
"id": "<string>",
"task_type": "EVALUATION_ANNOTATION",
"task_entity_parent_id": "<string>",
"task_entity_id": "<string>",
"account_id": "<string>",
"assigned_to": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
},
"assignment_expires_at": "2023-11-07T05:31:56Z",
"status": "NOT_READY",
"priority": 123,
"task_entity": {
"label_status": "PENDING",
"audit_status": "UNAUDITED",
"audit_required": true,
"audit_comment": "<string>",
"application_spec_id": "<string>",
"evaluation_id": "<string>",
"evaluation_dataset_id": "<string>",
"evaluation_dataset_version_num": "<string>",
"test_case_id": "<string>",
"test_case_evaluation_data": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"test_case_evaluation_data_schema": "GENERATION",
"result": {},
"completed_at": "2023-11-07T05:31:56Z",
"time_spent_labeling_s": 123,
"application_test_case_output_id": "<string>",
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"annotated_by_user_id": "<string>",
"edited_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z",
"edited_by": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
}
}
},
"metrics": {},
"custom_metrics": [
{
"id": "<string>",
"type": "<string>",
"name": "<string>",
"output": 123
}
],
"audit_tasks": [
{
"id": "<string>",
"task_type": "EVALUATION_ANNOTATION",
"task_entity_parent_id": "<string>",
"task_entity_id": "<string>",
"account_id": "<string>",
"assigned_to": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
},
"assignment_expires_at": "2023-11-07T05:31:56Z",
"status": "NOT_READY",
"priority": 123,
"task_entity": {
"label_status": "PENDING",
"audit_status": "UNAUDITED",
"audit_required": true,
"audit_comment": "<string>",
"application_spec_id": "<string>",
"evaluation_id": "<string>",
"evaluation_dataset_id": "<string>",
"evaluation_dataset_version_num": "<string>",
"test_case_id": "<string>",
"test_case_evaluation_data": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"test_case_evaluation_data_schema": "GENERATION",
"result": {},
"completed_at": "2023-11-07T05:31:56Z",
"time_spent_labeling_s": 123,
"application_test_case_output_id": "<string>",
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"annotated_by_user_id": "<string>",
"edited_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z",
"edited_by": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
}
}
}
],
"annotation_results": [
{
"question_id": "<string>",
"annotation_type": "llm_auto",
"test_case_result_lineage_id": "<string>",
"selected_choice": {
"label": "<string>",
"value": "<string>",
"audit_required": true
},
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"llm_auto_eval_metadata": {
"cost": 123,
"prompt_tokens": 123,
"completion_tokens": 123,
"time_elapsed_s": 123,
"llm_reasoning": "<string>",
"annotation_result_id": "<string>"
}
}
],
"label_status": "PENDING",
"audit_status": "UNAUDITED",
"audit_required": true,
"audit_comment": "<string>",
"application_spec_id": "<string>",
"evaluation_id": "<string>",
"evaluation_dataset_id": "<string>",
"evaluation_dataset_version_num": "<string>",
"test_case_id": "<string>",
"test_case_evaluation_data": {
"generation_output": "<string>",
"generation_extra_info": {
"schema_type": "CHUNKS",
"chunks": [
{
"text": "<string>",
"metadata": {}
}
]
}
},
"test_case_evaluation_data_schema": "GENERATION",
"result": {},
"completed_at": "2023-11-07T05:31:56Z",
"time_spent_labeling_s": 123,
"application_test_case_output_id": "<string>",
"invalidated_at": "2023-11-07T05:31:56Z",
"id": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"account_id": "<string>",
"created_by_user_id": "<string>",
"annotated_by_user_id": "<string>",
"edited_by_user_id": "<string>",
"archived_at": "2023-11-07T05:31:56Z",
"edited_by": {
"id": "<string>",
"first_name": "<string>",
"last_name": "<string>",
"email": "<string>",
"preferences": {}
}
}