How to create and evaluate a summarization application.
application_variant_id
in the top right:
SUMMARIZATION
and follow the formatting instructions. Supported file types include CSV
, XSLX
, JSON
and JSONL
.
SUMMARIZATION
as the Dataset Type and pick a dataset that matches that schema. If the dataset has multiple version, you will have to select the version of the dataset for which you want to upload the outputs. Ensure to follow the upload instructions for the file type you are choosing. We support the same file types as for the evaluation dataset upload, CSV
, XSLX
, JSON
and JSONL
.
Contributor
evlauations and no auto-evaluation from the UI.
SummarizationTestCaseSchema
function is a helper function that allows you to quickly create a Summarization Evaluation through the flexible evaluations framework. This function assumes the application you want to evaluate has a document as an input and the expected summary of the document as the expected input. It takes in a document
and a expected_summary
and creates a test case object.
In order to use this function, you start by creating a list of data for your test cases. The test case data is represented as an object that contains a document
(a string containing the text of a document you want to summarize) and expected_summary
(the expected summarization of this document) key.
SummarizationTestCaseSchema
function.