> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation Metrics

> Quantify the performance of an evaluation.

# Available Metrics

When creating an evaluation run, these five metrics are available out of the box:

* **Bleu** - Measure quality of translation

* **Rouge** - Measure quality of summary or translation

* **Meteor** - Measure quality of translation using semantic matching

* **Cosine Similarity** - Assess similarity by measuring their distance in vector space

* **F1 Score** - Measure token-level precision and recall

<img src="https://mintcdn.com/scalegp/oOTRqBiggqYXlYW4/images/evaluation-metrics-overview.png?fit=max&auto=format&n=oOTRqBiggqYXlYW4&q=85&s=274ea238a77f9a95b804c611c515bace" alt="Evaluation metrics selection interface" width="1238" height="690" data-path="images/evaluation-metrics-overview.png" />

The available fields to compare for the metrics are defined by the schema of the dataset. For example, summarization datasets will have `document`, `summary`, `expected_summary` as choices for comparison.

## Bleu

Library: `nltk.sentence_bleu`

Non-configurable parameters:

* weights - `0.25` for all n-grams

* tokenizer - `nltk.tokenize.word_tokenize`

## Rouge

Library:`rouge_score.rouge_scorer`

Configurable parameters:

* `score_types: List[str]` - defines which rouge-n metrics will be outputted. Defaults to `["rouge1", "rouge2", "rougeL"]`

## Meteor

Library: `nltk.translate.meteor_score`

Non-configurable parameters:

* stemmer -`PorterStemmer`

* wordnet -`nltk.corpus.wordnet`

* `alpha=0.9`, `beta=3.0`, `gamma=0.5`

## Cosine Similarity

Library: `sklearn.metrics.pairwise.cosine_similarity`

Non-configurable parameters:

* embedding model - `sentence-transformers/all-MiniLM-L12-v2`

## F1 Score

Matching algorithm: tokenize the case-insensitive ground truth and predicted answer, then do exact matching without considering the order of the tokens.

Non-configurable parameters:

* tokenizer - `nltk.tokenize.word_tokenize`