Skip to main content

Overview

This guide is designed for human annotators to understand how to label evaluations created through the human evaluations creation flow. For information on creating contributor evaluations, see Contributor Evaluation.
Flag for Review and auditing are off by default. Ask your Scale team to enable them for your account.

Start Labeling

Start Labeling To start labeling a task, click on the Label button next to a task that is assigned to your name. If you don’t have any assigned tasks, click Start Labeling to claim the next task in the queue.

Labeling Tasks

Labeling UI When you open a task, you will see the information related to an evaluation run on the left side of the task screen. On the right side, there will be a series of questions that you should answer as the annotator.
  • Save: There is no save button, and answers are not saved until you submit. If you close a task with unsubmitted answers, the UI asks Discard changes?, and discarding loses them.
  • Skip: Skip. If you are unsure about how to answer this task, use the Skip button. Skipping unassigns the task from you, moves it to the end of the queue, and opens the next task.
  • Submit: Submit. When you are done with a task, use the Submit button to submit it. Submitting a task allows it to be audited by an auditor. You are allowed to edit a task after you submit if it has not yet been audited.
  • Flag for Review: If you want to submit a task but have concerns about the task you want to flag to an auditor, toggle the flag for review button and leave a comment. The auditor will see your comment and see that you flagged this for review.