A queuing system for annotators is designed to manage and streamline the distribution of annotation tasks among a team of annotators. A queue-based system makes sense for annotator task management because it provides organization, fairness, and efficiency in handling tasks while ensuring that the right task is assigned to the right person at the right time. Here’s a breakdown of how the system works:

​
1. Task Ingestion

  • Source of Tasks: Tasks could come from various sources, such as uploaded datasets, user-submitted requests, or our external API.
  • Task Metadata: Each task is accompanied by metadata, such as priority, time taken, flags, etc.

​
2. Task Queue

  • Centralized Queue: Tasks are placed in a centralized queue (first-in, first-out) based on the order that they were created in.

​
3. Annotator Assignment

  • Assignment: Each annotator has their own queue based on tasks that are assigned to them. If no tasks are assigned to an annotator, they will automatically be assigned the next task at the front of the centralized queue when they start another task.
  • Skipping Tasks: Each time an annotator skips a task, the task is returned to the centralized queue and placed at the back of the Queue.

​
4. Workflow Management

  • Status Tracking: Each task moves through various statuses: Pending → In Progress → Completed → Audit (optional).
  • Timers and Deadlines: Tasks may have timers to ensure goals are met. Annotators may see time taken for a given task.

​
5. Quality Control

  • Review and Feedback: Some tasks are reviewed by a QA/audit team or a second annotator to ensure accuracy.
  • Rework Mechanism: Tasks flagged for errors are returned to the queue for re-annotation or correction.

​
6. Annotator Dashboard

  • Real-Time Updates: Annotators see their task queue, progress, and performance metrics on a dashboard.
  • Claim/Auto-Assign: Annotators may either claim tasks manually or receive auto-assigned tasks based on the system logic.

​
7. Reporting and Analytics

  • Throughput Metrics: Track the number of tasks completed over time.
  • Annotator Performance: Measure individual performance (e.g., accuracy, speed).

​
Allowed Operations on the Queue

  1. Task Insertion
    • Automatic: New tasks are automatically added to the queue when they’re ingested.
    • Priority-Based: Tasks can be inserted at specific positions (e.g., top for urgent tasks) rather than the end (not supported yet in SGP)
  2. Task Retrieval
    • Annotators pull a task from the front of the queue, ensuring they’re working on the next available item.
  3. Reordering
    • Admins or system logic may reorder tasks based on changing priorities (e.g., an urgent new project arrives).
  4. Task State Updates
    • As tasks progress, their state changes: Not Started → In Progress → Completed.
  5. Queue Monitoring
    • View queue size, task statuses, and distribution of work.
    • Analytics help optimize the queue by identifying bottlenecks or overdue tasks.

​
Disallowed Operations on the Queue

  1. Random Task Assignment
    • Annotators are not allowed to randomly pick tasks from the queue, as this disrupts prioritization and fairness.
    • Cherry-picking easy tasks over complex ones can lead to inefficiency and inequality. In most systems are not able to see the tasks before they claim it. In SGP, annotators can view a task before claiming.
  2. Bypassing Priority Rules
    • Tasks with lower priority can not be processed before high-priority tasks unless explicitly overridden by system logic.
  3. Manual Task Deletion
    • Tasks can not be arbitrarily deleted from the queue to prevent data loss or missed deliverables.
    • Deletions can only occur through an admin-verified process.
  4. Batch Assignments
    • Annotators should not be allowed to pull multiple tasks at once unless the queue system supports batch assignments (e.g., microtasks).
  5. Infinite Locking
    • Tasks should not remain “locked” to an annotator indefinitely. If an annotator doesn’t complete or reject a task in a reasonable timeframe, the task should return to the queue for reassignment.