Reviewer quality
AI QA reviewer badges.
The proposed way to qualify reviewers by task type instead of treating every worker as interchangeable: six badges, each earned through sample tasks in that category.
- Badges
- Six, one per review type
- Earned by
- Calibration tasks with evidence
- Status
- Proposed for the pilot
Badges by task type
Each badge maps to one fixed review service.
| Badge | What the reviewer checks |
|---|---|
| Fact-check reviewer | Checks claims against sources and flags hallucinations, weak evidence, and misleading statements. |
| Citation verifier | Checks whether citations exist, are relevant, and actually support the claim. |
| Support reply QA | Reviews customer-facing AI replies for policy fit, tone, correctness, and escalation risk. |
| RAG groundedness reviewer | Compares AI answers with retrieved documents and identifies unsupported statements. |
| Product listing QA | Checks ecommerce listings for accuracy, completeness, risky claims, and clarity. |
| Agent audit reviewer | Reviews AI-agent output and available trace or context for whether an action is appropriate. |
Calibration standards
How a badge would be earned and kept.
- A reviewer completes sample tasks in the category before receiving the badge.
- Reports must include human-written rationale and evidence, not only a score.
- Repeated low-effort or AI-only reviews lose eligibility.
- Quality is tracked by category, not only as one global rating.
Status. This is the proposed framework for the managed pilot. Badges are not yet shown on worker profiles, and a badge would describe completed calibration work, not a background check or identity check.
Related pages
Neighbouring pages in the AI QA cluster, each with a different scope.
Start with one category.
List a first AI QA service, complete a few reviews with evidence, and the calibration record follows.