QA Reviewer / Adjudicator

Check annotation and transcription work done by others, as paid work from home. It suits precise people who give clear written feedback and reject work that fails the criteria. Prior annotation, transcription or review experience is a plus, not a requirement.

What the work is for

Your reviews decide which work goes into a delivered dataset. Work must be reviewed before a project accepts it.

The work

• Review sampled batches against the written acceptance criteria. Score them using the project's rating criteria.
• Resolve disagreements between annotators. Record your reason and give clear feedback.
• Measure how often annotators agree, and help calibrate them where they drift apart.
• Report recurring errors and guideline gaps so the instructions improve.

What to expect

• Review and adjudication run in YPAI's own review console, next to the tool the annotators used.
• Each project has its own acceptance criteria, and batches can be annotation or transcription.

You need

• Precision, and the willingness to reject work that fails the written criteria.
• Structured, consistent judgement and clear written feedback.
• 18 years or older.

Good to know

• You judge by the written criteria, even where you would have labelled an item differently.
• Work comes project by project, and there can be gaps between projects.

Pay

$28–$40/hr. Rate shown is for the United States. This role is open in 12 countries, with local rates; pick your country on YPAI.

YPAI pays every two weeks for approved work via PayPal, Wise or bank transfer. Work is by invitation after your profile is reviewed.