★ Saved
Mercor logo

AI Evaluation Rubric Auditor

Mercor Mercor ContractRemote $60–85/hr
Posted Nov 16, 2026 · ✓ Still open — checked 6 hours ago
Apply now →

Quick facts

Pay
$60–85 / hour
Type
Contract
Location
Remote
Platform
Mercor — 364 open jobs
Typical pay at Mercor
$40–180/hour · median $85 (273 jobs on our site)
Payout
Stripe, Wise (weekly)
Apply
Free to apply — never pay to get AI training work

Pay: $60 – $85/hour
Commitment: Hourly, ~15 h/week
Open to: CAN, USA, ARG, BOL, BRA, CHL, COL, ECU, GUY, PRY, PER, URY, GBR
Field: Miscellaneous

The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

\- Was the information available?

\- Does the rubric accurately grade what the prompt asked for?

\- Would a valid answer have been marked wrong?

\- Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves

\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

\- Experience providing feedback to writers/reviewers

\- Consistent availability

\- Can read a long agent transcript and find where it went wrong.

\- Precise writing, and you check the data instead of trusting the reference answer.

Mercor is an AI-driven hiring marketplace that places professionals on contracts with leading AI labs. You apply once, complete a short AI interview, and are then matched to open projects.