SWE-Bench Task Auditor
Quick facts
- Pay
- $70–90 / hour
- Type
- Contract
- Location
- Remote — United States
- Skills
- Python, TypeScript, Java, C++, Django, Docker
- Platform
- Mercor — 291 open jobs
- Typical pay at Mercor
- $42–180/hour · median $85 (135 jobs on our site)
- Apply
- Free to apply — never pay to get AI training work
Pay: $70 – $90/hour
Commitment: Hourly, ~40 h/week
Open to: USA
Field: Software Engineering
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications
- 3+ years professional software engineering
- Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
- Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
- Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
- Prior code-review or task-grading experience
About Mercor
Mercor is an AI-driven hiring marketplace that places professionals on contracts with leading AI labs. You apply once, complete a short AI interview, and are then matched to open projects.