Math Reasoning RLHF Annotator

Pay: $35–55/hour

You'll spend your time evaluating model-generated solutions to math problems — ranging from algebra and calculus to combinatorics and proof-based reasoning. Your core task is to compare two or more responses side by side, rank them by correctness and clarity, and flag specific reasoning errors (wrong setup, arithmetic mistakes, skipped logical steps) using a structured rubric. You won't just say "this one is better" — you'll annotate exactly where and why.

The workflow runs inside a browser-based annotation tool. Each task presents a problem statement and two to four model responses. You'll read each solution, verify the math independently, score it across dimensions like factual accuracy, step completeness, and explanation quality, then submit a ranked preference with written rationale. Tasks average 8–15 minutes each depending on problem complexity. You'll work asynchronously across flexible hours with a weekly throughput target.

Strong candidates hold at least an undergraduate background in math, engineering, physics, or a related quantitative field. You should be comfortable doing manual verification — not just pattern-matching for "looks right." Familiarity with LaTeX notation is a plus since many prompts and responses are rendered that way. You won't need to write code, but you will need to show your own working when the rubric asks for it.

Skills & requirements

• Mathematical reasoning
• Step-by-step solution verification
• Preference ranking
• Rubric-based annotation
• LaTeX reading comprehension
• Error classification

Category: RLHF