Legal Reasoning AI Response Evaluator

Pay: $55–100/hour

You'll evaluate AI-generated responses to legal questions — covering case analysis, statutory interpretation, procedural questions, and litigation strategy. Your primary task is assessing whether a model's answer reflects sound legal reasoning, accurate doctrinal knowledge, and appropriate caveats. This isn't about copy-editing; you're judging whether the legal analysis would hold up in front of a practitioner, and flagging anything that could mislead a non-expert relying on the output.

Day-to-day work involves reading a legal prompt, reviewing the model's response, and scoring it across dimensions like analytical rigor, jurisdictional accuracy, and practical usefulness. For litigation-focused tasks, you'll also evaluate whether the model correctly identifies procedural posture, burden of proof, and viable arguments. Feedback is written in structured rubric fields, with an optional free-text section for nuanced critique. Tasks are asynchronous — you work batches on your own schedule through a browser-based annotation platform.

A minimum of 3 years of litigation or substantive legal practice is required. Familiarity with U.S. federal and state court procedure is expected; experience in additional jurisdictions is a bonus. Law professors and clerks are also strong candidates. You do not need to be currently practicing, but you must be able to credibly evaluate whether legal reasoning is sound, not just grammatically correct.

Skills & requirements

• Legal reasoning and case analysis
• Litigation and civil procedure
• Statutory and regulatory interpretation
• Analytical writing
• Jurisdictional knowledge (US federal/state)

Category: RLHF