Adversarial Evaluator, AI Search Red Teaming Evaluator

Contract · Remote · 5-week pilot · 15 openings

About the role

Truelancer AI is building a specialist red team for an AI search safety pilot with a global enterprise client. You’ll work as an individual contributor. You’ll own specific attack surfaces, design and run adversarial scenarios against an AI search and RAG system, and produce evidence that holds up under QA review.

We value complete evidence and defensible coverage more than prompt volume.

What you’ll do

• Plan attacks: form risk hypotheses and design original, bounded adversarial scenarios.

• Run attacks: execute multi-turn, adaptive prompt attacks, including context escalation, on authorised systems.

• Evaluate outputs for these failure types:

  • grounding errors

  • source contradictions

  • stale information

  • unverifiable citations

  • trust-boundary violations, such as indirect prompt injection

• Record evidence that can be reproduced:

  • exact prompt sequences

  • session and version context

  • timestamps

• Submit finding cards to QA, and adjust your judgements as rubrics and guidelines are updated.

• Classify each outcome as a candidate finding, a quality issue or inconclusive. Know when to stop and escalate an unsafe or uncertain case.

• Stay within scope: keep to client-approved boundaries and privacy policies at all times.

• Keep current on client taxonomies, system changes, policy revisions and new risk definitions.

• Work with the team: the Red-Team Lead, Project Manager and QA analysts.

Who you are

• Experience: 3 or more years in one or more of the following:

  • AI red teaming, LLM evaluation or GenAI testing and QA

  • Trust & Safety, content safety or content policy operations

  • Fraud, abuse or risk investigations

  • Responsible AI, AI governance or AI safety

• Methods: hands-on experience with LLM and model evaluation, RLHF or human-feedback work, and prompt engineering.

• Adversarial skill: working knowledge of prompt injection, jailbreak testing and adversarial techniques.

• Mindset: sceptical and analytical. You can rule out false positives and check citations against sources.

• Communication: excellent written and spoken US English, suited to structured reporting.

• Detail: very high attention to detail and ownership of what you deliver.

• Education: graduate or undergraduate degree.

How you’ll be measured

• Defensible coverage of your assigned surfaces, and how reliably your findings reproduce

• QA calibration and adjudication scores, with a target of at least 85% agreement

• Compliance with client boundary and privacy policies

• Quality of your finding cards and how well you take on calibration updates

Guidelines

Truelancer AI verifies each project before posting. Apply on Truelancer AI; never pay to get work.