AI Evaluation Specialist, Contract Talent Pool (Remote - Worldwide)
Pay: $30/hr - $60/hr (hourly rate). This is a contract position in a specialist talent pool that supports human-data projects evaluating AI models. Contributors design and run assessments that measure what models can and cannot do, and their work feeds into AI safety and capability research. Projects vary in length, typically from one week to several months, and pay depends on the project, subject area and experience.
What you'll do
- Write questions and tasks that target specific model capabilities
- Build scoring rubrics that define strong, partial and weak responses
- Produce reference answers and edge cases designed to challenge AI models
- Grade model outputs against rubrics and explain each scoring decision
- Analyze failed responses, group recurring error patterns and document findings
- Compare multiple models on identical prompts and check benchmarks for flawed or ineffective items
What you need
- Up to five years of professional experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or a relevant subject-matter area involving question writing
- Experience writing test cases, exam questions, rubrics or evaluation tasks, and applying written criteria consistently
- Comfort using spreadsheets or Python to organize and analyze results
- Clear written communication in English
- Familiarity with evaluation tools such as Inspect, lm-eval-harness, HELM or promptfoo is a plus, not required
Open to applicants anywhere in the world.
How to apply
Get the free i'llTip app to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.
Apply on i'llTipPrefer the website? Continue on illtip.comOr get the app: Android · iPhone
🔔 Get new jobs like this first: join Tech & AI Jobs on Telegram · or get personal job alerts
More AI & Machine Learning jobs
AI Safety Testing Associate (Remote - Worldwide)
The posting does not state country restrictions. This is a safety testing role with an expert network that supports AI model evaluation projects. You will look for ways AI systems can fail and document your findings so m…
RL Environment and Agent Evaluation Specialist (Remote - Worldwide)
Pay: $35/hr - $70/hr (hourly rate). Open to applicants in any country; project-based work with flexible hours. This is a paid talent pool for an AI company working on AI agents and reinforcement learning. Contributors he…
AI Response Evaluation and Feedback Associate (Remote - Worldwide)
Fully remote, no country restriction stated. Paid for hours worked. A project-based expert network that supports the evaluation and improvement of AI models. Contributors give structured feedback that helps model respons…
AI Data Annotation Specialist (Remote - Worldwide)
Pay: $40/hr - $75/hr (hourly rate). This is an expert-level data annotation role supporting advanced AI research teams. You will work on complex, multimodal datasets, take on the hardest and most ambiguous cases, and hel…
Senior Software Engineer I/II, Operational AI (Remote - US and Canada)
This is a senior engineering role on the Operational AI team at a public company that builds connected operations software for industries such as transportation, construction, agriculture and field services. The team tur…
Related job searches
Remote AI JobsAI jobsContract jobsWork From Home jobs in NigeriaWork From Home jobs