AI Evaluation Associate (Remote - Worldwide)
Pay: $30/hr - $60/hr (hourly rate). This is a remote role with a company that runs human-led evaluations of AI models. You will help measure what models can and cannot do by writing rigorous test questions, building scoring rubrics, and comparing model outputs side by side. Pay depends on the project, subject area and your experience, and is paid for hours worked.
What you'll do
- Write questions and tasks that target specific model capabilities.
- Build rubrics that describe strong, partial and weak responses, and write reference answers and edge cases.
- Grade model outputs against the rubric and clearly explain each score.
- Review incorrect answers, group recurring error patterns and document the causes.
- Compare several models on identical prompts and flag benchmark items that no longer measure reliably.
What you need
- Zero to five years of professional experience. Prior AI data work is not required.
- Background in QA, test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or a subject that involves rigorous question writing.
- Experience writing test cases, exam questions, rubrics or evaluation tasks and applying written criteria consistently.
- Ability to use spreadsheets or Python to organize results and analyze them.
- Clear, professional written English.
Open to applicants anywhere in the world.
How to apply
Get the free i'llTip app to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.
Apply on i'llTipPrefer the website? Continue on illtip.comOr get the app: Android · iPhone
🔔 Get new jobs like this first: join Tech & AI Jobs on Telegram · or get personal job alerts
More AI & Machine Learning jobs
AI Safety Testing Associate (Remote - Worldwide)
Pay: $30/hr - $60/hr (hourly rate). The posting does not restrict eligibility by country. This is an expert talent pool supporting AI model evaluation projects. Members are matched to safety testing work based on their b…
Physical AI Data Collection Associate (Remote - Worldwide)
Pay: $20/hr - $40/hr (hourly rate). Some on-site projects are also offered. This is a data-collection role supporting a physical AI network that runs paid projects to help train embodied AI systems. You will record real-…
Founding Harness Engineer, AI Agent Runtime (Remote - Eastern Europe)
Working-hour overlap with the team is still to be confirmed. This is a founding engineering role at an early-stage AI company building an AI coworker product for team workplaces. You will own the runtime that lets an AI …
Lead Engineer, AI Platform (Remote - EMEA)
A fast-growing advertising technology company that uses generative AI to help brands and small businesses produce creative content is hiring a hands-on technical lead for the platform that runs its AI agents. The product…
Related job searches
Remote AI JobsAI jobsEntry-Level jobsWork From Home jobs in NigeriaWork From Home jobs