Research Engineer, Benchmarks (Visa Sponsorship - Singapore, Singapore)
Visa sponsorship available. A small, technical team in an early-stage AI company is building benchmarks that test frontier AI agents on realistic, domain-specific workflows. The role owns evaluation design and implementation, with a focus on making sure benchmark results are rigorous, credible, and reflective of real-world performance.
What you'll do
- Design, implement, and maintain benchmarks that evaluate frontier agents on domain-specific tasks.
- Work with subject-matter experts to turn real workflows into realistic tasks and clear evaluation criteria.
- Build reliable infrastructure for running models and agents against benchmark tasks at scale.
- Develop metrics and analyses that measure benchmark difficulty, reliability, and failure modes.
- Run validation studies to check whether benchmark results correlate with real-world evaluations and user needs.
- Write clear documentation and reports for technical audiences.
What you need
- At least two years of experience in research engineering or machine learning engineering, including building AI benchmarks, evaluation infrastructure, or agent environments.
- Strong technical grounding with first-principles reasoning about task design, scoring, and failure modes.
- Proficiency in Python, Docker, and Linux, including experience operating infrastructure for AI models or agents.
- Hands-on experience designing and running evaluations for AI agents or large language models.
- Experience with metrics, statistical analysis, or validation studies, and translating workflows across domains into benchmark tasks.
- Strong technical writing and communication skills. Published work on benchmarking, evaluation, or model failure modes is a plus.
- Experience with reinforcement learning pipelines, data generation, or agent evaluation is a plus.
- Comfortable working independently in a fast-paced, early-stage environment with open-ended problems.
How to apply
Get the free i'llTip app to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.
Apply on i'llTipPrefer the website? Continue on illtip.comOr get the app: Android · iPhone
🔔 Get new jobs like this first: join Tech & AI Jobs on Telegram · or get personal job alerts
More Visa sponsored jobs jobs
Founding Engineer, Environments and Data (Visa Sponsorship - San Francisco, United States)
Visa sponsorship available. Applicants from any country may apply, but the role requires working on-site in San Francisco. This is a founding engineering role at an AI-focused startup building evaluation environments and…
Backend Engineer (Visa Sponsorship - Berlin, Germany)
Visa sponsorship available. An early-stage energy technology company is looking for a backend engineer to join its core infrastructure team. The team builds systems that power dynamic electricity plans and reliable billi…
Founding Engineer, NLP Systems & Infrastructure (Visa Sponsorship - Munich, Germany)
Visa sponsorship available. This is a founding engineering role at an early-stage AI company, where you would be the first technical hire. You will turn a research-stage prototype into a dependable production system and …
Research Engineer, Privacy and Anonymization (Visa Sponsorship - San Francisco, USA)
Visa sponsorship available. A company building AI infrastructure is hiring a research engineer to design privacy and anonymization systems. The work focuses on making sensitive real-world data safe to use for model train…
Full-Stack Software Engineer, Reinforcement Learning (Visa Sponsorship - San Francisco, USA)
Remote candidates may be considered if they can maintain 70% to 80% time zone overlap with San Francisco or Singapore. Visa sponsorship available. A fast-moving AI company is looking for a full-stack engineer to build th…