Research Engineer, Benchmarks (Visa Sponsorship - Singapore)

Visa sponsorship available. This is a full-time, on-site research engineering role at an early-stage company that builds evaluation benchmarks for frontier AI agents. You will own the design and build of rigorous, domain-specific evaluations that help research teams and customers understand how agents perform on real-world work. The team is small and technical, so you will work independently and have a direct effect on the benchmarks the company produces.

What you'll do

• Design, implement, and maintain benchmarks that test AI agents on domain-specific tasks.
• Work with subject-matter experts to turn real professional workflows into realistic tasks and scoring criteria.
• Build reliable infrastructure to run models and agents against benchmark tasks at scale.
• Develop metrics and analyses that measure benchmark difficulty, reliability, and failure modes.
• Check how benchmark results relate to real-world performance and user needs.
• Write clear technical documentation and reports describing benchmark methods and results.

What you need

• Two to four years of experience in research engineering, machine learning engineering, or a closely related technical role, including at least two years building or delivering AI benchmarks, evaluation infrastructure, or agent environments.
• Strong proficiency in Python, Docker, and Linux.
• Hands-on experience designing, running, and maintaining evaluations for AI agents or large language models.
• Experience building infrastructure for running models or agents reliably, plus metrics, statistical analysis, or validation work.
• Ability to work with subject-matter experts across domains and analyze workflows to create useful evaluation tasks.
• Strong technical writing, attention to detail, first-principles reasoning, and comfort working independently in a fast-paced startup.
• Published research or technical writing on benchmarking, model evaluation, or failure modes is a plus. Experience with reinforcement learning pipelines, data generation, or agent evaluation is also relevant.

How to apply

Get the free i'llTip app to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.

Prefer the website? Continue on illtip.comOr get the app: Android · iPhone

🔔 Get new jobs like this first: join Tech & AI Jobs on Telegram · or get personal job alerts