Senior Software Engineer, AI Evaluation Infrastructure (Remote - Worldwide)

A fast-growing AI company that works with frontier AI labs on datasets, reinforcement learning environments, and benchmarks is hiring a senior backend engineer for its AI evaluation platform. The platform runs coding agents and language models through benchmark tasks at scale and turns the results into trustworthy scores. The role suits an engineer who can take a loosely defined request and deliver it fully, including tests, migrations and edge cases.

What you'll do

• Design and ship features end to end, covering the data model, APIs, background workers and, when needed, the user interface.
• Build and improve how agents and models are evaluated, including running sandboxed trials, scoring results and aggregating thousands of runs.
• Improve automated task quality checks such as verifiers, rubric-based and LLM-as-a-judge grading, and decisions on whether a task is clear and fair.
• Integrate new models, agents and benchmarks through general designs rather than one-off special cases.
• Keep the system fast and reliable under load through efficient queries, safe migrations and sound concurrency.
• Review teammates' pull requests against written engineering guidelines.

What you need

• Typically 5+ years building and operating production backend systems, with ownership of at least one service from design through on-call.
• Strong Python, with experience in Django or a similar framework, PostgreSQL and a task queue such as Celery.
• Hands-on experience with LLM evaluation beyond calling a model API, including how benchmark tasks are built and how agent trials are scored and compared.
• Good judgment on task quality, including spotting ambiguous instructions, weak or gameable verifiers and leaked answers.
• Clear written communication in English, for pull request descriptions and design notes.
• Comfort with TypeScript and React for the occasional visible frontend change.

Open to applicants anywhere in the world.

How to apply

Sign up free on i'llTip to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.