Senior Software Engineer, AI Evaluation Infrastructure

A remote, full-time senior backend role building the infrastructure that frontier AI teams use to evaluate coding agents and language models. The work covers benchmark datasets, large-scale agent evaluation runs and automated task quality checks, from data models and background jobs through to user-facing screens. The organisation works closely with leading AI labs and enterprise clients, and the role suits an experienced engineer who works independently, makes sound design tradeoffs and takes fuzzy requests through to safe, well-tested delivery.

What you'll do

• Design and ship features end to end, including data model, API, background workers and UI where needed
• Build and improve how agents and models are evaluated, including sandboxed trials and scoring of results
• Improve benchmark task quality checks, such as verifiers and rubric-based or LLM-as-a-judge grading
• Integrate new models, agents and benchmarks in a general way rather than with one-off special cases
• Keep the system fast and reliable under growing load, including efficient queries and safe migrations
• Review teammates' pull requests against engineering guidelines

What you need

• Typically 5+ years building and operating production backend systems
• Strong Python with Django or a similar framework, PostgreSQL and a task queue such as Celery
• Hands-on experience with LLM evaluation, including how benchmark tasks are built and agent trials are scored
• Good judgment about task quality, including ambiguous instructions, weak verifiers and leaked answers
• Ownership of work from fuzzy request through to tests, migrations and edge cases
• Comfort with TypeScript and React when a change is visible in the frontend

Open to applicants anywhere in the world.

How to apply

Sign up free on i'llTip to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.