Founding Engineer, Environments and Data (Visa Sponsorship - San Francisco, United States)
Visa sponsorship available. Applicants from any country may apply, but the role requires working on-site in San Francisco. This is a founding engineering role at an AI-focused startup building evaluation environments and benchmarks that measure whether AI agents can complete real-world company work at or above human level. The work centers on agent capability assessment and production readiness, and the results directly shape how agent competence is judged. The role is research-oriented, combining environment design, grading systems and benchmark validity.
What you'll do
- Design synthetic company environments with realistic email, records and files that agents can use across long, multi-step tasks.
- Build graders and evaluation systems that reliably assess advanced AI agents.
- Develop benchmarks whose scores correlate with real-world production performance.
- Identify and close evaluation failure modes, including reward hacking and benchmark gaming.
- Simulate software tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.
What you need
- Mid-level or senior engineering or research experience, including at least 1 to 2 years at a fast-growing or AI-focused startup.
- Hands-on AI research or engineering experience with synthetic data pipelines, benchmarks, evaluations or post-training.
- Experience building environments or benchmarks used to train or evaluate agents, and designing agent graders or evaluation systems.
- Experience simulating complex software systems for agent tool use, finding flaws in established benchmarks, and detecting and fixing reward hacking.
- Background in AI evaluation, benchmark design, simulation, agent systems or a related field. Startup founding or accelerator experience, or high-level competition experience in mathematics, programming, debate or games, is a plus.
- Able to work collaboratively and adapt in a fast-moving environment, on-site five days a week in San Francisco.
How to apply
Get the free i'llTip app to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.
Apply on i'llTipPrefer the website? Continue on illtip.comOr get the app: Android · iPhone
🔔 Get new jobs like this first: join Tech & AI Jobs on Telegram · or get personal job alerts
More Visa sponsored jobs jobs
Backend Engineer (Visa Sponsorship - Berlin, Germany)
Visa sponsorship available. An early-stage energy technology company is looking for a backend engineer to join its core infrastructure team. The team builds systems that power dynamic electricity plans and reliable billi…
Founding Engineer, NLP Systems & Infrastructure (Visa Sponsorship - Munich, Germany)
Visa sponsorship available. This is a founding engineering role at an early-stage AI company, where you would be the first technical hire. You will turn a research-stage prototype into a dependable production system and …
Research Engineer, Privacy and Anonymization (Visa Sponsorship - San Francisco, USA)
Visa sponsorship available. A company building AI infrastructure is hiring a research engineer to design privacy and anonymization systems. The work focuses on making sensitive real-world data safe to use for model train…
Full-Stack Software Engineer, Reinforcement Learning (Visa Sponsorship - San Francisco, USA)
Remote candidates may be considered if they can maintain 70% to 80% time zone overlap with San Francisco or Singapore. Visa sponsorship available. A fast-moving AI company is looking for a full-stack engineer to build th…
Research Engineer, QC Automation (Visa Sponsorship - Singapore)
Visa sponsorship available. An early-stage AI infrastructure team building reinforcement learning environments and model evaluation tooling is hiring a research engineer to own quality control automation for training dat…