Senior Software Engineer, Model Evaluation (Remote - US only)
A early-stage AI company building software agents for industrial operations is hiring a senior engineer focused on research and evaluation. You will work alongside procurement and manufacturing specialists to design benchmarks, train models, and improve agents that run in real operational workflows. This is a hands-on role in a small team where you will help set the research and evaluation practices from the start. Equity is part of the compensation package.
What you'll do
- Design benchmarks and evaluation suites covering supplier sourcing, supplier qualification, compliance paperwork, purchase orders, freight, and order tracking.
- Partner with domain experts to define task scope, write reference answers and grading rubrics, and build automated graders.
- Train and fine-tune models directly, using evaluation results to decide what to try next.
- Investigate agent failures, build fixes, and confirm improvements with measured results.
- Establish evaluation and research standards as an early member of the team.
What you need
- At least 3 years of experience, with senior or staff-level ability.
- Experience designing and writing benchmarks or evaluations.
- Experience authoring tasks, including defining scope, creating reference solutions or rubrics, and checking task quality.
- Hands-on experience training and fine-tuning models, beyond calling models through APIs.
- Ability to work closely with non-ML domain experts and turn their knowledge into measurable tasks.
- Must be located in the United States. Supply chain, procurement, manufacturing, or early-stage startup experience is a plus.
Applicants must be based in: United States.
How to apply
Get the free i'llTip app to see the employer's details and apply. i'llTip's AI can tailor your résumé to this job, and you'll get alerts for similar roles.
Apply on i'llTipPrefer the website? Continue on illtip.comOr get the app: Android · iPhone
🔔 Get new jobs like this first: join i'llTip Jobs on Telegram · or get personal job alerts
More AI & Machine Learning jobs
Senior Software Engineer, Applied AI (Remote)
The posting does not list country restrictions, so confirm your eligibility to work for the employer before applying. A health benefits company focused on serving small and midsize businesses and their employees is hirin…
Industrial Plant Reliability & Maintenance Expert, AI Evaluation (Remote - Worldwide)
Pay: $90/hr - $90/hr (hourly rate). A project team is building a simulated industrial plant benchmark to test AI systems. The work is to judge whether the plant environment, the synthetic sensor data, and the fault scena…
ML Engineer
A remote contract role for an machine learning engineer open to candidates based in Nigeria. The work centres on building and validating machine-learning models, training pipelines and inference systems, along with the s…
AI Researcher / Engineer (Remote - Worldwide)
The posting does not list country restrictions, so confirm eligibility with the employer when applying. This is a full-time research and engineering role on a small, high-impact team building detection and forensic tools…
Technical Co-Founder, AI Agents (Remote - North America)
An early-stage AI venture is looking for a technical co-founder to build a personal AI agent that pairs automated task execution with human support. You would help define the product and engineering foundations from the …
Related job searches
Remote Software Engineer JobsSoftware Engineer jobsSenior jobsWork From Home jobs