AIML - Sr Machine Learning Platform Engineer, Evaluation
Apple
- Location
- Onsite (Seattle, Washington)
- Employment
- Full-time
- Level
- Senior Level
Posted 6 days ago
About the role
Apple's AIML Evaluation team seeks an engineer to build systems that evaluate and refine foundation models and agents. You will design benchmarks, simulation environments, and pipelines that connect failure analysis to measurable model improvements across Apple's AI products.
Skills
Machine Learning
Distributed Systems
Python
PyTorch
JAX
LLM Evaluation
Reward Modeling
Prompt Optimization
Agentic Systems
Simulation Environments
Post-training
SFT
DPO
RLHF
Infrastructure Engineering
Failure Analysis
Full job details
We are seeking a highly skilled and experienced machine learning engineer to join AIML Evaluation to build the systems that evaluate and refine Apple's foundation models and agents.
As a key member of the team, you will help design and develop benchmarks, evaluators, simulation environments, and prompt and context optimization pipelines that drive quality improvements across Apple's AI experiences.
You will collaborate with product teams and the foundation model team to close the loop between observation and improvement, contributing datasets, environments, and reward signals that drive model and agent quality.
Most model evaluation stops at a scorecard. The failing traces get thrown away, even though they're the most relevant training signal we have. Our team puts them to work. We evaluate across offline benchmarks, device-in-the-loop simulation, and on-device observation in production, and we turn what we find into persona-grounded datasets and reward signals for foundation model post-training. In this role, you will build the systems that connect failure analysis to measurable model and agent improvements across a broad range of AI products at Apple. The path from finding a problem to fixing it is short: the loop you build finds the failure, produces the data and checks the result. The work reaches millions of users.
Strong background in machine learning and distributed systems. A proven track record of solving ambiguous problems end to end: defining the problem and the metric, building the solution, and showing measured impact. Experience leading technical work across multiple codebases, teams, and organizations. Experience building and maintaining ML infrastructure for evaluation, training, or deployment. 6+ years of professional experience as a software engineer, preferably in machine learning or a related field. Bachelor's or Master's degree in Computer Science or a related field, or equivalent practical experience.
Experience with LLM evaluation, LLM-as-judge, or reward modeling. Experience with prompt optimization, agent harness development, or post-training (SFT, DPO, RLHF). Proficiency in Python and ML frameworks such as PyTorch or JAX. Experience with agentic systems, simulation environments, or trajectory-based data generation. Familiarity with on-device or privacy-preserving ML. Understanding of how training data is used in post-training and how data choices affect model behavior. Excellent communication skills.
Description
Most model evaluation stops at a scorecard. The failing traces get thrown away, even though they're the most relevant training signal we have. Our team puts them to work. We evaluate across offline benchmarks, device-in-the-loop simulation, and on-device observation in production, and we turn what we find into persona-grounded datasets and reward signals for foundation model post-training. In this role, you will build the systems that connect failure analysis to measurable model and agent improvements across a broad range of AI products at Apple. The path from finding a problem to fixing it is short: the loop you build finds the failure, produces the data and checks the result. The work reaches millions of users.
Minimum Qualifications
Strong background in machine learning and distributed systems. A proven track record of solving ambiguous problems end to end: defining the problem and the metric, building the solution, and showing measured impact. Experience leading technical work across multiple codebases, teams, and organizations. Experience building and maintaining ML infrastructure for evaluation, training, or deployment. 6+ years of professional experience as a software engineer, preferably in machine learning or a related field. Bachelor's or Master's degree in Computer Science or a related field, or equivalent practical experience.
Preferred Qualifications
Experience with LLM evaluation, LLM-as-judge, or reward modeling. Experience with prompt optimization, agent harness development, or post-training (SFT, DPO, RLHF). Proficiency in Python and ML frameworks such as PyTorch or JAX. Experience with agentic systems, simulation environments, or trajectory-based data generation. Familiarity with on-device or privacy-preserving ML. Understanding of how training data is used in post-training and how data choices affect model behavior. Excellent communication skills.