Note: The job is a remote job and is open to candidates in USA. Toast creates technology to help restaurants and local businesses succeed in a digital world. They are seeking a Staff Machine Learning Engineer to act as a technical leader on the ML Platform team, driving architectural direction and delivering foundational infrastructure that supports ML capabilities.
Responsibilities
- Own technical direction of the ML Platform — feature store, model hosting and serving, experimentation, training infrastructure — driving architectural decisions around scalability, reliability, latency, and cost
- Lead design and delivery of large-scope platform initiatives from conception through production, coordinating across ML, data, and infrastructure teams
- Identify and resolve systemic technical challenges: online/offline feature parity, model deployment friction, experimentation velocity, GPU utilization, cross-team dependencies
- Set and maintain a high engineering quality bar through hands-on code contributions, design reviews, and mentorship of platform and ML-adjacent engineers
- Partner with ML engineering, data science, product, and platform leadership to translate ML strategy into technical roadmaps
- Define the paved paths ML teams use to ship models safely — from feature registration through canary rollout, monitoring, and rollback
- Leverage AI-augmented development tools to increase development velocity and code quality
Skills
- 8+ years delivering complex backend or infrastructure systems at scale
- Direct experience building or operating core ML infrastructure — feature stores, model serving, experimentation platforms, training orchestration, or equivalent
- Mastery of a modern backend language such as Python, Java, Kotlin, Go, or Scala
- Deep proficiency with distributed systems concepts: consistency, latency, throughput, fault tolerance, and observability
- Strong understanding of data modeling, query languages, and the online/offline data patterns that underpin ML systems
- Demonstrated technical leadership, with ability to drive cross-team alignment and influence engineering, product, and business stakeholders
- Bachelor's degree in Computer Science or a related field, or equivalent practical experience
- Hands-on experience with open-source or commercial ML platform components (e.g. Tecton, MLflow, SageMaker, Databricks)
- Experience building or operating experimentation / A-B testing platforms at scale
- Familiarity with real-time streaming systems (Kafka, Flink, Spark Streaming) and their use in feature computation
- Experience serving LLMs or large deep-learning models in production, including GPU capacity planning and inference optimization
- Comfort with Kubernetes and modern cloud-native infrastructure
- Prior work supporting internal-developer-facing platforms with a product mindset
Benefits
- Cash compensation (overtime, bonus/commissions if eligible)
- Equity
- Benefits
- Hybrid work model that fosters in-person collaboration while valuing individual needs
- Reasonable accommodations for persons with disabilities to enable them to access the hiring process
Company Overview
Company H1B Sponsorship