Member of Technical Staff - Model Optimization and Inference (Experienced)
About the role
What we're looking for?
We are seeking an experienced ML Infrastructure/Systems Engineer with at least 2 years of full-time experience in building and maintaining production-level ML systems. You should be comfortable designing scalable infrastructure from scratch, making informed design decisions by comparing various technologies, and have a track record of optimizing systems for latency, throughput, and cost. We're looking for someone with a broad understanding of the ML infra space, including inference infrastructure, real-time video streaming, and data engineering, who can own complex projects and debug distributed systems. Bonus points if you have experience with video or audio models and low-level optimization techniques like CUDA kernels.
What you'll do:
- Own end-to-end inference optimization across our model stack — LLMs, audio models, and diffusion-based components
- Implement and tune KV cache strategies for long-context conversations, including eviction policies, compression, and memory-efficient attention
- Evaluate, deploy, and extend inference serving frameworks (vLLM, SGLang, TensorRT-LLM, etc.) for our specific workloads
- Profile and benchmark end-to-end latency and throughput; identify and systematically eliminate bottlenecks
- Build internal tooling that makes optimization work faster and more rigorous — profiling viewers, end-to-end inference test harnesses, and other infrastructure that helps the team move quickly
- Accelerate diffusion model inference — consistency models, step distillation, caching strategies, and custom kernel optimizations
- Apply and develop quantization techniques (INT8, INT4, GPTQ, AWQ, and beyond) to reduce memory footprint and increase throughput without meaningfully degrading quality
- Work closely with research and infrastructure to ensure new models ship with optimized serving from day one