About the role
About Ollive Ollive builds the technology infra for AI liability insurance AI agents. Before we underwrite coverage for an agent, we have to know exactly how it breaks — a big part of our product is continuously finding the risk exposure and vulnerabilities, failure modes, and residual risk in AI applications and pricing that risk. Our assessments map findings against OWASP LLM Top 10, MITRE ATLAS, and NIST AI RMF. About the Role We're looking for an engineer who has trained and fine-tuned models and knows how to stand up the infrastructure to do it reliably. You'll own the model layer behind our assessment engine: fine-tuning models for risk classification and adversarial evaluation, optimizing them to run at scale, and building the training/inference infra that keeps our assessment pipeline fast and cost-controlled. You'll work directly with the founders. Key Responsibilities Fine-tune: adapt open weight models for — risk assessment, adversarial/jailbreak detection, and evaluation of AI agents Design and own the training infrastructure: data pipelines (GT development), distributed training setup, experiment tracking, reproducibility and rails to run the models Optimize: quantization, distillation, and inference tuning — balancing latency, throughput, and cost Stand up and manage GPU/compute infra (cloud and/or self-hosted), including cost modeling and capacity planning Design evaluation harnesses to measure model quality, robustness, and drift against our Risk Framework Deploy, serve, and monitor models in production, including handling throttling, scaling, and failover Collaborate with the Ollive AI Risk Labs and insurance team to translate risk methodology into model behavior
Required Skills Machine Learning and Deep Learning fundamentals Experience with evals, red-teaming Hands-on fine-tuning of LLMs / foundation models (LoRA/QLoRA, full fine-tuning) PyTorch and the Transformers ecosystem Quantization, hyperparameter tuning, and inference optimization (Speculative Decoding) Distributed training (DeepSpeed, FSDP, Accelerate, or similar) Setting up and operating training/serving infra — GPU provisioning, containerization, serving frameworks (SGLang, vLLM preferred) Preferred Skills Experience training models from scratch or working with 2B+ parameter models Experience with cloud inference at scale (AWS Bedrock, SageMaker) and managing quota/throttling Good to have Familiarity with AI security or adversarial ML (Promptfoo, Garak, Giskard, or similar) Exposure to regulated or compliance-heavy environments (security, fintech, insurance)