Warm Start

A warm start occurs when an inference request lands on a server where the model weights are already loaded and kernels are compiled, allowing immediate execution with minimal latency.

  Cold vs Warm Latency

  Maintaining Warm State

1. Keep a minimum replica count even during off-peak hours.
2. Periodically send synthetic pings to prevent idle shutdown.
3. Use container snapshot restore to reload GPU memory quickly.

  Design Trade-offs

• Holding warm capacity incurs extra cost during low demand.
• Aggressive idling saves money but increases p99 latency when traffic resumes.

  Current Trends (2025)

• Predictive autoscalers maintain rolling warm pool sized from forecasted five-minute demand window.
• NVSwitch memory sharing warms multiple models concurrently on the same GPU¹.

  Implementation Tips

1. Separate health checks from synthetic warmers to avoid skewing metrics.
2. Track warm-hit ratio (warm / total requests) as KPI.
3. Tag warm nodes to avoid being selected for large batch offline jobs.

  References

1. NVIDIA Whitepaper, Multi-Model Serving with NVSwitch, 2025. ↩