How do you design a model serving system for low latency and high reliability?

Model serving is production engineering, not just ML. Key decisions: - Batch vs real-time inference - Model format and runtime (ONNX/TensorRT) - Caching and request shaping - Autoscaling and timeouts - Canary releases and rollback Monitor latency (p95/p99), errors, and input drift. Keep model/version metadata so you can reproduce and debug predictions quickly.