FlashcardAdvanced
Latency, Throughput, and Scaling
Latency measures per-request speed while throughput measures overall volume handled.
Question
How do latency, throughput, and scaling relate in model serving?
Click to reveal answer
Latency measures per-request speed while throughput measures overall volume handled.
How do latency, throughput, and scaling relate in model serving?