Latency, Throughput, and Cost Trade-offs

How batching affects latency versus throughput in serving.

How does increasing batch size in online serving typically affect performance?