Timing GPU Inference Latency
Find why GPU latency measurements come out unrealistically fast.
Codepython
import time
sample = sample.cuda()
model = model.cuda()
start = time.time()
output = model(sample)
latency = time.time() - start
print(f'Latency: {latency*1000:.2f} ms')What is the bug in this GPU latency measurement?