AI Engineering
Improving AI Inference Performance
Inference time is usually weights, the key-value cache, and the batch. Measure the slow path on the hardware that will serve traffic before changing the model.
This note is a starting point. A published study with measurements will appear on the projects page only after the work has been completed and checked.