{c}

<LLM inference batching> means running multiple independent queries in parallel on a given model.

This can be used to overcome the fact that most single prompt inference will be heavily <memory bound>, see also: <theoretical peak performance of GPT inference>{full}. Batching helps increase the GPU compute utilization and balance it out with the memory.

Bibliography:
* https://medium.com/@yohoso/llm-inference-optimisation-continuous-batching-2d66844c19e9
* https://www.hyperstack.cloud/technical-resources/tutorials/static-vs.-continuous-batching-for-large-language-model-inference


LLM inference batching

For inferencing just a single prompt, things appear to be very obviously memory bound, i.e. bound by the transfer speeds of <VRAM> to GPU cache for loading model parameters into GPU so they can be used, supposing that the model fits in <VRAM>, which is the case for many popular models.

It is however possible to make fuller utilization of the GPU's compute power by running multiple independent queries in parallel, this way you load the subset of model weights that you need, and then use those to do part of the inference for multiple input prompts. With this it should be possible to reach full utilization.

Bibliography:
* https://www.reddit.com/r/LocalLLaMA/comments/1brcnps/is_inferencing_memory_bandwidth_limited/
* https://zeux.io/2024/03/15/llm-inference-sol/
8 https://jax-ml.github.io/scaling-book/


Ciro Santilli @cirosantilli 37

 Incoming links: Theoretical peak performance of GPT inference