LLM inference batching means running multiple independent queries in parallel on a given model.
This can be used to overcome the fact that most single prompt inference will be heavily memory bound, see also: Section "Theoretical peak performance of GPT inference". Batching helps increase the GPU compute utilization and balance it out with the memory.

Tagged (1)

Ancestors (14)

  1. LLM inference optimization
  2. Large language model
  3. Text-to-text model
  4. AI text generation
  5. Generative AI by modality
  6. Generative AI
  7. AI by capability
  8. Artificial intelligence
  9. Machine learning
  10. Computer
  11. Information technology
  12. Area of technology
  13. Technology
  14. Home

Incoming links (1)

View article source

Discussion (0)

New discussion

There are no discussions about this article yet.

Articles by others on the same topic (0)

There are currently no matching articles.
See all articles in the same topic Create my own version