A CLI front-end for llama.cpp.
A decent test command as of llama.cpp 79e0b68c178656bb0632cb8602d2940b755077f8 tested on Ubuntu 25.04:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake ..
make -j
cd bin
time ./llama-cli \
  --no-display-prompt \
  --single-turn \
  --temp 0 \
  -c 16384 \
  -cnv \
  -m ~/Downloads/Llama-3.1-Tulu-3-8B-Q8_0.gguf \
  -n 1000 \
  -ngl 100 \
  -p 'What is quantum field theory?' \
  -t 10 |
tee output.txt
and that was deterministic due to --temp 0.
Also, this command ran 2x faster at 18 tokens/s for 1000 tokens on P14s on GPU via Vulkan than on CPU which is achievable by removing the -ngl 100.
As of llama.cpp 79e0b68c178656bb0632cb8602d2940b755077f8 there is a --parallel option but not sure what it does.

Ancestors (16)

  1. llama.cpp
  2. Ollama
  3. Open source LLM
  4. Large language model
  5. Text-to-text model
  6. AI text generation
  7. Generative AI by modality
  8. Generative AI
  9. AI by capability
  10. Artificial intelligence
  11. Machine learning
  12. Computer
  13. Information technology
  14. Area of technology
  15. Technology
  16. Home

Incoming links (3)

View article source

Discussion (0)

New discussion

There are no discussions about this article yet.

Articles by others on the same topic (0)

There are currently no matching articles.
See all articles in the same topic Create my own version