by Ciro Santilli (@cirosantilli, 37)
A CLI front-end for llama.cpp.
A decent test command:
time ./llama-cli \
  --no-display-prompt \
  --single-turn \
  --temp 0 \
  -c 16384 \
  -cnv \
  -m Llama-3.1-Tulu-3-8B-Q8_0.gguf \
  -n 1000 \
  -ngl 100 \
  -p 'What is quantum field theory?' \
  -t 10 |
  tee output.txt \
;
but it failed to be deterministic despite --temperature 0. This ran 2x faster at 18 tokens/s for 1000 tokens on P14s on GPU via Vulkan than on CPU which is achievable by removing the -ngl 100.

Ancestors (16)

  1. llama.cpp
  2. Ollama
  3. Open source LLM
  4. Large language model
  5. Text-to-text model
  6. AI text generation
  7. Generative AI by modality
  8. Generative AI
  9. AI by capability
  10. Artificial intelligence
  11. Machine learning
  12. Computer
  13. Information technology
  14. Area of technology
  15. Technology
  16. Home

Incoming links (2)

View article source

Discussion (0)

+ New discussion

There are no discussions about this article yet.

Articles by others on the same topic (0)

There are currently no matching articles.
See all articles in the same topic + Create my own version