by Ciro Santilli (@cirosantilli, 37)
This appears to be the backend library of Ollama.
They have a CLI front-end named llama-cli.
askubuntu.com/questions/1461564/install-llama-cpp-locally has some tutorials for Ubuntu. There was no nicely pre-packaged one for Ubuntu 25.04, but build worked on 79e0b68c178656bb0632cb8602d2940b755077f8 In particular it exposed Vulkan support before Ollama did: github.com/ollama/ollama/pull/5059 and it did seem to work, using up my AMD GPU.
A CLI front-end for llama.cpp.
A decent test command:
time ./llama-cli \
  --no-display-prompt \
  --single-turn \
  --temp 0 \
  -c 16384 \
  -cnv \
  -m Llama-3.1-Tulu-3-8B-Q8_0.gguf \
  -n 1000 \
  -ngl 100 \
  -p 'What is quantum field theory?' \
  -t 10 |
  tee output.txt \
;
but it failed to be deterministic despite --temperature 0. This ran 2x faster at 18 tokens/s for 1000 tokens on P14s on GPU via Vulkan than on CPU which is achievable by removing the -ngl 100.

Ancestors (15)

  1. Ollama
  2. Open source LLM
  3. Large language model
  4. Text-to-text model
  5. AI text generation
  6. Generative AI by modality
  7. Generative AI
  8. AI by capability
  9. Artificial intelligence
  10. Machine learning
  11. Computer
  12. Information technology
  13. Area of technology
  14. Technology
  15. Home

Incoming links (1)

View article source

Discussion (0)

+ New discussion

There are no discussions about this article yet.

Articles by others on the same topic (0)

There are currently no matching articles.
See all articles in the same topic + Create my own version