Ciro Santilli Highly automated wrapper for various open source LLMs.
curl https://ollama.ai/install.sh | sh
ollama run llama2And bang, a download later, you get a prompt. On P14s it runs on CPU and generates a few tokens at a time, which is quite usable for a quick interactive play.
As mentioned at github.com/jmorganca/ollama/blob/0174665d0e7dcdd8c60390ab2dd07155ef84eb3f/docs/faq.md the downloads to under
/usr/share/ollama/.ollama/models/ and ncdu tells me:
--- /usr/share/ollama ----------------------------------
3.6 GiB [###########################] /.ollama
4.0 KiB [ ] .bashrc
4.0 KiB [ ] .profile
4.0 KiB [ ] .bash_logoutWe can also do it non-interactively with:
which gave me:
but note that there is a random seed that affects each run by default.
/bin/time ollama run llama2 'What is quantum field theory?'0.13user 0.17system 2:06.32elapsed 0%CPU (0avgtext+0avgdata 17280maxresident)k
0inputs+0outputs (0major+2203minor)pagefaults 0swapsSome other quick benchmarks from Amazon EC2 GPU, on Nvidia T4:
On Nvidia A10G:
0.07user 0.05system 0:16.91elapsed 0%CPU (0avgtext+0avgdata 16896maxresident)k
0inputs+0outputs (0major+1960minor)pagefaults 0swaps
0.03user 0.05system 0:09.59elapsed 0%CPU (0avgtext+0avgdata 17312maxresident)k
8inputs+0outputs (1major+1934minor)pagefaults 0swapsSo it's not too bad, a small article in 10s.
It tends to babble quite a lot by default, but eventually decides to stop.
TODO is it possible to make it deterministic on the CLI? There is a "seed" parameter somewhere: github.com/jmorganca/ollama/blob/31f0551dab9a10412ec6af804445e02a70a25fc2/docs/modelfile.md#parameter