{c}
{wiki}

= GPT
{c}
{synonym}
{title2}

\Video[https://www.youtube.com/watch?v=YA0pzBYAV2Q]
{title=5 Years of GPTs by Finbarr Timbers}
{description=2023. Good talk.}

\Video[https://www.youtube.com/watch?v=eMlx5fFNoYc]
{title=Attention in transformers, step-by-step by <3Blue1Brown>}
{description=2024. Uses on <GPT-3> as basis.}

\Video[https://www.youtube.com/watch?v=9-Jl0dxWQs8]
{title=How might LLMs store facts by <3Blue1Brown>}
{description=Followup to the above video.}


Generative pre-trained transformer

{c}
{title2=175 B parameters}
{title2=2020-06}

* Vocabulary size (V): 50,257
* Hidden size (d_model): 12,288
* Context length	2048
* Q V size: (d_head): 128
* Attention heads	(h): 96
* FFN inner size (d_ff)	4 × 12,288 = 49,152
* Layers (L): 96


Ciro Santilli @cirosantilli 37

 Incoming links: GPT-3