Generative Pre-trained Transformer

Generative Pre-trained Transformer (GPT) is a family of large-scale language models developed by OpenAI. GPT models are based on a transformer architecture that has been pre-trained on vast amounts of text data using unsupervised learning. The pre-training process involves training the model to predict missing words or next words in a sentence and then fine-tuning the model on a specific downstream task such as language translation, text classification, or question answering.

GPT-3, the latest and largest version of the GPT model, has been trained on a massive corpus of text data that includes books, articles, and websites, and contains 175 billion parameters, making it one of the largest language models ever created. GPT-3 can generate human-like text, complete sentences, paragraphs, and even entire articles, and can perform a wide range of NLP tasks with remarkable accuracy.

This interactive shows a GPT generating text one token at a time: the current token position is highlighted, three color-coded arches (recency, first-token, repetition) reveal which earlier words each attention head focuses on, and a compact “next-token distribution” illustrates the top choice plus a smaller alternative before the model commits the next word; use Play/Pause to run or halt, Step to advance a single token, Stop to halt without finishing, and Restart to begin again from the prompt—the sequence ends automatically when the model emits the final token (or hits the set limit).

The success of GPT models has sparked significant interest in the field of natural language processing and has led to the development of many other large-scale language models that are now being used in a wide range of applications, from chatbots and virtual assistants to text analysis and summarization tools.