You can build a real GPT-style language model on a home computer. It will not rival ChatGPT, answer factual questions reliably or understand instructions. But it can learn the patterns in a text file and generate new text one character at a time—and building it is one of the clearest ways to understand what “GPT architecture” actually means.
This project uses Python and PyTorch to create a small decoder-only Transformer from scratch. You will supply the training text, turn characters into tokens, assemble masked self-attention blocks, train the model to predict the next token and sample its output. A few thousand training steps can run on a modern laptop; CPU users can lower the settings and still see the model learn.
The short answer: GPT is not one mysterious algorithm. It is a stack of familiar parts: token and position embeddings, causal self-attention, small feed-forward networks, residual connections, normalization and a final layer that predicts the next token.
What you are building
The “G” means generative, because the model produces text. “P” means pre-trained, although our hobby model will only do the pre-training stage. “T” means Transformer, the attention-based architecture introduced in the 2017 paper Attention Is All You Need. GPT adapts that idea into a decoder-only model trained with a simple objective: given everything so far, predict what comes next.
Commercial language models use large tokenizers, enormous datasets, distributed training and additional alignment stages. Our version deliberately uses characters as tokens. That makes words slower to learn, but it removes a major subsystem and lets you inspect every step.
Set up the project
Create a folder, add a UTF-8 file named input.txt containing text you have the right to use, and create a virtual environment. A long journal export, your own stories or public-domain material can work. Aim for at least several hundred thousand characters; a tiny file encourages memorization.
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install torch
python tiny_gpt.py input.txt --steps 2000Use PyTorch’s current installation selector if your NVIDIA setup needs a CUDA-specific command. The script chooses CUDA first, Apple’s MPS backend second and CPU otherwise.
Step 1: Turn text into tokens
A neural network accepts numbers, not strings. The tokenizer sorts the unique characters in the corpus, assigns each one an integer and provides encode and decode operations. “cat” might become [12, 4, 31]; decoding reverses the mapping.
characters = sorted(set(text))
to_id = {ch: i for i, ch in enumerate(characters)}
tokens = torch.tensor([to_id[ch] for ch in text])The training batch selects random windows of 128 tokens. The target is the same window shifted one place to the left. If the input is “hell,” the desired next characters are “ello.” One batch therefore supplies many next-token lessons at once.
Step 2: Add meaning and position
An embedding table turns each token ID into a learned vector. A second table represents position zero through position 127. Adding the two tells the model both what character it received and where that character sits in the current window.
These vectors have 128 values in the companion implementation. That size is a learning surface, not a magic constant: increasing it gives the model more capacity while raising memory and compute costs.
Step 3: Build causal self-attention
Each attention layer projects the input into queries, keys and values. A query at one position scores the keys at other positions; softmax turns those scores into weights; the weighted values carry relevant earlier information forward. Four heads perform this operation in parallel, allowing different relationships to emerge.
The causal mask is essential. It fills scores for future positions with negative infinity before softmax, so a token cannot peek at the answer it is supposed to predict. This lower-triangular pattern is what makes the model autoregressive.
scores = query @ key.transpose(-2, -1) / math.sqrt(head_size)
scores = scores.masked_fill(~causal_mask, float("-inf"))
weights = torch.softmax(scores, dim=-1)
attended = weights @ value
Step 4: Stack Transformer blocks
A block combines attention with a feed-forward network. Layer normalization stabilizes the values entering each sublayer. Residual connections add the original signal back after attention and after the feed-forward network, giving gradients a direct route through the stack.
The hobby model uses four blocks, four attention heads and 128-value embeddings. The final linear layer produces one score for every possible next character. Cross-entropy loss measures how much probability the model assigned to the correct answer.
Step 5: Train, then sample
AdamW updates the weights after backpropagation. The script prints loss every 100 steps; the important sign is a sustained fall, not any single number. At generation time, it takes the final position’s scores, applies a temperature and samples one character. That character is appended to the input, and the loop repeats.
If the output is chaotic, train longer or provide more clean text. If it copies long passages, reduce training, add data or shrink the model. If memory runs out, lower batch_size, block_size, embedding_size or layer_count. Change one setting at a time and keep a validation split so lower training loss does not fool you.
What this model can—and cannot—teach you
This project demonstrates next-token prediction, learned embeddings, multi-head causal attention, residual blocks, optimization and autoregressive sampling. It also reveals why data matters: the model only learns statistical structure present in its corpus. It has no fact checker, browsing system, safety layer or source awareness.
Do not train it on private messages, scraped personal information or copyrighted collections you do not have permission to use. Generated text may repeat fragments of the training set, especially when the dataset is small. Keep experiments local and label outputs as synthetic.
What to try next
- Track validation loss and save the best checkpoint instead of only the final one.
- Replace character tokens with a byte or subword tokenizer.
- Use PyTorch’s optimized scaled dot-product attention after you understand the handwritten version.
- Add top-k sampling and compare temperatures such as 0.6, 0.8 and 1.0.
- Read a compact reference implementation such as nanoGPT, while noting that its repository now points newcomers toward the newer nanochat project.
The win is not fluent prose. It is reaching the moment when each component stops looking like magic: text becomes integers, integers become vectors, attention moves information across a context window, and gradient descent makes the next-token guesses a little less wrong.
