GPT-2, implemented from the tensor up

The problem

• Calling a model API teaches you nothing about why a transformer behaves the way it does.
• I wanted the layer beneath the frameworks, so importing a prebuilt attention block would have defeated the point.
• The goal: build GPT-2 by hand across the 124M to 1.5B configs, then prove I could adapt it to real downstream tasks.

The core model

• Implemented by hand in PyTorch: token and positional embeddings, causal masked multi-head attention (scaled dot-product), GELU MLPs, residual streams, LayerNorm, and weight tying between the input embedding and output projection.
• Trained with AdamW, cross-entropy, learning-rate scheduling, and gradient clipping; tracked train/validation loss and perplexity.
• Generation controls: top-k sampling and temperature scaling for coherent, controllable output.

Classification fine-tuning

• Selective-freeze transfer: train only the final block, final LayerNorm, and the new head, which keeps transfer cheap and the train-validation gap small on a small labeled set.
• Custom tokenization, truncation, padding, and batching for stable training (AdamW, lr 5e-5).
• Result: 96.67% test accuracy, with train 99.43% and validation 97.32%, a small gap that argues against overfitting.

Instruction tuning

• Full fine-tune of the 355M model (no PEFT) on roughly 1.1k Alpaca-style pairs; train loss fell from 4.197 to about 0.17.
• Judged with Meta-Llama-3-8B-Instruct at about 53.79/100, and I documented that a single automated judge is a thin, biasable instrument.

The decision I'd defend

• Selective-freeze for classification instead of full fine-tuning: on a small labeled set, training the whole network would mostly overfit, so freezing keeps transfer cheap and the gap small.
• The opposite call for instruction tuning: a different objective with more signal per example earns a full fine-tune. Two tasks, two deliberate, defensible choices rather than one blanket approach.

How I knew it worked

• Classification: 96.67% test accuracy with train and validation close (99.43% / 97.32%), a small gap that argues against overfitting.
• Instruction tuning: about 53.79/100 on the LLM judge, useful for a fast, reproducible signal but only as trustworthy as one model's opinion, which I flagged as a limitation.