Skip to content

Stage 3 · Expert · A2

Training Your Own Small Model

A model of 449 numbers, trained in under a second, can be all your one task needs.

12 lessons · 124 minDemanding

Start chapter

About this chapter

Frontier models are general, and your problem is not. That gap is your advantage. This chapter takes one narrow question, starts with honest label work, and trains a model small enough to read in a text file. You will overfit on purpose to see the shape of it, fix it three ways, borrow a model trained for something else, let a big model teach a small one, and squeeze the weights down to a few bits. It ends by comparing your model with a giant the way an honest person would, cost included.

What you will be able to do

  1. 1

    A Task Worth a Small Model

    Tell whether a task is narrow enough for a model you can train yourself.

    9 min
  2. 2

    A Few Hundred Labels

    Write a label guide, measure disagreement, and know the ceiling it puts on your score.

    10 min
  3. 3

    The Number You Have to Beat

    Ship three untrained answers first and write down the one score a model has to clear.

    9 min
  4. 4

    Training a Tiny Classifier

    Run a full training loop on your own labels and read both scores it produces.

    12 min
  5. 5

    Overfit on Purpose

    Make a model memorize a tiny pile on purpose and read the three things it tells you.

    10 min
  6. 6

    Three Ways to Fix It

    Pick between more labels, weight decay and stopping early by reading what each one buys.

    11 min
  7. 7

    A Language Model You Can Read

    Train a character-level language model on this device and tell what it learned from what it memorized.

    12 min
  8. 8

    Fine-Tune or Start From Scratch

    Decide between borrowing a trained model and starting fresh, from the number of labels you have.

    10 min
  9. 9

    LoRA, by Picture

    Adapt a frozen model by training two skinny matrices instead of a whole layer.

    10 min
  10. 10

    Letting the Giant Teach

    Train a tiny model on a bigger model's beliefs instead of on human labels.

    10 min
  11. 11

    Shrinking It to Ship

    Round a model's weights onto a coarse ladder and find the point where shrinking starts to cost score.

    10 min
  12. 12

    Judging It Against the Giant

    Compare two models on the same held-back set honestly, and put the cost of an answer beside the score.

    11 min

Before you start

Keep going

All chapters