L3 · The Transformer
Mixture of Experts
Explain routing and the gap between total and active parameters.
The last lesson ended on waste: most feed-forward units stay silent, yet all of them get computed. So keep eight separate feed-forward layers, and let a tiny network pick two per token. Only those two run.