L4 · Training a Language Model
Small Numbers, Big Spikes
Say why training uses two number formats at once, and what a team does when the loss spikes.
Sixteen-bit numbers take half the memory of 32-bit ones, and modern chips multiply them far faster. But the classic 16-bit float cannot hold very small numbers. A gradient below its floor does not shrink. It becomes exactly zero.