L4 · Training a Language Model
Cleaning the Text
Say what a web filter throws away and why duplicates are worse than junk.
One 61-word English sentence appears in the C4 dataset more than 60,000 times. Nobody put it there on purpose. It is boilerplate, copied across the web by machines.