D6 · Sequences & Memory
One Vector for a Whole Sentence
Explain the encoder-decoder bottleneck and measure how it hurts long inputs.
In 2014 a Google team built a translator that read an English sentence, squeezed it into 8,000 numbers, and wrote French from those numbers alone. The same 8,000 for a three word greeting and a forty word legal clause. It worked. Why smaller versions of it broke is the reason attention exists.