L3 · The Transformer
GPT and BERT
Compare the two training objectives and what each model is good for.
Both models learn by hiding words and guessing them. They hide different things. GPT hides the future: at every position, guess the next token. BERT hides a few scattered words in a finished sentence, then guesses them from both sides.