D6 · Sequences & Memory
Gates That Choose What to Keep
Walk one time step through an LSTM's three gates and cell state.
Every problem in lesson 5 came from one habit. The plain cell rebuilds its state each step, so old information is multiplied again and again. The fix is almost rude. Keep a second vector, and mostly just add to it. An add passes the gradient back with a factor of exactly 1, however many steps you stack.