D3 · Backpropagation
Every Node Knows One Small Derivative
Write the local gradient for add, multiply and a nonlinearity.
A multiply node inside a language model has no idea it is inside a language model. It knows two numbers came in. Asked for its derivative, it hands back the other input.