D3 · Backpropagation
The Forward Pass
Push values through a graph and store what the backward pass will need.
When training runs out of GPU memory, the weights are often not the problem. The leftovers of the forward pass are. Every value on every edge is still sitting there, waiting.