D3 · BackpropagationWhy It Costs So LittleCompare the cost of backprop to computing each gradient separately.Every gradient of a model with 175 billion parameters costs about as much as three forward passes. Not 175 billion passes. Three.