D4 · Deep Learning
Depth Is Cheaper Than Width
Show that stacking folds multiplies the pieces while spreading them only adds.
Take 21 ReLU units. Lay them side by side in one layer and they cut a line into 22 pieces. Stack them as seven layers of three and they cut the same line into 128. Same units, same cost, about six times the detail.