D1 · The Perceptron
Why Activations Exist
Explain what the step, sigmoid, tanh and ReLU each do to a neuron's output, and why the step function cannot be trained.
The step function is honest. The neuron fires or it does not, exactly like the cell it copies. It is flat, then it jumps, then it is flat again. Standing anywhere on it, the ground tells you nothing about which way to walk.