Why Non-Linearity Matters

Without non-linear activations, stacked layers collapse into a single linear function.

What happens if a deep network uses only linear activation functions in every layer?