Understand neurons, layers, activation functions, backpropagation, and why deep networks are powerful.
Published May 2, 2025
A neural network is a function approximator inspired by biological neurons. It learns to map inputs to outputs by adjusting millions of parameters (weights) through a process called backpropagation.
Inputs: x₁, x₂, x₃
Weights: w₁, w₂, w₃
Bias: b
Output: f(w₁x₁ + w₂x₂ + w₃x₃ + b)
Where f is an activation function
| Function | Formula | Use Case |
|---|---|---|
| ReLU | max(0, x) | Hidden layers (most common) |
| Sigmoid | 1/(1+e⁻ˣ) | Binary classification output |
| Softmax | eˣⁱ/Σeˣʲ | Multi-class output |
| Tanh | (eˣ-e⁻ˣ)/(eˣ+e⁻ˣ) | RNNs, centered at 0 |
| GELU | x·Φ(x) | Transformers (BERT, GPT) |
Without activation functions, stacking layers = one linear transformation. Activations introduce non-linearity allowing networks to learn complex patterns.
import numpy as np
def forward(X, W1, b1, W2, b2):
# Layer 1
Z1 = X @ W1 + b1 # linear combination
A1 = np.maximum(0, Z1) # ReLU activation
# Layer 2 (output)
Z2 = A1 @ W2 + b2
A2 = sigmoid(Z2) # sigmoid for binary classification
return A2
Binary Cross-Entropy (binary classification):
L = -[y log(ŷ) + (1-y) log(1-ŷ)]
Categorical Cross-Entropy (multi-class):
L = -Σ yᵢ log(ŷᵢ)
MSE (regression):
L = Σ(yᵢ - ŷᵢ)²/n
Backprop computes gradients of the loss with respect to each weight using the chain rule:
dL/dW₁ = dL/dA₂ × dA₂/dZ₂ × dZ₂/dA₁ × dA₁/dZ₁ × dZ₁/dW₁
This flows backward through the network, computing how much each weight contributed to the error.
Simple Gradient Descent:
w = w - η × dL/dw (η = learning rate)
Adam (most common in practice):
Adapts learning rate per parameter
Combines momentum + RMSProp
Works well without tuning
SGD + Momentum:
Better for fine-tuning pre-trained models
| Architecture | Use Case |
|---|---|
| Feedforward (MLP) | Tabular data, regression |
| CNN (Conv layers) | Images, spatial patterns |
| RNN/LSTM | Sequential data, time series |
| Transformer | NLP, images (ViT), multi-modal |
| GAN | Generative tasks (images, audio) |
# Dropout: randomly zero out neurons during training
# Prevents co-adaptation, forces robust features
model.add(Dropout(0.5)) # 50% of neurons dropped each step
# L2 Regularization (Weight Decay)
# Adds λ Σ wᵢ² to loss → penalizes large weights
optimizer = Adam(learning_rate=0.001, weight_decay=1e-4)
# Batch Normalization
# Normalizes layer inputs → faster training, more stable
model.add(BatchNormalization())
A neural network with a single hidden layer of sufficient width can approximate any continuous function on a compact domain. This is why neural networks are so general-purpose.