← Back to Blog

Model notes · Chapter 3

Residual connections: learn changes, keep a direct path

Understand x + F(x), the gradient shortcut, and why adding more layers still requires care.

Muxuan Liu · Prepared from my model-learning notes; examples and equations checked for this edition.

A layer can learn an update instead of a replacement

Suppose a representation already contains useful information. Rather than asking a block to rebuild everything, a residual connection lets it add an update: y = x + F(x). Think of revising a draft while keeping the original text available.

A residual connection is the addition path. A residual block contains that path and a learned branch, such as convolutions or attention plus a feed-forward component. ResNet is a particular family built from residual blocks; the same connection idea is used more widely.

Why the shortcut helps optimization

If an identity mapping would be useful, the learned branch can approach zero and preserve the input. This gives a deep network an accessible way to retain a representation while learning changes around it. Residual learning addresses the difficulty of optimizing deep networks, including degradation when more layers are added—not only vanishing gradients.

The word “residual” does not mean that every branch output must be small, or that the network always wants to copy the input. The branch learns whatever update the task and architecture support.

Write the recurrence before expanding it

Use xₗ₊₁ = xₗ + Fₗ(xₗ). After L blocks, xL = x0 + Σ Fₗ(xₗ), where each branch is evaluated on its own intermediate state. This is easier to read correctly than a long nested expansion.

For one block, the input gradient is (I + JF(x))ᵀ times the output gradient, using column-vector gradients. I is the identity matrix and JF is the branch Jacobian. The direct term gives an additional route, but it does not put a lower bound of 1 on the total gradient: the branch can amplify or partly cancel it.

A small example catches a common mistake

Take a scalar branch F(x)=a·x repeated for L blocks. Each block multiplies its input by 1+a, so the total derivative is (1+a)^L. It is not 1+a+a²+…+a^L for this architecture.

With a=−0.5 and L=3, the derivative is 0.125; with a=1 it is 8. A residual path therefore does not guarantee that gradients never shrink or grow. Initialization, normalization, branch scale, and the learned transformations still matter.

Why activation variance can grow

For scalar random variables, Var(x+F) = Var(x)+Var(F)+2Cov(x,F). The covariance term matters because a trained branch depends on its input. A simple independent-noise thought experiment is useful, but it is not a complete model of a trained Transformer.

If L independent zero-mean updates each have variance 1, their sum has variance L and standard deviation √L. Scaling each update by 1/√L keeps that update sum’s variance at 1. This illustrates a motivation for residual scaling; it is not a universal prescription for every architecture.

Where this leaves a practical implementation

Addition requires compatible shapes. If a block changes width, a projection shortcut may be needed. In Transformers, the placement of normalization changes the actual recurrence, so distinguish the simple y=x+F(x) identity from the full block being implemented.

The code below checks the scalar recurrence directly with autograd. My practical takeaway is to follow both forward values and backward gradients: a shortcut improves the route through a deep model, but successful training still depends on how all the pieces fit together.

import torch

# Minimal residual-gradient example: three blocks, F(x) = a*x.
def residual_gradient(a, depth=3):
    x = torch.tensor(1.0, requires_grad=True)
    y = x
    for _ in range(depth):
        y = y + a * y
    y.backward()
    return x.grad.item()  # (1+a)**depth

for a in [-0.5, 0.1, 1.0]:
    print(a, residual_gradient(a), (1 + a) ** 3)