← Back to Blog

Model notes · Chapter 1

Position encoding: how Transformers keep track of order

From sine waves to RoPE: where position enters a Transformer, and what “longer context” really requires.

Muxuan Liu · Prepared from my model-learning notes; examples and equations checked for this edition.

Why position needs a separate signal

“The dog chased the cat” and “The cat chased the dog” use the same words but mean different things. A model needs to distinguish their order. Plain full self-attention, without position information or an order-dependent mask, treats a permutation of its inputs consistently with the same permutation of its outputs. It has no dedicated notion of first, second, or third.

The engineering question is therefore: where should order enter the computation? One choice is to attach a position vector to each input. Another is to modify how pairs of positions interact. Causal masks also introduce an order constraint, so “Transformers are completely unordered” is too broad a description.

Absolute position: give each slot an address

Absolute encoding describes a token’s own position. In the original Transformer, a sinusoidal vector is added to its input embedding before the stack processes it. It is not separately added to every Q, K, and V by definition. A learned position table is another option: each row is a trainable address for one position.

A learned table has a configured range. Extending it requires a design choice, such as interpolation or additional training. The existence of separate rows does not prevent the network from learning relationships between positions; it simply does not explicitly encode a relative distance.

Sinusoidal encoding: a clock with several speeds

Imagine several clock hands rotating at different speeds. Together, their readings identify a position. Sinusoidal encoding uses sine and cosine pairs with progressively lower frequencies. In the formula below, p is the token position, d is the embedding width, and i indexes a pair of dimensions.

The frequency factor is 10000^(−2i/d): the minus sign matters. A positive exponent would implement a different frequency schedule. The formula can be evaluated for unseen positions, but that alone does not guarantee good predictions beyond the training context. Computing a position and using it reliably are separate questions.

Relative position: describe the gap

Sometimes “three places to the left” is more useful than “position 17.” Relative encodings make that relationship explicit. Shaw-style attention introduces learned relative vectors into key/value computations. A distance clipped to the inclusive range [−k, k] has 2k+1 possible buckets.

Relative-position methods are a family, not one fixed operation on the attention matrix. Some add score biases, some modify key/value contributions, and RoPE rotates queries and keys. It is useful to inspect the actual attention equation rather than classify methods only by name.

RoPE: turn position into an angle

RoPE rotates pairs of query and key coordinates by position-dependent angles. When the rotated query and key are compared, their dot product depends on the relative rotation. The identity R(m)ᵀR(n) = R(n−m) captures the idea: two absolute rotations produce a relative offset.

RoPE itself does not change dense attention’s quadratic dependence on sequence length. Nor is unlimited context automatic. Frequency choices, scaling methods, training length, and the attention implementation all matter. RoPE supplies a position mechanism, not a general speed-up or a guarantee of extrapolation.

Read the code through its shapes

The example accepts [batch, sequence, d_model]. It creates a [1, max_len, d_model] buffer, which broadcasts across the batch. register_buffer keeps the table in the module state and moves it with the model without making it a trainable parameter. It does not turn off autograd for every operation using the buffer.

The implementation handles both odd and even embedding widths. max_len is a storage limit of this example, not a limit of the sine/cosine formula. Dropout is disabled for deterministic checks and follows the module’s train/eval mode in normal use.

import math
import torch
from torch import nn

class SinusoidalPositionEncoding(nn.Module):
    """Input: [batch, sequence, d_model]."""
    def __init__(self, d_model, max_len=5000, dropout=0.1):
        super().__init__()
        if d_model < 1 or max_len < 1:
            raise ValueError('d_model and max_len must be positive')
        pos = torch.arange(max_len, dtype=torch.float32)[:, None]
        freq = torch.exp(
            torch.arange(0, d_model, 2, dtype=torch.float32)
            * (-math.log(10000.0) / d_model)
        )
        pe = torch.zeros(max_len, d_model)
        pe[:, 0::2] = torch.sin(pos * freq)
        pe[:, 1::2] = torch.cos(pos * freq[:d_model // 2])
        self.register_buffer('pe', pe[None, :, :])
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        if x.ndim != 3 or x.shape[-1] != self.pe.shape[-1]:
            raise ValueError('Expected [batch, sequence, d_model]')
        if x.shape[1] > self.pe.shape[1]:
            raise ValueError('Sequence exceeds configured max_len')
        return self.dropout(x + self.pe[:, :x.shape[1]].to(dtype=x.dtype))