Key notation

• x: input tensor [B, L, d_model]
• W_Q, W_K, W_V, W_O: attention matrices
• W_in, W_out: MLP matrices
• d_model: model dimension
• n_heads: number of attention heads

Token embedding

$E \in \mathbb{R}^{V \times d_{model}}$

$x_0 = E[\text{tokens}]$

Positional encoding

If learned positions: $x_0 = x_0 + P[\text{positions}]$

Attention

$Q = x W_Q, \; K = x W_K, \; V = x W_V$

$\text{attn}(x) = \text{softmax}(QK^T / \sqrt{d_{head}}) V$

MLP

$\text{mlp}(x) = W_{out} \sigma(W_{in} x)$

Residual block

$x_{i+1} = x_i + \text{attn}(x_i)$

$x_{i+2} = x_{i+1} + \text{mlp}(x_{i+1})$

No snippets available yet.