Session 07⚡

Attention & Transformers

Context vectors, query-key-value attention, masking, positional information, multi-head attention, Transformer blocks, and BERT.

Learning Objectives

✓Explain how attention removes the fixed-vector encoder-decoder bottleneck
✓Calculate alignment scores, attention weights, and a context vector
✓Trace query, key, and value tensor shapes through scaled dot-product attention
✓Explain causal masking, positional encoding, and multi-head attention
✓Contrast Transformer encoders and decoders and describe BERT pretraining

From One Context Vector to Dynamic Attention

A basic encoder-decoder RNN compresses an entire source sequence into one fixed hidden vector. Long inputs make this a severe information bottleneck. Attention lets each decoder step build a fresh context vector from all encoder annotations. The query is the current decoder state, candidate source features are scored for relevance, softmax normalizes the scores, and the context is a weighted sum. In image captioning, the same process attends to different spatial CNN features for different generated words.

Alignment, Weights & Context

An alignment function can be a dot product, a scaled dot product, a bilinear form, or a small additive network. It produces one scalar per source element. Softmax turns those scalars into nonnegative weights that sum to one, so the context vector is a differentiable weighted average. Attention maps are useful for inspecting behavior, but a bright region should not automatically be treated as a complete causal explanation.

e_{t,j}=\operatorname{align}(s_{t-1},h_j) \qquad \alpha_{t,j}=\frac{e^{e_{t,j}}}{\sum_k e^{e_{t,k}}} \qquad c_t=\sum_j\alpha_{t,j}h_j

Score, normalize, combine: these three operations define the core attention pattern.

Self-Attention as Query, Key & Value

Self-attention derives queries, keys, and values from the same input sequence. A query asks what information one position needs; keys describe what each position offers for matching; values contain the information to combine. The matrix $QK^{\top}$ contains every query-key similarity. Dividing by $\sqrt{d_k}$ keeps logits from growing with dimension, softmax normalizes each query row, and multiplication by $V$ returns one context vector per query.

\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V

For

Q\in\mathbb{R}^{N\times d_k}

and

K\in\mathbb{R}^{M\times d_k}

, the attention matrix is

N\times M

Scaled dot-product attentionPython

import math
import torch

def attention(q, k, v, mask=None):
    scores = q @ k.transpose(-2, -1) / math.sqrt(q.size(-1))
    if mask is not None:
        scores = scores.masked_fill(~mask, float('-inf'))
    weights = torch.softmax(scores, dim=-1)
    return weights @ v, weights

Order & Causality: Position and Masking

Self-attention by itself is permutation-equivariant: reordering the inputs reorders the outputs but does not tell the layer which position came first. Position embeddings or sinusoidal encodings inject order. A decoder also needs a causal mask so position $t$ cannot read future targets. The efficient implementation computes all positions in parallel, sets forbidden future logits to negative infinity before softmax, and therefore gives them zero attention weight. Encoder self-attention is normally unmasked because the full source is available.

Multi-Head Attention

One attention head creates one similarity geometry. Multi-head attention projects the model state into several smaller query, key, and value spaces, runs attention independently, concatenates the results, and applies an output projection. Different heads can specialize in different relations or distance scales. The total model dimension is commonly divided across the heads, so adding heads does not automatically multiply the representation width. Head count must divide the model dimension in standard implementations.

\operatorname{MHA}(Q,K,V)=\operatorname{Concat}(head_1,\ldots,head_h)W^O

Each

head_i

is attention over separately learned projections

QW_i^Q

KW_i^K

, and

VW_i^V

Transformer Encoder & Decoder Blocks

A Transformer block surrounds attention and a position-wise feed-forward network with residual connections and normalization. The encoder uses bidirectional self-attention to produce contextual source representations. The decoder uses masked self-attention over generated targets, cross-attention to encoder outputs in encoder-decoder models, and a feed-forward network. Decoder-only language models omit the encoder and use causal self-attention; encoder-only models such as BERT are designed for understanding tasks with full bidirectional context.

BERT: Contextual Pretraining

BERT is a bidirectional Transformer encoder pretrained on unlabeled text, then fine-tuned for a target task. Its input is the sum of token, segment, and position embeddings. Masked Language Modeling selects 15% of tokens: 80% become [MASK], 10% become a random token, and 10% remain unchanged, while the model predicts the originals. The original BERT also used Next Sentence Prediction to classify whether one sentence followed another. Context now changes each token vector, solving the single-static-vector limitation of Word2Vec and GloVe.

←PreviousNLP & Word Embeddings →NextQuestion Answering & Prompt Engineering