Understanding the Abstract of "Attention Is All You Need"

Big Picture: What's This Section Doing?

This abstract is announcing a paradigm shift in how we build neural networks for sequence-to-sequence tasks (like translation). Instead of the architectures that dominated the field in 2016, which relied heavily on recurrence (processing sequences one element at a time) and convolutions (looking at local patterns), the authors propose a radically simpler alternative: build everything using only attention mechanisms.

The abstract makes three key claims:

1. It's simpler: No recurrence, no convolutions—just attention
2. It's better: Achieves state-of-the-art results on machine translation tasks
3. It's faster: More parallelizable and requires less training time

Let me break down what this means and why it matters.

---

Part 1: The Problem Statement

"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder."

What is sequence transduction? It means transforming one sequence into another—the canonical example is machine translation (English → German). Mathematically, we're learning a function:

$$f: X^* \to Y^*$$

where $X^*$ represents sequences of any length from an input vocabulary (like English words) and $Y^*$ represents output sequences (like German words).

What are the old approaches?

• Recurrent Neural Networks (RNNs): Process sequences step-by-step:

  • At time step $t$, they compute: $h_t = \text{RNN}(h_{t-1}, x_t)$
  • The hidden state $h_t$ depends on the previous state $h_{t-1}$ and current input $x_t$
  • Problem: Sequential dependency means you can't parallelize—must process position 1, then 2, then 3, etc.

• Convolutional Neural Networks (CNNs): Use fixed-size local windows

  • Look at neighboring positions simultaneously
  • Problem: To see distant dependencies, you need many layers, increasing computation

• Encoder-Decoder Architecture:

  • Encoder: Reads input sequence and produces a compressed representation
  • Decoder: Uses that representation to generate output sequence one token at a time
  • Input sequence: $x = (x_1, x_2, \ldots, x_n)$ where each $x_i$ is a word/token
  • Compressed context: $c = \text{Encoder}(x)$ (typically a fixed-size vector)
  • Output: Generated token-by-token

The attention mechanism improvement: Previous state-of-the-art models added attention to the encoder-decoder framework:

$$y_t = \text{Decoder}(y_{t-1}, c_t)$$

where the context $c_t$ varies at each decoding step, computed as a learned weighted average over the input:

$$c_t = \sum_{i=1}^{n} \alpha_{t,i} \cdot h_i$$

Here:

• $\alpha_{t,i}$ is an attention weight (between 0 and 1) indicating how much to focus on input position $i$ when generating output position $t$
• $h_i$ is the hidden representation of input position $i$
• These weights are learned during training to figure out which input positions are relevant for each output position

This was an improvement, but it still required the complexity of RNNs or CNNs for the encoder and decoder.

---

Part 2: The Key Innovation

"We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely."

The radical idea: What if we could build the encoder and decoder using only attention, with no recurrence or convolution at all?

Why is this significant?

1. Parallelizability: Since there's no sequential dependency, all positions can be processed simultaneously (in parallel on a GPU)
2. Simplicity: Fewer moving parts means easier to implement and understand
3. Efficiency: No need to wait for $h_{t-1}$ before computing $h_t$

The Transformer architecture (which we'll see later in the paper) uses "multi-head self-attention" to:

• Let each position in the sequence attend to every other position
• Compute these attentions in parallel across all positions
• Stack multiple layers of such attention operations

---

Part 3: Empirical Results

"Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train."

What are BLEU scores? BLEU (Bilingual Evaluation Understudy) is a metric for machine translation quality, ranging from 0 to 100:

$$\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right)$$

Where:

• $p_n$ is the precision of n-grams (matching sequences of $n$ consecutive words)
• $w_n$ are weights (typically equal)
• $BP$ is a brevity penalty (to penalize outputs that are too short)

The concrete results:

Task	Model	BLEU Score	Training Cost
WMT 2014 EN→DE	Transformer	28.4	Lower
WMT 2014 EN→FR	Transformer	41.8	3.5 days on 8 GPUs

Why is this impressive?

• The improvement of >2 BLEU is significant (BLEU improvements are hard-won in the ML community)
• Training time is a fraction of previous methods—meaning faster iteration and less computational cost
• These aren't ensembles (averaging multiple models); it's a single model

---

Part 4: Generalization

"We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data."

What is constituency parsing? It's a different NLP task: identifying the grammatical structure of sentences.

Why mention this? It shows the Transformer isn't just good for machine translation—it's a general architecture that works across different sequence-to-sequence and structured prediction tasks. This suggests the core insight (using attention for everything) is broadly applicable.

---

Key Takeaways

Aspect	Old Approaches	Transformer
Main components	RNN/CNN + Attention	Attention only
Parallelization	Limited (sequential)	Full (all positions at once)
Training speed	Slow	Fast (3.5 days for best EN-FR result)
Performance	State-of-the-art at the time	Better on benchmark tasks
Generality	Task-specific tuning	Generalizes across tasks

The abstract is essentially claiming: "We found a simpler, faster, and better architecture. Here's the proof."

The rest of the paper reveals how the Transformer achieves this through the careful design of attention mechanisms—which we'll explore in subsequent sections.

Introduction to "Attention Is All You Need"

Big Picture

This introduction section sets up the problem that the paper is solving and explains why existing approaches are limited. The authors are arguing that current state-of-the-art methods for sequence tasks (like machine translation) rely on recurrent neural networks, which have a fundamental computational constraint: they must process sequences one element at a time. The Transformer paper proposes to solve this by replacing recurrence entirely with attention mechanisms.

Think of it this way: imagine you're translating a book from English to French. Current methods read the English word-by-word, keeping a "memory" (hidden state) that updates at each step. The problem? You can't parallelize this—you can't read word 5 until you've read words 1-4. The Transformer says: "What if we could look at all words at once and figure out which ones matter for understanding each other?"

---

Part 1: The Current State of the Art (Recurrent Models)

What are RNNs and their variants?

The section begins by noting that Recurrent Neural Networks (RNNs), particularly:

• LSTMs (Long Short-Term Memory networks)
• GRUs (Gated Recurrent Units)

...have been the dominant approach for sequence modeling and sequence transduction problems.

Key definitions:

• Sequence modeling: Predicting the next element in a sequence given previous elements (e.g., predicting the next word in a sentence)
• Sequence transduction: Converting one sequence into another (e.g., translating English to French, where input sequence = English words, output sequence = French words)

How RNNs work: The sequential bottleneck

The paper describes RNNs with this critical observation:

"Recurrent models typically factor computation along the symbol positions of the input and output sequences."

This is saying that RNNs break up the computation into steps corresponding to positions in the sequence. Mathematically, at each time step $t$, an RNN computes:

$$h_t = f(h_{t-1}, x_t)$$

Where:

• $h_t$ is the hidden state at time $t$ (the "memory" of what we've seen so far)
• $h_{t-1}$ is the previous hidden state
• $x_t$ is the input at position $t$
• $f$ is some function (in LSTMs/GRUs, this is a complex gated function)

Why is this a problem?

This sequential dependency means:

• To compute $h_t$, you must have finished computing $h_{t-1}$
• You cannot compute $h_2, h_3, h_4, ...$ in parallel
• On modern hardware (GPUs), parallel processing is much faster than sequential processing
• For long sequences, memory becomes a bottleneck because you must keep all previous states in memory

The paper states this explicitly:

"This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples."

What does this mean?

• Batching: In deep learning, we typically train on multiple examples simultaneously to be efficient. If each example requires sequential processing, we can't batch as many examples together on a GPU (due to memory limits).
• The sequential constraint thus becomes a fundamental bottleneck even on powerful hardware.

---

Part 2: Attempted Solutions (That Don't Fully Work)

The paper acknowledges that recent work has tried to improve RNN efficiency through:

• Factorization tricks [21]: Mathematical tricks to decompose computations into simpler parts
• Conditional computation [32]: Only computing parts of the network that are needed

While these help, the paper emphasizes:

"The fundamental constraint of sequential computation, however, remains."

Translation: "Nice try, but you can't escape the fact that the architecture itself is inherently sequential."

---

Part 3: Attention Mechanisms (The Existing Workaround)

What is attention?

Attention mechanisms have emerged as a way to help models handle long-range dependencies. The key insight is that a model doesn't need to process information sequentially if it can "attend to" (focus on) relevant parts of the input regardless of distance.

Simple intuition: When translating "The bank executive visited the river bank," you need to understand that "bank" means different things in each context. An attention mechanism lets the model look at the entire sentence at once and figure out: "When I'm processing the second 'bank', I should pay special attention to the word 'river' nearby."

The paper notes:

"Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences."

Key phrase: "without regard to their distance"—meaning the model can directly connect words that are far apart in the sequence, not just neighboring positions.

The problem with existing attention approaches

However, here's the critical limitation:

"In all but a few cases, however, such attention mechanisms are used in conjunction with a recurrent network."

What does this mean?

Existing approaches use attention as an add-on to RNNs. A typical architecture looks like:

1. RNN encodes the input sequentially, producing hidden states
2. Attention mechanism helps the decoder focus on relevant encoder hidden states
3. But the RNN itself still processes sequentially

The bottleneck remains—you still have to run the RNN sequentially first.

---

Part 4: The Transformer Solution (What This Paper Proposes)

The authors propose a radical departure:

"In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output."

"Eschewing recurrence" = completely removing RNNs from the architecture

Key insight: If attention can model dependencies between any two positions in a sequence, then you don't need recurrence at all. You can replace sequential processing with parallel attention computations.

Why this works

Instead of computing: $h_t = f(h_{t-1}, x_t)$

The Transformer computes something more like: $\text{Output}_t = \text{Attention}(\text{All inputs}, t)$

Where the attention mechanism considers all input positions simultaneously to compute the output for position $t$, without any sequential dependency.

Computational advantage: All positions can be computed in parallel, not sequentially.

---

Part 5: Empirical Results Preview

The paper concludes the introduction with a bold claim supported by empirical results:

"The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs."

What they're saying:

• More parallelizable: Because there's no sequential dependency, you can process all positions at once
• Faster training: 12 hours on 8 GPUs (vs. weeks for RNN-based models)
• Better performance: Achieves state-of-the-art results despite (or because of) the architectural simplification

---

Summary: The Core Argument

Aspect	RNN/LSTM	Transformer
Processing	Sequential: $h_t$ depends on $h_{t-1}$	Parallel: all positions computed simultaneously
Memory bottleneck	Yes: can't batch many examples	No: can batch efficiently
Long-range dependencies	Difficult (gradient vanishing)	Natural via attention
Training speed	Slow (weeks)	Fast (hours)
Architecture complexity	Complex (gates, cells)	Simple (mostly matrix multiplications)

The fundamental thesis: By replacing sequential recurrence with parallel attention, we can build faster, simpler, and more powerful models.

Section 2 Background: Explaining the Path to the Transformer

Big Picture: Why This Section Matters

This section is setting up the intellectual foundation for why the Transformer is important. The authors are essentially saying: "Previous models tried to solve the sequential computation problem using convolutions. Here's why that approach has limitations, and here's why self-attention is better."

The key tension being resolved:

• Old problem: RNNs must process sequences one step at a time (sequential), which is slow
• Previous attempted solution: Use convolutional networks instead (they can be parallelized)
• The limitation: Convolutions have a hidden cost—distance matters
• The Transformer's solution: Use self-attention, which relates any two positions with constant computational cost

Let's unpack each piece.

---

Part 1: The Convolutional Alternative and Its Distance Problem

The Setup: ConvS2S and ByteNet

The authors reference three models that tried to replace RNNs with convolutional neural networks:

• Extended Neural GPU and ByteNet: Early attempts using convolutions for sequence tasks
• ConvS2S (Convolutional Sequence-to-Sequence): The most successful convolutional approach

All of these could compute representations for all input and output positions in parallel—a huge advantage over RNNs.

The Hidden Cost: Distance Effects

Here's where the problem emerges. When you want to connect two distant positions in a sequence, convolutional networks have a fundamental limitation:

In ConvS2S:

• To relate position $i$ to position $j$ in your sequence, the convolutional layers must "stack up"
• The number of layers needed grows linearly with the distance $|i - j|$
• This means distant dependencies require proportionally more computation

In ByteNet:

• They improved this with dilated convolutions (skip connections)
• Now the number of layers grows logarithmically with distance: $\approx \log_2(|i - j|)$
• Better, but still not constant

Why This Matters Mathematically

When you have a network with $L$ layers, and information must travel through all $L$ layers to connect distant positions:

$$\text{Distance to relate positions} = \text{Number of layers in path} \propto \text{Distance in sequence}$$

For ByteNet specifically:

$$\text{Number of layers} \approx \log_2(\text{distance})$$

The intuition: Each convolutional layer has a fixed receptive field (it can only "see" a local window of the input). To aggregate information from far away, you need many layers. Even with logarithmic scaling, this adds complexity.

How Transformers Are Different

The Transformer reduces this to a constant:

$$\text{Number of operations to relate any two positions} = \mathcal{O}(1)$$

This is the game-changer. No matter how far apart two positions are, the Transformer can relate them with a single attention operation. But—and this is important—there's a trade-off:

"reduced effective resolution due to averaging attention-weighted positions"

What does this mean? When you compute attention, you're essentially taking a weighted average of all positions. This averaging process can lose some fine-grained information. The authors will address this with Multi-Head Attention (Section 3.2), which we'll see later.

---

Part 2: Self-Attention—The Core Innovation

What Is Self-Attention?

Definition: Self-attention (also called intra-attention) is an attention mechanism that relates different positions within the same sequence to compute a representation.

Let's unpack this:

• "Attention mechanism": A way of weighting different inputs based on their relevance
• "Relating positions within the same sequence": For each position $i$ in your sequence, you ask "which other positions $j$ should I pay attention to?"
• "Compute a representation": Use these weighted relationships to build a better encoding of what each position means

An Intuitive Example

Consider the sentence: "The cat sat on the mat because it was comfortable."

For the word "it":

• Standard approaches might struggle because "it" is far from "cat"
• Self-attention can directly ask: "What words in this sentence should I focus on?"
• The attention mechanism can assign high weights to "cat" and "mat," effectively capturing that "it" refers to "the mat being on the mat"

Why Self-Attention Is Powerful

Previous work showed self-attention succeeds in many tasks:

• Reading comprehension: Understanding which parts of a document answer a question
• Abstractive summarization: Identifying which sentences are most important
• Textual entailment: Determining if one statement logically follows from another
• Learning sentence representations: Building general-purpose encodings of text

The key insight: Self-attention works well because it can capture long-range dependencies directly, without the distance penalty that convolutions have.

---

Part 3: Other Attention-Based Models

Memory Networks

The authors mention end-to-end memory networks:

• These use a recurrent attention mechanism (attention applied repeatedly over time steps)
• They showed promise on simple-language QA and language modeling
• But they still rely on recurrence, which has the sequential computation problem

The distinction here is subtle but important:

• Memory networks: Recurrence with attention applied at each time step
• Transformer: Only attention, no recurrence at all

---

Part 4: The Unique Contribution of the Transformer

The Key Claim

"To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution."

Let's break down what makes this unique:

"Transduction model": A model that maps from one sequence to another (like machine translation: English → German)

"Relying entirely on self-attention":

• No RNNs (no sequential bottleneck)
• No convolutions (no distance-dependent computational cost)
• Only attention mechanisms

"Without sequence-aligned RNNs": Unlike memory networks that apply RNNs at each step, the Transformer bypasses RNNs entirely

Why This Matters

$$\begin{align} \text{Computational parallelization:} \quad &\text{RNN} < \text{CNN} < \text{Attention}\\ \text{Distance dependency:} \quad &\text{Attention} < \text{CNN} < \text{RNN} \end{align}$$

The Transformer optimizes both dimensions simultaneously:

• ✅ Can parallelize across all positions (like CNNs)
• ✅ Constant-time distance relations (unlike CNNs)
• ✅ No sequential bottleneck (unlike RNNs)

---

Summary: The Bridge to What's Coming

This section establishes:

1. The problem: RNNs are sequential; CNNs have distance penalties
2. The solution: Self-attention relates positions with constant cost
3. The evidence: Self-attention has worked in other domains
4. The novelty: The Transformer is the first to use only self-attention for full sequence transduction

In the next section (3.2), the authors will show you exactly how self-attention works mathematically, and how Multi-Head Attention helps overcome the "reduced effective resolution" issue mentioned here.

Section 3: Model Architecture - Detailed Explanation

The Big Picture

Before diving into the mathematical details, let's understand what this section is establishing:

The authors are presenting the overall structural blueprint of the Transformer. The key insight is that while the Transformer is revolutionary in how it processes information (through attention rather than recurrence), it maintains a familiar high-level structure: an encoder-decoder architecture. Think of this as saying "we're keeping the tried-and-true organizational pattern, but replacing the internal machinery."

This section does two important things:

1. Defines the notation for input/output sequences
2. Introduces the core architectural pattern that will be elaborated in subsequent sections

Let's work through it carefully.

---

Part 1: The Encoder-Decoder Framework (Background Context)

What is sequence transduction?

Sequence transduction is the task of converting one sequence into another. Examples include:

• Machine translation (English sentence → French sentence)
• Summarization (long text → short summary)
• Speech-to-text (audio waveform → text)

The Standard Structure

The passage describes the traditional encoder-decoder pattern that was dominant before the Transformer:

Input sequence: $(x_1, x_2, \ldots, x_n)$

• These are the initial symbols or tokens you're feeding into the model
• The subscript indicates position: $x_1$ is the first token, $x_n$ is the $n$-th and final token
• $n$ can vary depending on the sentence length

Encoder processing: $\mathbf{z} = (z_1, z_2, \ldots, z_n)$

• The encoder processes the entire input sequence and produces a sequence of continuous representations
• $\mathbf{z}$ (bold indicates this is a sequence/collection) contains $n$ vectors
• Each $z_i$ is typically a vector of floating-point numbers (e.g., 512 or 768 dimensions)
• The key point: $z_i$ is an encoded version of the input that captures its meaning in a learned representation

Why "continuous representations"?

• The input tokens are discrete symbols (words from a vocabulary)
• The encoder converts these into continuous vectors in high-dimensional space
• These vectors capture semantic meaning in a way that neural networks can process

Decoder generation: $\text{Output sequence: } (y_1, y_2, \ldots, y_m)$

• Given the encoded input $\mathbf{z}$, the decoder generates output one token at a time
• $m$ is the output length (often different from input length $n$)
• The model is autoregressive: this is technical jargon meaning "self-regressive" or "feeding its own output back as input"
• Concretely: to generate $y_2$, the model uses $\{y_1, \mathbf{z}\}$ as input; to generate $y_3$, it uses $\{y_1, y_2, \mathbf{z}\}$, and so on

Visual intuition: Think of it like a translator working with notes: they read the entire source text first (encoder), take notes ($\mathbf{z}$), then write the translation word-by-word (decoder), always remembering what they've written so far.

---

Part 2: How the Transformer Implements This Pattern

Now here's the crucial claim:

"The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder"

Breaking down this statement:

"Follows this overall architecture"

• The Transformer still has an encoder and a decoder (maintaining the same organizational structure)
• Input still flows: $x_1, \ldots, x_n \to \mathbf{z} \to y_1, \ldots, y_m$
• The interface between encoder and decoder is still $\mathbf{z}$

"Stacked self-attention"

• Instead of recurrent layers (like LSTMs that process sequentially), the Transformer uses self-attention mechanisms
• Self-attention is a module that lets each position "look at" all other positions in the sequence
• "Stacked" means multiple attention layers are placed on top of each other
• (We'll see the formal definition in Section 3.2, but intuitively: it's a way for the model to figure out which parts of the input are most important for understanding each position)

"Point-wise, fully connected layers"

• Between attention layers, there are standard neural network layers applied independently to each position
• "Point-wise" means: apply the same transformation to $z_1, z_2, \ldots, z_n$ separately (this is sometimes called a feed-forward network or MLP—multi-layer perceptron)
• These are simple: each position gets transformed by the same function

Why is this important?

The key architectural difference from previous models:

• Old approach (RNNs, LSTMs): Process sequentially, position-by-position, where each position depends on previous ones

  • Position 1 → Position 2 → Position 3 → ...
  • This is inherently sequential (can't be parallelized well)

• New approach (Transformer): All positions can be processed in parallel using attention

  • All positions simultaneously attend to all other positions
  • Much better for parallel computation on modern GPUs

---

Part 3: Reference to Figure 1

The passage mentions:

"shown in the left and right halves of Figure 1, respectively"

What Figure 1 shows:

• Left side: The encoder architecture

  • Input embeddings enter
  • Multiple layers of (Self-Attention + Feed-Forward)
  • Output: the encoded representations $\mathbf{z}$

• Right side: The decoder architecture

  • Output embeddings enter (with positional information)
  • Multiple layers of (Masked Self-Attention + Cross-Attention + Feed-Forward)
  • Output: the final predictions

The "stacked" nature is visible in the diagram—you'll see repeated blocks of layers, one on top of another.

---

Key Takeaways

Aspect	Traditional Models	Transformer
Encoder mechanism	RNN/LSTM (sequential)	Stacked Self-Attention
Decoder mechanism	RNN/LSTM (sequential)	Stacked Self-Attention + Cross-Attention
Parallelization	Limited (sequential)	Excellent (parallel-friendly)
Overall structure	Encoder → Decoder	Encoder → Decoder (same pattern!)

The profound insight: you don't need recurrence to model sequences effectively. Pure attention is enough, and it's faster to train.

---

What Comes Next

The remainder of Section 3 will elaborate on:

• What self-attention actually computes mathematically (3.2)
• Multi-head attention (why multiple attention heads help) (3.2.3)
• The specific encoder stack (3.4)
• The specific decoder stack (3.5)
• Positional encoding (how the model knows word order without recurrence) (3.5)

Each of these builds on the architectural blueprint you've now seen.

Section 3.1: Encoder and Decoder Stacks — Deep Dive

The Big Picture

This section describes the fundamental building blocks of the Transformer architecture. Rather than using recurrent connections (like LSTMs) or convolutional layers, the Transformer stacks multiple identical layers, each containing attention mechanisms and feed-forward networks. The key innovations here are:

1. Stack-based architecture: Multiple identical layers repeated 6 times
2. Residual connections: Shortcuts that allow gradients to flow more easily
3. Layer normalization: Stabilizes training
4. Causal masking in decoder: Ensures the model can't "cheat" by looking at future tokens

Let's break each component down.

---

The Encoder Stack

Overall Structure

The encoder consists of N = 6 identical layers stacked on top of each other. Think of this like building a tower where each floor has the same architecture but processes information slightly differently as it flows upward.

Each layer has two sub-layers:

1. Multi-head self-attention mechanism — allows the model to look at all positions in the input sequence simultaneously
2. Position-wise fully connected feed-forward network — applies the same neural network to each position independently

The Residual Connection + Layer Normalization

Here's the critical formula for the encoder:

$$\text{Output} = \text{LayerNorm}(x + \text{Sublayer}(x))$$

Let me break down what's happening:

What each component means:

• $x$ = the input to this sub-layer (dimension: $d_{\text{model}} = 512$)
• $\text{Sublayer}(x)$ = the output of either the attention mechanism or feed-forward network
• $x + \text{Sublayer}(x)$ = the residual connection (also called a "skip connection")
• $\text{LayerNorm}(\cdot)$ = layer normalization (discussed below)

Why the residual connection matters:

In earlier deep neural networks, when you stack many layers, gradients computed during backpropagation would become vanishingly small in early layers. This is called the vanishing gradient problem. The residual connection solves this by creating a direct shortcut: instead of forcing all information to flow through $\text{Sublayer}(x)$, we add it to the original input $x$.

Think of it this way:

• If $\text{Sublayer}(x)$ learns small adjustments to $x$, the gradient can flow directly back through the $x$ term
• This makes it easier to train very deep networks (with 6 stacked layers)

Layer Normalization:

Layer normalization normalizes the activations across the feature dimension for each sample independently. Given an output vector $\mathbf{y} = [y_1, y_2, ..., y_d]$ of dimension $d$:

$$\text{LayerNorm}(\mathbf{y}) = \gamma \text{odot} \frac{\mathbf{y} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

Where:

• $\mu = \frac{1}{d}\sum_{i=1}^{d} y_i$ = mean across the $d$ features
• $\sigma^2 = \frac{1}{d}\sum_{i=1}^{d}(y_i - \mu)^2$ = variance across features
• $\gamma$ and $\beta$ = learnable scale and shift parameters (dimension $d$)
• $\text{odot}$ = element-wise multiplication
• $\epsilon$ = small constant (like $10^{-5}$) to prevent division by zero

Intuition: Layer normalization centers and scales each sample's activations to have mean 0 and variance 1, which stabilizes training and allows higher learning rates.

Maintaining Consistent Dimensions

The paper specifies that all sub-layers and embedding layers produce outputs of dimension $d_{\text{model}} = 512$. This is crucial because:

• The residual connection $x + \text{Sublayer}(x)$ requires both terms to have the same dimension
• It allows layers to be stacked uniformly
• It sets the internal representation size (we'll see this affects attention head dimensions in Section 3.2)

---

The Decoder Stack

Additional Complexity

The decoder is more complex than the encoder because it must:

1. Generate output sequentially (one token at a time, auto-regressively)
2. Attend to the encoder's output
3. Not look at future positions it hasn't generated yet

Three Sub-layers Per Decoder Layer

Each of the N = 6 decoder layers contains:

1. Masked multi-head self-attention — self-attention over previously generated outputs
2. Multi-head attention over encoder output — attends to the encoder stack (called "encoder-decoder attention")
3. Position-wise feed-forward network — same as in encoder

All three are wrapped with the same residual connection + layer normalization pattern:

$$\text{Output} = \text{LayerNorm}(x + \text{Sublayer}(x))$$

Causal Masking (The Critical Difference)

Here's where the decoder differs fundamentally. The paper states:

"We modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions."

Why is this necessary?

During training, we have the ground truth output sequence available. But during inference, we generate one token at a time. To train the model realistically for inference, we must simulate the inference condition during training: position $i$ can only attend to positions $1, 2, ..., i-1$.

How masking works mathematically:

Recall from Section 3 that attention is computed as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The masking is applied before the softmax. For each position $i$, we set attention scores to $-\infty$ for all positions $j > i$:

$$\text{Attention}(Q, K, V)_i = \text{softmax}\left(\frac{QK^T_{i,:}}{\sqrt{d_k}} \text{ with } QK^T_{i,j} = -\infty \text{ for } j > i\right)V$$

When you apply softmax to $[-\infty, a, b]$, it becomes $[0, \text{softmax}(a,b), \text{softmax}(a,b)]$, completely zeroing out the future positions.

The offset mechanism:

"The output embeddings are offset by one position"

This means:

• At training time, when predicting position $i$, the model has access to ground-truth outputs at positions $1, 2, ..., i-1$ (not $i$ itself)
• At inference time, when generating position $i$, the model has access to previously generated tokens at positions $1, 2, ..., i-1$
• These conditions match, ensuring the training procedure actually teaches the model how to generate sequences

---

Putting It Together: Information Flow

Here's how a single sample flows through the Transformer:

1. Input → embedding layer → produces sequence of vectors, each of dimension 512

2. Encoder layer 1 → attention sees all input positions → feed-forward → output (512 dimensions)

3. Encoder layers 2-6 → same process, each seeing outputs from previous layer, progressively refining representations

4. Decoder layer 1 →

   • Masked self-attention on shifted output embeddings (can only see past)
   • Cross-attention to encoder output (attends to input)
   • Feed-forward network

5. Decoder layers 2-6 → same process, refining output predictions

6. Final layer → project to vocabulary size → softmax → predict next token

---

Summary Table

Aspect	Encoder	Decoder
Number of layers	N = 6	N = 6
Sub-layers per layer	2 (attention + feed-forward)	3 (masked attention + cross-attention + feed-forward)
Self-attention type	Full attention (all-to-all)	Masked attention (causal)
Cross-attention	None	Yes (to encoder)
Output dimension	512	512
Purpose	Process input once	Generate output sequentially

The elegance of this design is that both encoder and decoder use the same fundamental building blocks (attention + feed-forward + residual + norm), but with strategic modifications (masking in decoder) to handle the directional nature of generation.

Scaled Dot-Product Attention: A Detailed Explanation

Big Picture: What's This Section About?

This section introduces the core computational mechanism that powers the entire Transformer architecture. Think of attention as a way for the model to selectively focus on different parts of the input when processing information.

Previously (in Section 2), the paper mentioned that RNNs process sequences sequentially (slow for parallel computation) while CNNs struggle to relate distant positions efficiently. The authors are now proposing a better alternative: Scaled Dot-Product Attention, which relates any two positions with constant computational cost.

The key innovation here is deceptively simple: use dot products to measure relevance, scale them to prevent numerical problems, and use softmax to create attention weights.

---

Part 1: Conceptual Foundation – What is Attention?

Let me start with the intuitive definition before the math:

Attention is a mechanism for computing a weighted sum of values, where the weights depend on how "compatible" or "relevant" each value is to a query.

Imagine you're skimming a document to answer a specific question (the query). You don't read every word equally—you focus on (assign higher weight to) words relevant to your question. Attention works similarly:

• Query (Q): "What am I looking for?" (what we want to find)
• Keys (K): "What does each position contain?" (metadata about potential matches)
• Values (V): "What information is at each position?" (the actual content we care about)

The output is a blend of all values, weighted by how well each key matches the query.

---

Part 2: The Mathematical Formulation

The Core Equation

The paper gives us the formula for Scaled Dot-Product Attention:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Let me break down every component and dimension:

Variables and Their Dimensions

Symbol	What it represents	Dimensions	Explanation
$Q$	Query matrix	$(n_{\text{queries}}, d_k)$	Contains all queries packed as rows; $d_k$ is the query dimension
$K$	Key matrix	$(n_{\text{keys}}, d_k)$	Keys for each position; same dimension as queries for dot product compatibility
$V$	Value matrix	$(n_{\text{keys}}, d_v)$	Actual values we'll average; can have different dimension $d_v$ than keys
$d_k$	Key dimension	scalar	Dimension of key vectors (e.g., 64 in the paper's implementation)
$d_v$	Value dimension	scalar	Dimension of value vectors (e.g., 64)
Output	Attention output	$(n_{\text{queries}}, d_v)$	Weighted combination of values, same dimension as values

Key insight about matrix shapes: The multiplication $QK^T$ gives us an $(n_{\text{queries}}, n_{\text{keys}})$ matrix—essentially a compatibility score between every query and every key.

Step-by-Step Breakdown

Let me walk through the computation:

Step 1: Compute Query-Key Compatibility ($QK^T$)

$$QK^T \in \mathbb{R}^{n_{\text{queries}} \times n_{\text{keys}}}$$

This matrix multiplication computes dot products between each query and each key. The entry at position $(i,j)$ is:

$$(QK^T)_{ij} = q_i \cdot k_j = \sum_{l=1}^{d_k} Q_{il}K_{jl}$$

where $q_i$ is the $i$-th query vector and $k_j$ is the $j$-th key vector.

Why dot product? The dot product measures geometric similarity: parallel vectors (pointing the same direction) have large positive dot products, while orthogonal vectors have dot products near zero. This naturally captures relevance.

Step 2: Scale by $\frac{1}{\sqrt{d_k}}$

$$\frac{QK^T}{\sqrt{d_k}} = \frac{1}{\sqrt{d_k}} \cdot QK^T$$

Why scale? This is crucial. The paper explains:

• When $d_k$ is large, dot products $q_i \cdot k_j$ tend to have large magnitudes (because you're summing $d_k$ products)
• Large inputs to softmax compress to extreme values (nearly 0 or 1), with very small gradients
• Dividing by $\sqrt{d_k}$ keeps the dot products in a reasonable range where softmax has meaningful gradients for training

Analogy: Imagine softmax as a temperature control—dividing by $\sqrt{d_k}$ cools down hot (large) dot products.

Step 3: Apply Softmax

$$\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)$$

The softmax function converts scores to probabilities. For a vector $x = [x_1, x_2, \ldots, x_m]$:

$$\text{softmax}(x)_j = \frac{e^{x_j}}{\sum_{l=1}^{m} e^{x_l}}$$

Applied to each row of our matrix, this gives us attention weights:

• Each row sums to 1 (they're probability distributions)
• Higher scores → higher weights
• All weights are positive and normalized

So each query gets a probability distribution over all keys, representing "how much should I attend to each position?"

Step 4: Multiply by Values ($\times V$)

$$\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \in \mathbb{R}^{n_{\text{queries}} \times d_v}$$

Matrix dimensions:

• Attention weights: $(n_{\text{queries}}, n_{\text{keys}})$
• Values: $(n_{\text{keys}}, d_v)$
• Result: $(n_{\text{queries}}, d_v)$ ✓

Each output query vector is a weighted average of all value vectors, with weights from the attention distribution.

---

Part 3: Comparison with Other Attention Mechanisms

The paper mentions two main approaches:

Additive Attention

Uses a learned neural network to compute compatibility:

$$\text{score}(q_i, k_j) = w^T \tanh(W_1 q_i + W_2 k_j)$$

where $W_1, W_2, w$ are learned weight matrices.

Pros: Theoretically expressive (can learn complex compatibility functions) Cons: Slower (requires forward passes through a small neural network for each pair)

Dot-Product Attention (Our Approach)

$$\text{score}(q_i, k_j) = q_i \cdot k_j$$

Pros:

• Much faster (pure matrix multiplication)
• Space-efficient (no extra parameters needed)
• Hardware-optimized (GPUs have specialized matrix multiplication circuits)

Cons:

• Without scaling, fails for large $d_k$ (as the paper explains)

---

Part 4: The Key Innovation – Why Scaling Matters

This deserves special attention because it's critical:

The Problem with Unscaled Dot Products

When $d_k$ is large, say $d_k = 512$:

• Each dot product $q \cdot k$ sums 512 products
• Variance of dot products grows with $d_k$
• Many dot products become very large (or very negative)

How Softmax Breaks Down on Large Values

The softmax derivative is:

$$\frac{\partial}{\partial x_i} \text{softmax}(x)_j = \text{softmax}(x)_j(\delta_{ij} - \text{softmax}(x)_i)$$

When softmax inputs are large, the output is nearly 0 or 1, making the gradient term $\text{softmax}(x)_i(1 - \text{softmax}(x)_i)$ vanish.

Result: Gradients disappear, training stalls.

The Solution: Scale by $\frac{1}{\sqrt{d_k}}$

By dividing by $\sqrt{d_k}$, we keep dot products at comfortable magnitudes. If we assume $Q$ and $K$ are normalized:

• Unscaled dot products have variance $\approx d_k$
• Scaled dot products have variance $\approx 1$
• Softmax stays in the comfortable middle range with healthy gradients

---

Part 5: Computational Advantage

The paper emphasizes practical benefits:

Matrix Implementation: We don't compute attention separately for each query-key pair. Instead, we pack everything into matrices $Q, K, V$ and compute:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

This is three matrix multiplications, which can be:

• Parallelized efficiently across thousands of GPU cores
• Computed using highly optimized libraries (BLAS, cuBLAS)
• Much faster than additive attention's sequential neural network evaluations

---

Summary Table: The Four Steps of Scaled Dot-Product Attention

Step	Operation	Purpose	Dimensions
1	Compute $QK^T$	Measure query-key compatibility	$(n_q, n_k)$
2	Divide by $\sqrt{d_k}$	Keep values in reasonable range	$(n_q, n_k)$
3	Apply softmax	Convert to probability weights	$(n_q, n_k)$
4	Multiply by $V$	Compute weighted sum of values	$(n_q, d_v)$

---

Why This Matters for the Transformer

Remember from Section 3.1 that the Transformer has:

• All positions can attend to all other positions in constant operations (just matrix multiplications)
• No sequential dependency like RNNs (embarrassingly parallel)
• Better gradient flow than CNNs with their hierarchical position dependencies

Scaled Dot-Product Attention makes this possible and practical. It's elegant, efficient, and—as the experimental results show—remarkably effective.

Multi-Head Attention: A Deep Dive

The Big Picture

You've just learned about Scaled Dot-Product Attention in the previous section—a single attention mechanism that lets the model look at different positions and compute a weighted combination of values based on relevance (query-key matches).

Multi-Head Attention takes this idea further by asking: Why look at the data in just one way?

The core insight is that different aspects of your data might be more relevant when viewed through different "lenses" or representations. A single attention head might conflate multiple types of relationships. By using multiple heads in parallel, each with its own learned perspective on the data, the model can simultaneously attend to different types of information—grammatical structure, semantic meaning, word relationships, etc.—all at once.

This is analogous to how humans use different cognitive processes simultaneously: you might parse syntax, recognize semantic meaning, and track discourse structure all in parallel when reading a sentence.

---

The Mechanism: Step-by-Step

Step 1: Linear Projections (The Multiple Lenses)

Instead of using queries, keys, and values directly at their full dimension $d_{\text{model}} = 512$, the model linearly projects them $h$ times (where $h = 8$ in this work) into lower-dimensional spaces.

For each head $i$ (where $i = 1, 2, ..., h$):

$$\text{head}_i = \text{Attention}(QW^Q_i, KW^K_i, VW^V_i)$$

Let me break down what each component represents:

• $Q$, $K$, $V$: These are the original query, key, and value matrices (defined in section 3.2.1). Each has shape related to sequence length and dimensionality.

  • Specifically: $Q \in \mathbb{R}^{n \times d_{\text{model}}}$, $K \in \mathbb{R}^{n \times d_{\text{model}}}$, $V \in \mathbb{R}^{n \times d_{\text{model}}}$
  • Here $n$ is the sequence length and $d_{\text{model}} = 512$

• $W^Q_i$, $W^K_i$, $W^V_i$: These are learned projection matrices with dimensions:

  • $W^Q_i \in \mathbb{R}^{d_{\text{model}} \times d_k}$ (projects from 512-dim to $d_k$-dim)
  • $W^K_i \in \mathbb{R}^{d_{\text{model}} \times d_k}$ (projects from 512-dim to $d_k$-dim)
  • $W^V_i \in \mathbb{R}^{d_{\text{model}} \times d_v}$ (projects from 512-dim to $d_v$-dim)

  Different subscripts $i$ mean these are different matrices for each head—this is how each head learns its own "lens."

What happens geometrically: Matrix multiplication $QW^Q_i$ takes each of the $n$ rows of $Q$ (representing positions in your sequence) and transforms them into a new vector space of dimension $d_k$. This is a linear transformation that the model learns during training.

Step 2: Apply Attention in Parallel

For each head, you apply the Scaled Dot-Product Attention formula (Equation 1 from the previous section):

$$\text{head}_i = \text{softmax}\left(\frac{(QW^Q_i)(KW^K_i)^T}{\sqrt{d_k}}\right)(VW^V_i)$$

This produces an output of shape $\mathbb{R}^{n \times d_v}$—a $d_v$-dimensional representation for each of the $n$ positions.

Key point: All $h = 8$ heads do this in parallel on their own projected versions of the data. They don't interfere with each other.

Step 3: Concatenation

After all heads finish their attention computations, you have 8 outputs, each of shape $n \times d_v$:

$$\text{head}_1, \text{head}_2, ..., \text{head}_h$$

You concatenate these along the dimension axis:

$$\text{Concat}(\text{head}_1, ..., \text{head}_h) \in \mathbb{R}^{n \times (h \cdot d_v)}$$

This gives you a matrix where each position has information from all 8 different attention perspectives stacked together.

Step 4: Final Linear Projection

Finally, you apply one more learned linear projection to combine all this information back to the original dimension:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$

Where:

• $W^O \in \mathbb{R}^{(hd_v) \times d_{\text{model}}}$ is a projection matrix taking from the concatenated dimension $(hd_v = 8 \times 64 = 512)$ back to $d_{\text{model}} = 512$

Shape tracking:

$$\text{Concat} (\in \mathbb{R}^{n \times 512}) \times W^O (\in \mathbb{R}^{512 \times 512}) = \text{Output} (\in \mathbb{R}^{n \times 512})$$

---

The Complete Formula

Putting it all together:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$

where

$$\text{head}_i = \text{Attention}(QW^Q_i, KW^K_i, VW^V_i)$$

---

Why This Works: The Key Insight

The paper makes a crucial observation:

"Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."

What does "averaging inhibits this" mean?

Intuition: Imagine you're trying to understand a word in context. You might need to:

1. Look at grammatical relationships (subject-verb agreement, etc.)
2. Look at semantic relationships (what topics are being discussed?)
3. Look at discourse structure (what was mentioned before?)

With a single attention head, the softmax must create one probability distribution that balances all these considerations simultaneously. The weighted average blurs together these different types of relationships.

With multiple heads, each head can specialize:

• Head 1 learns to focus on subject-verb relationships
• Head 2 learns to focus on topical coherence
• Head 3 learns to focus on discourse dependencies
• ... and so on

The model can learn these specializations because each head has its own set of parameters ($W^Q_i, W^K_i, W^V_i$) that are optimized independently during training.

---

Computational Efficiency

The paper notes an important practical point:

"Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality."

The math:

• Single-head attention would compute: $\text{Attention}(Q, K, V)$ with $d_k = d_v = 512$

  • Computation involves matrix multiplies of size $\sim n \times 512$ and $512 \times 512$
  • Complexity: $O(n \cdot 512^2)$

• Multi-head (8 heads):

  • Each head: $\text{Attention}$ with $d_k = d_v = 64$
  • 8 heads in parallel: $8 \times O(n \cdot 64^2) = O(n \cdot 512^2)$

Same computational cost, but with the benefits of multiple representation subspaces. This is why multi-head attention is such a powerful design choice.

---

Specific Hyperparameters in This Paper

The authors made specific choices:

Parameter	Value	Why?
$h$ (number of heads)	8	Balances model capacity with computational efficiency
$d_k = d_v$	$d_{\text{model}} / h = 512/8 = 64$	Keeps per-head dimension manageable while using full model dimension across all heads
$d_{\text{model}}$	512	Standard choice for this model size

These aren't magical constants—they're design choices that work well empirically. Researchers could experiment with other values (e.g., 16 heads with $d_k = 32$).

---

Putting It in Context

In the Transformer architecture:

• The Encoder and Decoder (Section 3.1) both use multi-head attention
• This is how they can simultaneously capture multiple types of relationships while maintaining computational efficiency
• The feed-forward network that follows each attention layer (also mentioned in 3.1) then processes these rich multi-perspective representations further

This design—combining insights from multiple representation subspaces—is a major reason the Transformer became so successful and influential.

Section 3.2.3: Applications of Attention in our Model

The Big Picture

This section is the practical payoff of everything discussed so far. We've learned how attention works (Scaled Dot-Product Attention from 3.2.1) and how to make it more powerful (Multi-Head Attention from 3.2.2). Now we need to understand where these attention mechanisms are actually used in the Transformer architecture.

The key insight: attention can be used in fundamentally different ways depending on what you feed into it. By changing where the queries, keys, and values come from, we can implement three distinct types of information flow, each serving a different purpose in translating a sentence from one language to another.

---

The Three Applications of Attention

1. Encoder-Decoder Attention (Cross-Attention)

Purpose: Allow the decoder to look back at the entire input sequence while generating output.

How it works:

• Queries ($Q$): Come from the previous decoder layer
• Keys ($K$): Come from the encoder's output
• Values ($V$): Come from the encoder's output

Intuition: Imagine you're translating English to German. The decoder is deciding what to write next in German, and it needs to look at the entire English sentence to figure out what word to generate. This attention mechanism does exactly that—it lets each position in the decoder "look at" all positions in the input.

Mathematical interpretation: When we compute $\text{Attention}(Q, K, V)$ using equation (1):

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

We're computing a weighted sum of the encoder's output values ($V$), where the weights depend on how well the decoder's current query ($Q$) matches each encoder position's key ($K$). The softmax ensures these weights sum to 1 (a probability distribution).

Why this matters: This is the bridge between encoder and decoder—without it, the decoder would have no way to access the input sequence. The paper notes this "mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models," meaning this is how previous neural machine translation systems also worked, but now it's implemented via the elegant attention mechanism.

---

2. Encoder Self-Attention

Purpose: Allow each position in the encoder to integrate information from all other positions at the previous layer.

How it works:

• Queries ($Q$): Come from the previous encoder layer
• Keys ($K$): Come from the previous encoder layer
• Values ($V$): Come from the previous encoder layer

The key word: "Self" means queries, keys, and values all come from the same source.

Intuition: Consider the word "bank" in English—it could mean a financial institution or the side of a river. To understand which meaning is intended, you need to look at surrounding words. Self-attention allows each word position to attend to (look at) all other word positions in the same layer. This way, the representation of "bank" can be refined based on context from neighboring words like "river" or "account."

How it propagates information:

• In layer 1 of the encoder: each position attends to input embeddings
• In layer 2 of the encoder: each position attends to layer 1 outputs (which already contain some context from layer 1's self-attention)
• In layer 3+: each position attends to deeper representations with increasingly rich contextual information

This is how context propagates through the encoder—each layer builds richer, more contextually-aware representations.

---

3. Decoder Self-Attention (with Masking)

Purpose: Allow each position in the decoder to look at previously generated tokens, but not future tokens (preserving autoregressive generation).

How it works:

• Queries ($Q$): Come from the previous decoder layer
• Keys ($K$): Come from the previous decoder layer
• Values ($V$): Come from the previous decoder layer
• Special twist: Masking prevents attending to future positions

The critical constraint - Autoregressive property: In machine translation, you generate words sequentially: first word, then second word, then third, etc. When generating the second word, you should only have access to the first word—you can't "cheat" by looking at words you haven't generated yet.

How masking works mathematically:

The paper states: "We implement this inside of scaled dot-product attention by masking out (setting to $-\infty$) all values in the input of the softmax which correspond to illegal connections."

Let's unpack this. Recall the attention computation:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The matrix $\frac{QK^T}{\sqrt{d_k}}$ has shape $(T, T)$ where $T$ is the sequence length. The entry at position $(i, j)$ represents how much position $i$ should attend to position $j$.

Before softmax, for each query position $i$, we set all entries corresponding to "illegal" future positions (where $j > i$) to $-\infty$:

$$\text{Scores}_{i,j} = \begin{cases} \frac{QK^T}{\sqrt{d_k}} & \text{if } j \leq i \text{ (allowed: current or past)} \\ -\infty & \text{if } j > i \text{ (blocked: future)} \end{cases}$$

When we apply softmax to each row:

$$\text{softmax}(\text{Scores})_{i,j} = \frac{e^{\text{Scores}_{i,j}}}{\sum_{j'} e^{\text{Scores}_{i,j'}}}$$

Any entry with $-\infty$ becomes $e^{-\infty} = 0$, so the softmax weights for future positions become exactly zero. This completely eliminates the "look at future" contribution.

Concrete example: Imagine you're generating a 4-word translation and you're at position 2 (generating the second word). The attention mask looks like this:

$$\begin{pmatrix} \text{allowed} & -\infty & -\infty & -\infty \\ \text{allowed} & \text{allowed} & -\infty & -\infty \\ \text{allowed} & \text{allowed} & \text{allowed} & -\infty \\ \text{allowed} & \text{allowed} & \text{allowed} & \text{allowed} \end{pmatrix}$$

Position 2 (second row) can only attend to positions 1 and 2—positions 3 and 4 are blocked.

Why this matters: This constraint is essential for training. During training, the ground truth words are available, but the model must still learn to generate word-by-word without "looking ahead." During inference (actual translation), you generate one word at a time anyway, so this constraint matches the actual deployment scenario.

---

Putting It All Together

These three attention mechanisms work in concert:

1. Encoder processes input: Self-attention lets each word understand its context (layers 1-6 of encoder)
2. Decoder looks back at encoder: Encoder-decoder attention lets the decoder see the full input sequence
3. Decoder generates output: Self-attention (with masking) lets the decoder leverage previously generated words while maintaining autoregressive generation

This is why the Transformer is elegant—by simply controlling where queries, keys, and values come from and applying masking when needed, we get three fundamentally different but complementary behaviors from the same underlying attention mechanism.

Section 3.3: Position-wise Feed-Forward Networks

Big Picture: Why Does This Matter?

You might be wondering: "We just spent all this time on attention mechanisms—why do we need feed-forward networks too?"

Here's the key insight: attention mechanisms are great at figuring out which positions should talk to each other, but they don't inherently add expressive power to transform the information itself. The feed-forward networks in this section are the Transformer's solution to this problem. They act as "information processors" that can apply learned transformations to each piece of information independently, adding nonlinearity and computational power beyond what attention alone provides.

In traditional deep learning, you alternate between mixing information (like attention does) and transforming it (like these feed-forward networks do). The Transformer does exactly this pattern: attention → feed-forward → attention → feed-forward, and so on.

---

Understanding the Architecture

The Basic Structure

Each layer in both the encoder and decoder contains:

1. An attention sub-layer (which we learned about in section 3.2)
2. A position-wise feed-forward network (what we're covering now)

The word "position-wise" is crucial. It means: the same feed-forward network is applied independently to each position in the sequence. There's no interaction between positions within this feed-forward layer—each token gets processed by identical transformation rules.

This is different from the attention layer, where every position can "look at" every other position. Here, position 1's transformation doesn't depend on position 2, position 3, etc. The transformation is purely local to each position.

The Mathematical Formula

$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2 \quad \text{(Equation 2)}$

Let me break down each part:

Variables and their meanings:

• $x$: the input vector at a single position, with dimensionality $d_{\text{model}} = 512$ (a column vector in $\mathbb{R}^{512}$)
• $W_1$: a weight matrix of shape $512 \times 2048$ (projects input to a higher-dimensional "hidden" space)
• $b_1$: a bias vector of shape $2048$ (added after the first linear transformation)
• $W_2$: a weight matrix of shape $2048 \times 512$ (projects back down to the original dimension)
• $b_2$: a bias vector of shape $512$ (added after the second linear transformation)
• $d_{\text{model}} = 512$: the model's standard embedding dimension
• $d_{ff} = 2048$: the inner (hidden) layer dimension

Walking through the computation step by step:

1. First linear transformation: $xW_1 + b_1$

   • Takes the input $x \in \mathbb{R}^{512}$ and transforms it to dimension $2048$
   • This is a simple matrix multiplication: the $(512 \times 2048)$ matrix $W_1$ projects $x$ from a 512-dimensional space to a 2048-dimensional space
   • Then we add $b_1$ (a learnable bias term) element-wise

2. Activation function: $\max(0, \cdot)$ (ReLU)

   • This is the Rectified Linear Unit function: it returns $z$ if $z > 0$, and returns $0$ if $z \leq 0$
   • Mathematically: $\text{ReLU}(z) = \max(0, z)$
   • This introduces nonlinearity, which is critical—without it, two linear transformations would just be equivalent to a single linear transformation, and we'd lose expressive power
   • The ReLU also has nice computational properties: it's simple to compute and it helps with training

3. Second linear transformation: $(\text{ReLU output}) \cdot W_2 + b_2$

   • Takes the 2048-dimensional hidden representation and projects it back down to the original dimension of 512
   • This matches the output shape of the attention layers, so it can feed into the next layer

The overall data flow: $512 \to 2048 \to 512$ (expand, apply nonlinearity, contract)

---

Why This Structure?

Expansion and Contraction

The architecture expands from 512 → 2048 (4x larger) and then contracts back to 512. Why?

• The expansion creates a "hidden layer" with much higher dimensionality. This increased dimension allows the network to learn richer, more complex transformations. Think of it like having more "capacity" to express ideas.

• The contraction brings it back to 512 so the output can be fed into the next layer (which expects inputs of dimension 512, matching $d_{\text{model}}$)

• The factor of 4 (since $2048 = 4 \times 512$) is a design choice in this paper. It's one of the hyperparameters they selected. Different values could work, but this ratio seems to balance expressiveness with computational cost.

Why ReLU in the Middle?

The ReLU activation is essential for introducing nonlinearity. Here's why:

Consider two linear transformations stacked together: $(\text{Linear}_1) \to (\text{Linear}_2)$

Mathematically, composing two linear operations still gives you a linear operation! Specifically: $f(x) = (xW_1 + b_1)W_2 + b_2 = x(W_1W_2) + (b_1W_2 + b_2)$

This is still just a linear transformation (with a combined weight matrix $W_1W_2$). You haven't gained any extra expressiveness.

But with ReLU in between: $f(x) = \max(0, xW_1 + b_1) \cdot W_2 + b_2$

The $\max(0, \cdot)$ operation is nonlinear, so you can no longer simplify this to a single linear transformation. The network can now learn much more complex functions—this is what gives it expressive power.

---

Alternative Description: Convolutions with Kernel Size 1

The paper mentions: "Another way of describing this is as two convolutions with kernel size 1."

This might seem odd if you're thinking of traditional convolutions, so let me clarify:

A 1×1 convolution is a convolution operation where the filter has spatial size 1×1. In the context of processing sequences with a channel dimension:

• The first "convolution" with kernel size 1 is equivalent to applying $xW_1 + b_1$ to each position independently
• The ReLU activation is applied
• The second "convolution" with kernel size 1 is equivalent to applying $(\cdot)W_2 + b_2$ to each position independently

This framing is just another way to think about the same operation—it emphasizes that each position is processed independently (there's no interaction across positions within this layer). In the literature, 1×1 convolutions are sometimes called "pointwise convolutions" for exactly this reason.

---

Computational Efficiency

Here's an important practical note: the same parameters $W_1$, $b_1$, $W_2$, $b_2$ are used for every position.

This means:

• If the sequence has 10 tokens, we apply the same transformation 10 times (once per token)
• If the sequence has 100 tokens, we apply it 100 times
• The learned parameters don't increase with sequence length—only the computation does

This is actually implementable very efficiently because you can: $\text{FFN}(\text{Sequence}) = \max(0, \text{Sequence} \cdot W_1 + b_1) \cdot W_2 + b_2$

where $\text{Sequence}$ is a matrix with shape $(L \times 512)$ for a sequence of length $L$. Then:

• $\text{Sequence} \cdot W_1$ is an $(L \times 2048)$ matrix (each row is one position, transformed)
• Apply ReLU element-wise
• $(\cdot) \cdot W_2$ is an $(L \times 512)$ matrix (back to original dimension)

You compute this all at once using optimized matrix operations rather than looping over positions—that's why it's efficient.

---

Putting It All Together

The feed-forward network serves as a per-position transformation mechanism that:

1. Adds nonlinearity through ReLU, enabling complex learned transformations
2. Increases expressiveness by expanding to a higher dimension (512 → 2048) where more complex patterns can be represented
3. Maintains efficiency by using the same parameters across all positions
4. Complements attention by transforming information after attention has determined which information should interact

In the full Transformer architecture, you see a repeating pattern: $\text{Self-Attention} \to \text{Feed-Forward} \to \text{Self-Attention} \to \text{Feed-Forward} \to \cdots$

This interleaving of "communication" (attention) and "transformation" (feed-forward) is what gives the Transformer its power.

Section 3.4: Embeddings and Softmax - Detailed Explanation

The Big Picture

This section describes how the Transformer converts discrete input tokens (words) into continuous vectors that the neural network can process, and how it converts the network's output back into predictions about which token should come next. While this might sound like a simple bookkeeping detail, the authors make two clever design choices here:

1. Sharing weights between the input embeddings, output embeddings, and the final prediction layer
2. Scaling the embeddings by a mysterious factor of $\sqrt{d_{\text{model}}}$

These choices have subtle but important effects on how the model learns. Let's break down what's happening.

---

Part 1: Token Embeddings - From Discrete to Continuous

The Problem We're Solving

The Transformer works entirely with real-valued vectors (as you've seen in the attention and feed-forward layers). But language is inherently discrete—we have a finite vocabulary of tokens (words, subwords, or characters). We need a way to convert each token into a real-valued vector.

The Solution: Learned Embeddings

The paper says: "we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{\text{model}}$."

What does this mean mathematically?

Let's say our vocabulary has $V$ possible tokens. An embedding layer is defined by an embedding matrix:

$$E_{\text{input}} \in \mathbb{R}^{V \times d_{\text{model}}}$$

Here:

• $V$ is the size of the vocabulary (the total number of unique tokens)
• $d_{\text{model}} = 512$ is the dimension we want (as mentioned in Section 3.3)
• Each row of this matrix is a $d_{\text{model}}$-dimensional vector representing one token

How it works in practice:

When we encounter token $t$ in our input sequence (where token $t$ is represented as an integer from $0$ to $V-1$), we look up row $t$ of the embedding matrix:

$$\text{embedding}(t) = E_{\text{input}}[t, :] \in \mathbb{R}^{d_{\text{model}}}$$

This is essentially a lookup table operation. The values in $E_{\text{input}}$ are learned parameters—they're updated during training via backpropagation to make the model better at its task.

Why learn embeddings?

Initially, all token embeddings are random. But as the model trains on thousands or millions of examples, it learns to position tokens in this high-dimensional space such that:

• Related tokens end up near each other
• The geometry of the space captures linguistic meaning

For example, the embedding for "king" minus the embedding for "man" plus the embedding for "woman" will be close to the embedding for "queen"—a famous property of word embeddings.

Scaling by $\sqrt{d_{\text{model}}}$

$$$$

Here's where it gets interesting. The paper states: "In the embedding layers, we multiply those weights by $\sqrt{d_{\text{model}}}$."

$$**What does this mean?**$$

After looking up the embedding vector, we multiply every component by $\sqrt{d_{\text{model}}} = \sqrt{512} \approx 22.6$:

\text{scaled\_embedding}(t) = \sqrt{d_{\text{model}}} \cdot E_{\text{input}}[t, :] **Why do this?** This is somewhat of a technical trick related to how the model combines embeddings with positional encodings (which are mentioned in the context but not the main focus here). Here's the intuition: 1. **Embeddings and positional encodings have different scales**: The embedding vectors have one typical scale, while positional encodings (vectors that indicate where in the sequence a token appears) have a different scale. 2. **Matching scales matters**: When you add two vectors together, you want them to contribute relatively equally. If one is much larger than the other, it dominates. By scaling up the embeddings by $\sqrt{d_{\text{model}}}$, the authors ensure that the embedding and positional encoding contributions are roughly balanced in magnitude. 3. **Empirical motivation**: The authors mention this works well in practice—it's an engineering choice that improves training stability and model performance, though the exact mechanism is somewhat empirical. **A rough analogy:** Imagine you're mixing two ingredients in a recipe—one measured in grams, one in ounces. If you want them to contribute roughly equally, you need to scale them appropriately before mixing. That's what this factor does. --- ## Part 2: The Output Layer - From Continuous Back to Discrete ### Converting to Token Probabilities The paper says we use *"the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities."* **What does this mean mathematically?**

At the end of the decoder stack, after passing through all the attention and feed-forward layers, we have a vector $h \in \mathbb{R}^{d_{\text{model}}}$ representing the model's "understanding" of what should come next.

$$To predict the next token, we: 1. **Apply a linear transformation:** \text{logits} = h \cdot W_{\text{softmax}} + b$$

where:

• $W_{\text{softmax}} \in \mathbb{R}^{d_{\text{model}} \times V}$ is a learned weight matrix
• $b \in \mathbb{R}^{V}$ is a learned bias vector
• $\text{logits} \in \mathbb{R}^{V}$ is a vector with one score for each possible token

2. Apply the softmax function to convert scores to probabilities:

P(\text{token } i) = \text{softmax}(\text{logits})_i = \frac{e^{\text{logits}_i}}{\sum_{j=1}^{V} e^{\text{logits}_j}} The softmax function ensures: - All probabilities are between 0 and 1 - All probabilities sum to 1 - More likely tokens have higher probabilities --- ## Part 3: Weight Sharing - A Clever Design Choice ### The Key Insight Here's the truly clever part: *"we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation"* **What does this mean?**

Instead of having three separate weight matrices:

• $E_{\text{input}} \in \mathbb{R}^{V \times d_{\text{model}}}$ for input embeddings
• $E_{\text{output}} \in \mathbb{R}^{V \times d_{\text{model}}}$ for output embeddings
• $W_{\text{softmax}} \in \mathbb{R}^{d_{\text{model}} \times V}$ for the softmax layer

$$...the authors use only **one** matrix and reuse it: W_{\text{shared}} \in \mathbb{R}^{V \times d_{\text{model}}}$$

Then:

• Input embeddings: $\text{embedding}(t) = \sqrt{d_{\text{model}}} \cdot W_{\text{shared}}[t, :]$
• Output projection: $\text{logits} = h \cdot W_{\text{shared}}^T$ (note the transpose)

So we have: $W_{\text{softmax}} = W_{\text{shared}}^T$

Why share weights?

1. Fewer parameters: This reduces the total number of learnable parameters, which can improve generalization and reduce computational cost. In a model with a vocabulary of 50,000 tokens, this saves $50,000 \times 512 = 25.6$ million parameters.

2. Geometric interpretation: It creates a symmetry in the model. The same embedding space is used for both interpreting input tokens and predicting output tokens. This encourages the model to learn a shared representation space where:

   • Token similarity in embeddings reflects token similarity in the output predictions
   • The dot product $h \cdot W_{\text{shared}}^T$ naturally measures how well the current hidden state aligns with each token's embedding

3. Initialization: Shared weights provide a better initialization. The embeddings start with meaningful random values rather than zeros.

---

Summary Table

Component	Dimension	Purpose
Input embeddings	$V \times d_{\text{model}}$	Convert input tokens to vectors
Output embeddings	Same as input	Used in softmax layer (transpose)
Scaling factor	$\sqrt{512} \approx 22.6$	Balance embedding with positional encoding
Softmax output	$V$	Probability distribution over tokens

---

Connection to the Bigger Picture

Remember from earlier sections:

• Section 3.2.3: The decoder produces an output vector $h$ of dimension $d_{\text{model}}$ at each position
• Section 3.3: The feed-forward network maintains this $d_{\text{model}}$ dimension throughout

This section completes the pipeline: the embeddings get the input into the model, and the softmax gets the model's output back into a format humans can understand (predicted tokens).

Section 3.5: Positional Encoding — Detailed Explanation

The Big Picture: Why This Section Matters

The Transformer is fundamentally different from previous sequence models. Traditional approaches like RNNs (Recurrent Neural Networks) process sequences one token at a time, naturally capturing order information through their sequential processing. Transformers, by contrast, process the entire sequence in parallel using attention mechanisms.

The core problem: If you feed all tokens into the model simultaneously, how does the model know which token comes first, second, third, etc.? Without some way to encode position information, the sentence "The cat ate the mouse" would be treated identically to "The mouse ate the cat" — just a jumbled collection of tokens with no inherent order.

The solution: Positional encodings inject information about token positions directly into the input representations. This allows the attention mechanism to leverage both the semantic meaning of tokens AND their sequential positions.

---

The Core Concept

What Are Positional Encodings?

Positional encodings are vectors of dimension $d_{\text{model}}$ (which equals 512 in this model) that get added to the token embeddings. Think of it like this:

$\text{Input to Transformer} = \text{Token Embedding} + \text{Positional Encoding}$

Since both have the same dimension, element-wise addition is straightforward. The positional encoding is designed to:

1. Capture the absolute position of each token in the sequence
2. Allow the model to learn relative distances between tokens
3. Scale to sequences longer than those seen during training

Why Not Learn Positional Encodings?

The authors mention they experimented with learned positional embeddings — treating positions like learnable parameters, similar to token embeddings. These performed nearly identically (Table 3, row E). However, they chose the sinusoidal version for one key reason: extrapolation to longer sequences. A learned embedding might not generalize well to sequences longer than training examples, while sinusoidal encodings can theoretically extend infinitely.

---

The Mathematical Details

The Sinusoidal Positional Encoding Formula

The positional encoding at position $pos$ and dimension $i$ is defined by:

$PE_{(pos,2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

$PE_{(pos,2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

Let me break down every component:

Symbol	Meaning	Notes
$PE_{(pos, d)}$	The positional encoding at position $pos$ in dimension $d$	This is a scalar value that gets added to the embedding
$pos$	The position in the sequence	$pos \in \{0, 1, 2, ..., T-1\}$ where $T$ is sequence length
$i$	A dimension index	$i \in \{0, 1, 2, ..., d_{\text{model}}/2 - 1\}$
$d_{\text{model}}$	The embedding dimension	Set to 512 in this paper
$10000$	A frequency scaling constant	Determines the wavelength range

Understanding the Structure

Notice the clever construction:

• Even dimensions ($2i$): Use the sine function
• Odd dimensions ($2i+1$): Use the cosine function

For a sequence position $pos$, you compute the full $d_{\text{model}}$-dimensional positional encoding by evaluating these formulas for $i = 0, 1, 2, ..., d_{\text{model}}/2 - 1$.

Example: If $d_{\text{model}} = 512$ and $pos = 5$:

• Dimension 0: $\sin(5/10000^{0/512})$
• Dimension 1: $\cos(5/10000^{0/512})$
• Dimension 2: $\sin(5/10000^{2/512})$
• Dimension 3: $\cos(5/10000^{2/512})$
• ... and so on for 512 dimensions total

---

The Intuition: Wavelengths and Frequencies

What Are These Sinusoids Doing?

Each dimension oscillates at a different frequency. Think of tuning knobs on different radio stations:

The denominator term $10000^{2i/d_{\text{model}}}$ controls the wavelength (or period) of each sinusoid:

• For small $i$ (early dimensions): The exponent $2i/d_{\text{model}}$ is small, so $10000^{2i/d_{\text{model}}} \approx 1$, giving short wavelengths (high frequencies, oscillates rapidly)
• For large $i$ (later dimensions): The exponent $2i/d_{\text{model}}$ approaches 2, so $10000^{2i/d_{\text{model}}} \approx 10000$, giving long wavelengths (low frequencies, oscillates slowly)

The wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$.

Visualization Intuition

Imagine stacking sine and cosine waves of increasing wavelengths:

• Short wavelengths (bottom rows): Wiggle a lot — encode fine-grained position details
• Long wavelengths (top rows): Barely change — encode coarse-grained position structure

Together, these create a unique "fingerprint" for each position in the sequence.

---

The Key Mathematical Property: Relative Position Representation

Why Sinusoids for Relative Positions?

Here's the critical claim from the paper: "for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$."

This is saying: the positional encoding at position $pos + k$ can be computed as a linear combination of the encoding at position $pos$.

Let me prove why this works for a single dimension:

Using the angle addition formulas from trigonometry:

$\sin(A + B) = \sin A \cos B + \cos A \sin B$ $\cos(A + B) = \cos A \cos B - \sin A \sin B$

With $A = pos/f$ and $B = k/f$ (where $f = 10000^{2i/d_{\text{model}}}$ is the frequency):

$\sin\left(\frac{pos+k}{f}\right) = \sin\left(\frac{pos}{f}\right)\cos\left(\frac{k}{f}\right) + \cos\left(\frac{pos}{f}\right)\sin\left(\frac{k}{f}\right)$

$\cos\left(\frac{pos+k}{f}\right) = \cos\left(\frac{pos}{f}\right)\cos\left(\frac{k}{f}\right) - \sin\left(\frac{pos}{f}\right)\sin\left(\frac{k}{f}\right)$

In matrix form for the pair of sine-cosine dimensions:

$$\begin{pmatrix} PE_{(pos+k, 2i)} \\ PE_{(pos+k, 2i+1)} \end{pmatrix} =$$ $$\begin{pmatrix} \cos(k/f) & \sin(k/f) \\ -\sin(k/f) & \cos(k/f) \end{pmatrix}$$ $$\begin{pmatrix} PE_{(pos, 2i)} \\ PE_{(pos, 2i+1)} \end{pmatrix}$$

This is a rotation matrix! The encoding at position $pos+k$ is a linear transformation (specifically, a rotation) of the encoding at position $pos$.

Why This Matters

This property means:

1. The model can learn relative distances — the offset $k$ between any two positions is encoded as a consistent transformation
2. Generalization to longer sequences — since the relationship is mathematical rather than learned, it works for positions beyond training data
3. Efficient attention — the attention mechanism can learn to use relative position information through this linear structure

---

Design Choices Explained

Why $10000$?

The constant 10000 is somewhat arbitrary but chosen to be large enough that:

• The longest wavelength (for the highest $i$) is $10000 \cdot 2\pi$
• This is typically much longer than the longest training sequence
• Thus, different positions throughout realistic sequences get distinct encodings

You could use other constants; 10000 just happens to work well empirically.

Why Sine and Cosine Together?

Using both sine and cosine (in alternating dimensions) is elegant because:

• They're orthogonal: $\int_0^{2\pi} \sin(x)\cos(x)dx = 0$
• Together, they form a complete basis for representing rotations
• This makes the relative position property (shown above) work smoothly

---

Practical Summary

Aspect	Details
What it does	Adds position information to token embeddings so the Transformer knows token order
How it works	Uses sine/cosine functions of different frequencies, one unique encoding per position
Key property	Relative positions are encoded as linear transformations (rotations)
Why not learned?	Sinusoids generalize to longer sequences better than learned embeddings
Integration	Simply added to embeddings: $\text{embedding} + PE$

The elegance of this approach lies in its simplicity and mathematical structure — without any learnable parameters or sequence length limitations, it provides the order information essential for making the parallel, attention-based Transformer architecture work.