# Attention is all you need
> Recurrent models typically factor computation along the symbol positions of the input and output sequences.
**RNNs process a sequence one position/token at a time**, rather than processing all positions independently at once. For example, take:
> **"I love pizza"**
The input has 3 symbol positions:
```plaintext
I → love → pizza
1 2 3
```
A recurrent model computes something like:
```plaintext
h₁ = f(I, h₀)
h₂ = f(love, h₁)
h₃ = f(pizza, h₂)
```
So computation is **factored along the sequence positions.**
> This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples.
this is **why RNNs are hard to train efficiently**, especially for long sequences.
RNNs have to process each sequence position one after another. This prevents GPUs from fully parallelizing a single sequence. The problem gets worse for long sequences, because long sequences also consume more memory, forcing us to use smaller batches.
> Longer sequence → more memory per example → fewer examples fit in a batch → smaller batch size.
```plaintext
RNN:
x₁ → x₂ → x₃ → x₄ → x₅
sequential 🚶
Transformer:
x₁ x₂ x₃ x₄ x₅
↘ ↓ ↙ ↓ ↘ ↓
attention
parallel ⚡
```
💡
Why make the GPU wait for the previous token when we can process positions together ?
**sequence modelling** : learning from data where the order of elements matters.
a sequence is simply :
```plaintext
x₁ → x₂ → x₃ → x₄ → ...
```
* Text: `I → love → pizza`
* Speech: audio frames over time
* Stock prices: price₁ → price₂ → price₃
* Time series: temperature₁ → temperature₂ → ...
* DNA: `A → C → G → T`
The model learns relationships between elements in that sequence.
**transduction models** : taking one sequence and transforming it into another sequence.
Think:
```plaintext
Input sequence
↓
Model
↓
Output sequence
```
for example, **machine translation**:
```plaintext
English:
"I love pizza"
↓
French:
"J'aime la pizza"
```
both sides are sequences, but they don't necessarily have the same length.
About transformer :
> trained for as little as 12 hours on eight P100 GPUs.
**Tesla P100** was a serious **data-center/HPC GPU**, not a consumer gaming GPU. It launched in **2016** as part of NVIDIA's Pascal generation.
| **Spec** | **Tesla P100** |
| --- | --- |
| CUDA cores | **3,584** |
| GPU memory | **16 GB HBM2** |
| Memory bandwidth | **732 GB/s** |
| FP32 | **9.3 TFLOPS** PCIe / 10.6 TFLOPS NVLink |
| Power | **250 W** PCIe / 300 W NVLink |
| Architecture | **Pascal** |
| Intended use | AI + scientific/HPC workloads |
**FP32 (32-bit floating-point numbers) performance :** how quickly the GPU can perform 32-bit floating-point calculations
**TFLOPS :** trillion floating-point operations per second.
**PCIe :** Peripheral Component Interconnect Express, high-speed connection used to connect components inside a computer.
**NVLink :** NVLink is NVIDIA's high-speed GPU interconnect.
![](https://cdn.hashnode.com/uploads/covers/625a673852fb7bd53380e365/0ef8e778-b0c0-4206-b1c7-e259c0e9ad07.png align="center")
**Convolutional Neural Networks ( CNNs ) :**
a specialized type of neural network designed to understand spatial patterns
we can think of patterns like these :
| **Pattern** | **What it means** | **Commonly handled by** |
| --- | --- | --- |
| **Spatial** | Where things are and their local arrangement | CNNs |
| **Temporal** | How something changes over time | RNNs, LSTMs, Transformers |
| **Sequential** | Relationships based on order | RNNs, Transformers |
| **Hierarchical** | Simple features combine into complex ones | CNNs, deep networks |
| **Local** | Relationships between nearby elements | CNNs, attention |
| **Global** | Relationships between distant elements | Transformers, attention |
| **Relational** | How entities relate to each other | Transformers, GNNs |
| **Graph/structural** | Connections between nodes | GNNs |
| **Frequency** | Patterns in frequency rather than position | CNNs, Fourier-based methods |
before **transformer**, nn architectures for sequence processing :
**Extended Neural GPU**
**Neural GPU** was designed to process sequences using **convolution-like operations**. It used recurrent computation plus convolutional operations to process sequences.
**Extended Neural GPU** improved the original architecture to handle longer sequences and tasks such as algorithmic/sequence transformations.
**ByteNet**
was a neural architecture for seq-to-seq tasks, its important idea was **dilated convolutions**. This lets the network look at a **larger context** without requiring as many layers.
used for : machine translation, language modeling and text generation
**ConvS2S**
**Convolutional Sequence to Sequence**, alternative to RNN for sequence-to-sequence problems. made training more parallelizable than traditional RNNs.
used for : machine translation
Q. If information at position A needs to affect position Z, how many computational steps does the model need?
**A. linear fo ConvS2S and logarithmic for ByteNet**
### **Transformer** : constant path length
* **Constant number of ops** means **constant path length O(1)** between any two positions.
* token can directly attend to another token, regardless of distance.
* It does **not** mean total Transformer computation is O(1).
* attention combines information from multiple positions using weighted averaging : `Output = 0.2×A + 0.5×B + 0.3×C`
* Benefits
* **short path :** easier long-range information flow and gradient/credit assignment
* **parallel computation :** all token relationship can be computed together during training and much better CPU / GPU utilisation.
**Problems with Transformers**
* **Problem 1 :** transformers connects distant positions directly
* one attention operations mixes information : information isn't literally destroyed, but different relationships can become **mixed together**.
* so how do we preserve different relationships ? **multi-head attention mechanism**
* Instead of `what should this token attend to ?` we ask it m**ultiple times using different learned projections**.
* each head has its **own learned parameters**, so each can learn to focus on different relationships. eg.: ( this is what heads can learn, they aren’t hardcoded rules )
```plaintext
Head 1 → grammatical relationship
Head 2 → nearby relationship
Head 3 → positional/local relationship
Head 4 → another long-range relationship
```
* Instead of forcing all these relationships into **one attention pattern**, we have **multi attention patterns,** then the outputs from all heads are combined. so the model gets a richer representation containing **different kinds of relationships at same time**.
**the entire chain**
![](https://cdn.hashnode.com/uploads/covers/625a673852fb7bd53380e365/72c6763b-f63e-4dfc-9dbd-56edfad1dfaa.png align="center")
Each head has separate learned parameters, and during training, different heads may naturally specialize in different relationships.
##### **Self Attention**
> Self-attention, sometimes called intra-attention, is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.
self-attention allows each position to look at **other positions in the same sequence**.
eg.: when processing `eating` in `The dog is eating`
```plaintext
"The" → 0.05
"dog" → 0.60
"is" → 0.10
"eating"→ 0.25
```
so the representation of "eating" is computed using information from the other words, that's **self-attention**.
query and the information **( Q, K and V )** being attended to come from the same sequence. every token can attend to other tokens **within same seq**.
use cases : reading comprehension, abstractive summarisation, textual entailment and learning task-independent sentence representations.
cross-attention
Imagine two different seq : The dog is eating → Le chien mange
decoder's french tokens can attend to the english encoder seq.
french seq query → english seq key/value
that's cross-attention, because attention is happening between two different sequences.
> e2e memory networks are based on a recurrent attention mechanism instead of sequence-aligned recurrence & have been shown to perform well on simple-language question answering and language modeling tasks.
memory network is an **architecture** that gives the model an explicit memory.
**recurrent attention mechanism :** recurrence happens as we move through the seq.
**sequence-aligned recurrence :** recurrence happens by repeatedly attending to mem.
💡
Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.