Attention weights
A small attention matrix with a causal mask you can switch off and inspect.
| The | cat | sat | on | the | mat | |
|---|---|---|---|---|---|---|
| The | ||||||
| cat | ||||||
| sat | ||||||
| on | ||||||
| the | ||||||
| mat |
Hover a cell. The causal mask removes every future position before softmax.
Related: Self-attention
A small attention matrix with a causal mask you can switch off and inspect.
| The | cat | sat | on | the | mat | |
|---|---|---|---|---|---|---|
| The | ||||||
| cat | ||||||
| sat | ||||||
| on | ||||||
| the | ||||||
| mat |
Hover a cell. The causal mask removes every future position before softmax.
Related: Self-attention