Skip to content
AI Engineer Guide.

Attention weights

A small attention matrix with a causal mask you can switch off and inspect.

Thecatsatonthemat
The
cat
sat
on
the
mat

Hover a cell. The causal mask removes every future position before softmax.

Related: Self-attention