Attention as Soft Retrieval

Single-head dot-product attention over a small key-value bank. Each key is a point in 2D; values are scalars rendered as bar heights. Drag the query around the $(k_x, k_y)$ plane to recompute the attention weights, then watch the output bar approach the weighted average of the value bars. As temperature $\tau$ drops to zero the attention collapses to one-hot retrieval of the nearest key; as $\tau$ grows the distribution flattens toward uniform.

Figure 1. Soft retrieval via scaled dot-product attention with temperature. Method: standard softmax with the 1/sqrt(d) scale (d = 2 here).

tau0.50

q_x0.00

q_y0.00

WHAT TO TRY

Drag the query among the keys: the attention lines thicken toward whichever key it sits nearest and the weighted output slides toward that key value, content-based soft retrieval.
Lower the temperature tau: the softmax sharpens toward a hard argmax with one key winning and entropy near zero; raise it and attention spreads evenly across all keys.
This single head is the building block of every transformer: a query reading a soft, differentiable average of values addressed by their keys.