Attention

Definition

Query: what I am looking for. Keys: what I have. Values: what I want to communicate. Compare the query against the keys to get attention weights, then aggregate the values with these weights.

Formula

Scalar attention:

$$\mathbf{y}=\sum _j\text{softmax}\left(\frac{{\mathbf{q}}^T{\mathbf{k}}_j}{\sqrt{d_k}}\right){\mathbf{v}}_j$$

Vector attention (Point Transformer):

$${\mathbf{y}}_i=\sum _j\text{softmax}(\gamma ({\mathbf{q}}_i-{\mathbf{k}}_j+\delta ))\odot ({\mathbf{v}}_j+\delta )$$

Scalar: one weight per pair, all channels weighted the same. Vector: a weight per channel.

• Without a positional encoding, attention is permutation-invariant and works for a variable number of tokens, which makes it a natural fit for point clouds.
• In NLP and ViT, tokens carry no position, so a positional encoding is added. Point clouds already have coordinates; the Point Transformer adds a relative encoding $\delta =\theta ({\mathbf{p}}_i-{\mathbf{p}}_j)$.
• Global attention costs $\mathcal{O}(N^2)$; the Point Transformer restricts it to kNN neighborhoods.

Appears in

• Lecture 6.2, Refresher: Attention
• Lecture 6.2, Scalar vs. vector attention
• Lecture 9, DUSt3R: self- and cross-attention decoder; VGGT: alternating frame/global attention
• Lecture 10, Stable Diffusion: cross-attention conditioning in the U-Net
• Lecture 11, 3DShape2VecSet: cross-attention encoder and decoder for vector sets

Related concepts

• Positional Encoding
• PointNet