$$ \begin{bmatrix} \cos(\phi) & -\sin(\phi)\\ \sin(\phi) & \cos(\phi) \end{bmatrix} $$

$$ A'= \begin{bmatrix}

$\quad\quad \phi_m(p) = p \cdot B^{-\frac{2m}{d}}$

• $\phi$: sine/cosine frequency
• $p$: token position
• $B$: the base constant, 10000 by default
• $m$: dimension pair index ($2m$ for even and $2m+1$ for odd)
• $d$: dimension size of the model or one attention head

$$ \frac{1}{10000^{\frac{2i}{d}}} = 10000^{-\frac{2i}{d}} = e^{ln(10000^{(-\frac{2i}{d})})} = e^{-\frac{2i}{d}ln10000} $$

$$ \text{PE}{(pos, 2i)} = \sin\left(\frac{pos}{10000^{\frac{2i}{d}}}\right) $$ $$ \text{PE}{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{\frac{2i}{d}}}\right) $$

$$ RN(x_i) = \frac{x_i}{RMS(x)}g_i, \quad \text{RMS}(a) = \sqrt{\frac{1}{n}\sum_{i=1}^n a_i^2} $$

$g \in \mathbb{R}^n$ is a learned scaling parameter

$Attention(Q, K, V, M, D) = \left[D \odot Softmax\left(\frac{QK^T}{\sqrt{d_k}} + mask(M)\right)\right]V$

• Q: query matrix of shape (batch_size, seq_len, $d_q$)
• K: key matrix of shape (batch_size, seq_len, $d_k$)
• V: value matrix of shape (batch_size, seq_len, $d_v$)
• M: Mask matrix (seq_len, seq_len), 0 for masked positions and 1 for allowed positions
• D: Dropout (seq_len, seq_len), let p be the dropout probability, each element x_i has probability p to be set to 0, and probability 1-p to be kept and scaled up by 1/(1-p) to compensate for the removed units and preserve the expected sum

$$Attention(Q,K,V)=Softmax(\frac{QK^T}{\sqrt{d_k}})V$$

• source: Meta's Llama 3.1 paper The Llama 3 Heard of Models
• Benchmark details generated by ChatGPT using GPT-4o model