> For the complete documentation index, see [llms.txt](https://imhuay.gitbook.io/studies/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://imhuay.gitbook.io/studies/notes/_archives/2022/05/attention.md).

# Attention

![last modify](https://img.shields.io/static/v1?label=last%20modify\&message=2022-10-13%2001%3A56%3A19\&color=yellowgreen\&style=flat-square)

* [Multi-head Self Attention](#multi-head-self-attention)
  * [前向过程（PyTorch 实现）](#前向过程pytorch-实现)

## Multi-head Self Attention

### 前向过程（PyTorch 实现）

```python
def forward(x, mask, H, D):
    q = k = v = x  # [B, L, N]
    B, L, N = x.shape

    # linear
    q = W_q(q).reshape([B, L, H, D]).transpose(1, 2)  # [B, H, T, D]
    k = W_k(k).reshape([B, L, H, D]).transpose(1, 2)  # [B, H, T, D]
    v = W_v(v).reshape([B, L, H, D]).transpose(1, 2)  # [B, H, T, D]

    # attention
    logits = matmul(q, k.transpose(-2, -1)) / sqrt(D) + mask
    a = softmax(logits)

    # output
    o = matmul(a, v)
    o = W_o(o).reshape([B, L, N])
    return o
```