HyperGAN
Search…
multi_head_attention
1
description: 'layer multi_head_attention for configurable component'
2
---
3
4
# multi_head_attention layer
5
6
Adds an attention layer with multiple heads. Uses softmax. Ends in a linear
7
8
## required arguments
9
10
`size` - output size
11
12
## optional arguments
13
14
`heads` - Number of heads to use. Defaults to 4
15
16
## input size
17
18
Any 2-d tensor
19
20
## output size
21
22
First argument
23
24
## syntax
25
26
```json
27
"ez_norm CHANNELS heads=HEADS"
28
```
29
30
## examples
31
32
```json
33
"multi_head_attention 1024 heads=4"
34
```
Copied!
Last modified 1yr ago
Copy link