The graph convolutional network methods either weight the contribution of the neighbors equally or in a way that depends on the graph topology. Conversely, in graph attention layers, the weights depend on the data at the nodes.
A linear transformation is applied to the current node embeddings so that:
Then, the similarity of each transformed node embedding to the transformed node embedding is computed by concatenating the pairs, taking a dot product with a column vector of learned parameters, and applying an activation function:
These variables are stored in an matrix , where each element represents the similarity of every node to every other. As in dot-product self-attention, the attention weights contributing to each output embedding are normalized to be positive and sum to one using a softmax. However, only those values corresponding to the current node and its neighbors should contribute. The attention weights are applied to the transformed embeddings:
where is a second activation function. The function computes the attention values by applying softmax operation separately to each column of its argument , but only after setting values where the second argument is zero to negative infinity, so they do not contribute. This ensures that the attention to non-neighboring nodes is zero.
This is very similar to the dot-product self-attention computation in transformers, except that:
- The keys, queries, and values are all the same
- The measure of similarity is different
- The attentions are masked so that each node only attends to itself and its neighbors
As in transformers, this system can be extended to use multiple heads that are run in parallel and recombined.
