Vector Embeddings

We’ve seen that vectors can be used to represent inputs and outputs:

$$\begin{array}{rl}2& =\left[\begin{array}{cccccccccc}0& 0& 1& 0& 0& 0& 0& 0& 0& 0\end{array}\right]\in \{0,1{\}}^{10}\\ e& =\left[\begin{array}{cccccccc}0& 0& 0& 0& 1& 0& \dots & 0\end{array}\right]\in \{0,1{\}}^{26}\end{array}$$

When it comes to word representations, we can create a vocabulary: the set of all words encountered in a dataset. We can order and index the vocabulary and represent words using one-hot vectors. Let ${\text{word}}_i$ be the $i$th word in the vocabulary:

$$\text{“cat"}=v\in \mathcal{W}\text{where}\mathcal{W}=\{0,1{\}}^{N_v}\subset {\mathbb{R}}^{N_v}$$

where $N_v$ is the number of words in our vocabulary. The vector $v$ is the one-hot encoding of the word

$$v_i=\{\begin{array}{ll}0& {\text{if word}}_i\ne \text{“cat"}\\ 1& {\text{if word}}_i=\text{“cat"}\end{array}$$

How do we handle the semantically similar words?

• “AMATH 449 is interesting” vs. “AMATH 449 is fascinating”

We would like to find a different representation for each word that incorporates their semantics.

Word embeddings can be learned end-to-end as part of a larger task, like in the case of transformer models for NLP tasks. However, they can also be learned by training a standalone embedding model whose sole objective is to produce useful vector representations of words.

Embedding Space

The embedding space is a relatively low-dimensional space where similar inputs are mapped to similar locations.

Why does this work? Words with similar meanings will likely co-occur with the same set of words, so the network should produce similar outputs, and therefore have similar hidden-layer activations.

Cosine Similarity

The cosine angle is often used to measure the distance between two vectors in the embedding (latent) space.

Vector Arithmetic with Embeddings

To some extent, we can do vector addition on embedding representations.