Why train embedding?

• Embedding depends on the task: movie review sentiment analysis v.s. legal document classification
• Keras makes it easy to train your own embedding (layer_embeeding)
• Use pre-trained word embedding
  • word2vec, 2013: https://arxiv.org/pdf/1301.3781.pdf
  • Global Vectors for Word Representation (GloVe, 2014): https://nlp.stanford.edu/projects/glove/

Learn word embedding¹

Learn word embedding

Learn word embedding

Other context/target pairs

I want a glass of orange juice to go alone with my cereal.

Context:

• Last 4 words: a glass of orange \(\longrightarrow\) ?
• 4 words on the left & right: a glass of orange ? to go alone with
• Last 1 word: orange \(\longrightarrow\) ?
• Nearby 1 word (skip gram model): glass \(\longrightarrow\) ?

Word2Vec²: Skip-grams

Come up with a few context to target pairs to create our supervised learning problem

Rule: randomly pick a word as context word; randomly pick another word within some window (\(\pm 3\)) as target word

Word2Vec: Model

\[O_c \rightarrow E \rightarrow e_c \rightarrow softmax \rightarrow \hat{y}\]

\[Softmax: p(t|c)= \frac{e^{\theta_t^Te_c}}{\Sigma_{j=1}^{10,000}e^{\theta_j^Te_c}}\]

\[L(\hat{y}, y)=-\Sigma_{i=1}^{10,000} y_i log\hat{y}_i\]

• Parameters:
  • \(\theta_t\):parameter associated with output t
  • E: embedding matrix

Problems with softmax classification