Why train embedding?

• Embedding depends on the task: movie review sentiment analysis v.s. legal document classification
• Keras makes it easy to train your own embedding (layer_embeeding)
• Use pre-trained word embedding
  • word2vec, 2013: https://arxiv.org/pdf/1301.3781.pdf
  • Global Vectors for Word Representation (GloVe, 2014): https://nlp.stanford.edu/projects/glove/

Learn word embedding¹

Learn word embedding

Learn word embedding

Other context/target pairs

I want a glass of orange juice to go alone with my cereal.

Context:

• Last 4 words: a glass of orange ⟶ ?
• 4 words on the left & right: a glass of orange ? to go alone with
• Last 1 word: orange ⟶ ?
• Nearby 1 word (skip gram model): glass ⟶ ?

Word2Vec²: Skip-grams

Come up with a few context to target pairs to create our supervised learning problem

Rule: randomly pick a word as context word; randomly pick another word within some window (±3) as target word

Word2Vec: Model

Oc→E→ec→softmax→ˆy

Softmax:p(t|c)=eθTtecΣ10,000j=1eθTjec

L(ˆy,y)=−Σ10,000i=1yilogˆyi

• Parameters:
  • θt:parameter associated with output t
  • E: embedding matrix

Problems with softmax classification