University of Amsterdam

Distributed Representations of Sentences and Documents

Valentin Vogelmann and Cassandra Loor

April 18, 2016

Outline Introduction Algorithms Paragraph2Vec Experiments Conclusions Discussion Sources

|

|

2

Introduction What do we want to do?



Text classification



Existing techniques aren’t good enough



New techniques have shown new hope

|

Introduction

|

3

Algorithms Which approaches exist?



Bag-of-words    

Simple Efficient Loses word order Little sense of semantics

|

Algorithms

|

4

Algorithms What techniques will we apply?



Remember Word2Vec?



We just expand it!

|

Algorithms

|

5

Algorithms What does Word2Vec do?  

Map a word to a vector King - Man + Woman = Queen

Source: Presentation by Tom Kenter [1]

|

Algorithms

|

6

Algorithms How does Word2Vec work?

Source: [4]

|

Algorithms

|

7

Algorithms How can Word2Vec be trained?

Source: [3]

|

Algorithms

|

8

Algorithms What does this look like in the paper?

Source: [2]

|

Algorithms

|

9

Paragraph2Vec How was this adjusted?

Figure: PV-DBOW

Figure: PV-DM

Source: [2]

|

Paragraph2Vec

|

10

Experiments

Two tasks, three datasets: 

Sentiment analysis: Stanford Sentiment Treebank, IMDB



Information retrieval: Queries

→ understand how the model is different from bag-of-words et al. → insight into how the model captures the semantics of words and paragraphs

|

Experiments

|

11

Experiments



11855 parsed sentences with labels for subphrases (ratings between 0 and 1)



logistic regression on vector representation predicts rating



testing: freeze word vectors, learn representation of new subphrase, feed into regression

|

Experiments

|

12

Experiments

|

Experiments

|

13

Experiments



100,000 paragraphs (labeled and unlabeled; binary labels)



NN with 1 hidden layer to predict labels from vector representation predicts label



Testing: freeze word vectors, learn representation of new paragraph, feed into NN

|

Experiments

|

14

Experiments

|

Experiments

|

15

Experiments



Answers to 1,000,000 queries, segmented into triplets



Distance of paragraphs determines which belong together



Tf-idf scores instead of raw counts

|

Experiments

|

16

Experiments

|

Experiments

|

17

Experiments



PV-DM is consistently better than PV-DBOW, combination is best (and used)



Concatenation of vectors is used rather than sum



Best context size between 5 and 12 words (empirically obtained)



Needs to be trained in parallel during testing

|

Experiments

|

18

Conclusions



unsupervised algorithm for learning fixed-length representations



extends successful paradigm of distributed representations → overcomes some of the problems of these approaches



outperforms some state-of-the-art approaches in common NLP tasks → captures semantics of the broader context

→ alternative to bag-of-words models where parsing is not available

|

Conclusions

|

19

Discussion



What is an appropriate number of words to sample from each paragraph? Does this depend on the paragraph?



Which other methods for evaluation would you choose?



Does the model capture ambiguity in words?



Does the constantly increasing number of columns in the paragraph matrix hurt the model?

|

Discussion

|

20

Sources Tom Kenter. Word2vec, April 2016. Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053, 2014. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. Xin Rong. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738, 2014. | Sources

|

21