Unsupervised, efficient and semantic expertise retrieval Christophe Van Gysel, Maarten de Rijke and Marcel Worring

What is expertise retrieval? Ò

The task of finding the right person with the appropriate skills and knowledge w.r.t. a topic.
 Ò

For example, an area chair looking for reviewers.


Ò

Document collections where documents are associated with one or more experts.


Ò

Given a textual topic (e.g., “Information Retrieval”), rank experts in descending order of expertise. Semantic expertise retrieval

2

Semantic expertise retrieval

3

Experts

Semantic expertise retrieval

3

Experts

Semantic expertise retrieval

3

Experts

Documents

Semantic expertise retrieval

3

Experts

Documents

Semantic expertise retrieval

3

Experts

Documents

Semantic expertise retrieval

3

Example:
 area chair looking for a review committee on “information retrieval"

Semantic expertise retrieval

3

Example:
 area chair looking for a review committee on “information retrieval"

Rank experts in decreasing order of expertise.

Semantic expertise retrieval

3

Query: "information retrieval"

1.

Example:
 area chair looking for a review committee on “information retrieval"

2.

3.

Rank experts in decreasing order of expertise.

4.

5.

Semantic expertise retrieval

3

How to do this? Experts

Documents

Semantic expertise retrieval

4

How to do this? Experts

Documents

First score documents using language models, then aggregate scores per expert.

Semantic expertise retrieval

4

How to do this? Experts

Documents

Concatenate documents associated with each expert into a pseudo-document for every expert.
 
 Perform retrieval using language models.

Semantic expertise retrieval

4

Challenges

Semantic expertise retrieval

5

Challenges Ò

Queries and documents use different representations to describe the same concepts.


Semantic expertise retrieval

5

Challenges Ò

Queries and documents use different representations to describe the same concepts.


Ò

Scoring the whole document collection during retrieval is costly when we are only interested in experts.


Semantic expertise retrieval

5

Challenges Ò

Queries and documents use different representations to describe the same concepts.


Ò

Scoring the whole document collection during retrieval is costly when we are only interested in experts.


Ò

Improve retrieval performance without requiring relevance judgments for machine-learned ranking.

Semantic expertise retrieval

5

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C .

Semantic expertise retrieval

6

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C .

Term

ti

Semantic expertise retrieval

6

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C .

ti

Embedding of t_i

Term

Semantic expertise retrieval

6

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C .

ti

Embedding of t_i

Term

Transform and apply soft-max

Semantic expertise retrieval

6

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C .

Distribution over experts

ti

Embedding of t_i

Term

Transform and apply soft-max

P (C | ti )

Semantic expertise retrieval

6

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C .

Semantic expertise retrieval

7

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C . P (C | “information”)

P (C | “retrieval”)

Semantic expertise retrieval

7

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C . P (C | “information”)

P (C | “retrieval”)

⇥

Semantic expertise retrieval

7

How to learn representations? Ò

Given a query q = t1 , . . . , tk , consisting of k terms, and a set of candidate experts C . P (C | “information”)

P (C | “retrieval”)

⇥

P (C | “information”“retrieval”)

=

Semantic expertise retrieval

7

How to learn representations? Ò

The embeddings and transformation is trained using batched stochastic gradient descent.


Ò

Word embeddings are specialised for the domain.

Semantic expertise retrieval

8

How to learn representations?

Semantic expertise retrieval

8

How to learn representations?

Semantic expertise retrieval

9

How to learn representations?

Semantic expertise retrieval

9

How to learn representations?

Semantic expertise retrieval

9

How to learn representations?

Semantic expertise retrieval

9

How to learn representations?

Semantic expertise retrieval

9

How to learn representations?

⇥

⇥

Semantic expertise retrieval

9

How to learn representations?

⇥ = ⇥

Semantic expertise retrieval

9

How to learn representations?

⇥ = Compare with reference distribution
 using cross-entropy

⇥

Semantic expertise retrieval

9

How to learn representations?

⇥ = Compare with reference distribution
 using cross-entropy

⇥

Backpropagate errors! Semantic expertise retrieval

9

Experimental set-up

Semantic expertise retrieval

10

Experimental set-up Ò

Build and evaluate models on expert finding benchmarks Ò TREC Enterprise Track (2006 - 2008): Ò W3C (715 experts, 331k docs, 99 queries) Ò CERC (3 479 experts, 370k docs, 127 queries)
 Ò

TU Expert Collection
 (977 experts, 31k docs, 1 662 queries)

Semantic expertise retrieval

10

Experimental set-up Ò

Build and evaluate models on expert finding benchmarks Ò TREC Enterprise Track (2006 - 2008): Ò W3C (715 experts, 331k docs, 99 queries) Ò CERC (3 479 experts, 370k docs, 127 queries)
 Ò

Ò

TU Expert Collection
 (977 experts, 31k docs, 1 662 queries)

Compare the log-linear model to LSI, TF-IDF and language modelling approaches (Model 1 and Model 2).

Semantic expertise retrieval

10

What window size to choose? W3C

CERC 0.45

2005 2006

0.5

2007 2008

0.40 0.35

0.4

0.3

MAP

MAP

0.30

0.2

0.25 0.20 0.15 0.10

0.1 0.05 0.0

1 2

4

8

16

Window size

32

0.00

1 2

4

8

16

32

Window size

Semantic expertise retrieval

11

What window size to choose? W3C

CERC 0.45

2005 2006

0.5

2007 2008

0.40 0.35

0.4

0.3

MAP

MAP

0.30

0.2

0.25 0.20 0.15 0.10

0.1 0.05 0.0

1 2

4

8

16

Window size

32

0.00

1 2

4

8

16

32

Window size

Semantic expertise retrieval

11

Results

Semantic expertise retrieval

12

Results Ò

Outperforms language models on 4 out of 6 benchmarks.

Semantic expertise retrieval

12

Results Ò

Outperforms language models on 4 out of 6 benchmarks. Ò 17% to 86% relative increase in MAP over state-of-the-art language models.

Semantic expertise retrieval

12

Results Ò

Outperforms language models on 4 out of 6 benchmarks. Ò 17% to 86% relative increase in MAP over state-of-the-art language models. Ò No significant difference for the other benchmarks.


Semantic expertise retrieval

12

Results Ò

Outperforms language models on 4 out of 6 benchmarks. Ò 17% to 86% relative increase in MAP over state-of-the-art language models. Ò No significant difference for the other benchmarks.


Ò

Compared to semantic matching methods (LSI):

Semantic expertise retrieval

12

Results Ò

Outperforms language models on 4 out of 6 benchmarks. Ò 17% to 86% relative increase in MAP over state-of-the-art language models. Ò No significant difference for the other benchmarks.


Ò

Compared to semantic matching methods (LSI): Ò Relative increase in MAP ranges from 83% to 1000%. Semantic expertise retrieval

12

Per-topic difference in MAP w.r.t.
 document-centric language models CERC 1.0

0.5

0.5

0.0

4M AP

1.0

0.0

0.5

0.5

1.0

1.0

1.0

1.0

0.5

0.5

0.0

4M AP

4M AP

4M AP

W3C

0.0

0.5

0.5

1.0

1.0

Semantic expertise retrieval

13

Per-topic difference in MAP w.r.t.
 document-centric language models CERC 1.0

0.5

0.5

0.0

4M AP

1.0

0.0

0.5

0.5

1.0

1.0

1.0

1.0

0.5

0.5

0.0

4M AP

4M AP

4M AP

W3C

0.0

0.5

0.5

1.0

1.0

Semantic expertise retrieval

13

Per-topic difference in MAP w.r.t.
 document-centric language models CERC 1.0

0.5

0.5

0.0

4M AP

1.0

0.0

0.5

0.5

1.0

1.0

1.0

1.0

0.5

0.5

0.0

4M AP

4M AP

4M AP

W3C

0.0

0.5

0.5

1.0

1.0

Semantic expertise retrieval

13

Per-topic difference in MAP w.r.t.
 document-centric language models CERC 1.00

0.5 .5

0.55

0.0 .0

4M AP

1.0 .0

0.00

0.5 .5

0.55

1.0 .0

1.00

1.0 .0

1.00

0.5 .5

0.55

0.0 .0

4M AP

4M AP

4M AP

W3C

0.00

0.5 .5

0.55

1.0 .0

1.00

Semantic expertise retrieval

13

What if we combine our approach with language models? Ò

For every topic qi, rank experts using the loglinear model and Model 2.


Ò

Combine these two rankings according to the reciprocal rank of experts cj .

rankensemble /

1

·

1

rankmodel 2 (cj , qi ) ranklog-linear (cj , qi ) Semantic expertise retrieval

14

What if we combine our approach with language models? Ò

Outperforms our approach on 5 out of 6 benchmarks.


Ò

Relative increase of 15% to 31% MAP compared to our discriminative approach by combining it with generative language models.

Semantic expertise retrieval

15

Can generative language models benefit from the log-linear model? Ò

Perform semantic query expansion by using the learned word embeddings.


Ò

For a particular query (e.g. “information retrieval”),
 add k terms closest to each of the terms in embedding space (e.g. “knowledge" and “search”).


Ò

Instead of querying for “information retrieval”,with k = 1, we query for: “information retrieval knowledge search”

Semantic expertise retrieval

16

Can generative language models benefit from the log-linear model? Ò

We notice an increase in MAP by performing semantic query expansion on most benchmarks.


Ò

Benchmarks that did not benefit were those for which our method did not outperform language models.


Ò

Our intuition: some benchmarks require semantic matching, while others benefit from lexical matching.

Semantic expertise retrieval

17

How efficient is the log-linear model compared to generative models? Ò

During retrieval, time complexity is linear with the number of experts.


Ò

Previous state-of-the art:
 linear w.r.t. the number of documents.

Semantic expertise retrieval

18

Code is available at
 https://github.com/cvangysel/sert

Semantic expertise retrieval

19

Code is available at
 https://github.com/cvangysel/sert [cvangysel@ilps SERT] ./W3C-expert-finding.sh

Semantic expertise retrieval

19

Code is available at
 https://github.com/cvangysel/sert [cvangysel@ilps SERT] ./W3C-expert-finding.sh Verifying W3C corpus. Creating output directory. Fetching topics and relevance judgments. Constructing log-linear model on W3C collection. Evaluating on TREC Enterprise tracks. 2005 Enterprise Track: ndcg=0.5474; map=0.2603; recip_rank=0.6209; P_5=0.4098; 2006 Enterprise Track: ndcg=0.7883; map=0.4937; recip_rank=0.8834; P_5=0.7000; Semantic expertise retrieval

19

Conclusions Ò

Our log-linear model performs competitively with existing methods, while taking time complexity linear w.r.t. the number of experts.


Ò

An ensemble between the log-linear model and generative language models performs best.


Ò

Word embeddings learned by the log-linear model can be used to improve retrieval with language models.

Semantic expertise retrieval

20

Thank you! Christophe Van Gysel @cvangysel

Maarten de Rijke @mdr

Marcel Worring @marcelworring

Slides will be made available on http://chri.stophr.be

Semantic expertise retrieval

21