Detecting highly overlapping communities with Model-based Overlapping Seed Expansion Aaron McDaid and Neil Hurley

March 10, 2010

I

Clique percolation. I

Faster than alternatives (preliminary finding)

I

GCE

I

Model-based community finding

I

MOSES (Model-based Overlapping Seed ExpanSion) What next?

I

I I I

Benchmarks NMI follow the research basic structure. Kronecker?

MOSES

I

(Relatively) simple model

I

Optimize objective using heuristics

I

Scales

I

(a lot more to do though)

MOSES

MOSES

MOSES

Model that: I

Every pair of nodes has a (small) chance of having an edge.

I

Communities increase the probability of edges forming between member nodes.

I

Try to put most edges inside communities

I

Try to put most disconnected pairs outside communities s(i, j) number of communities in common between nodes i and j.

I

I I

Maximize s(i, j) where i and j are connected. Minimize s(i, j) where i and j are not connected.

MOSES A community around every edge. Maximizes P(edges|grouping )

MOSES I

So, maximizing P(edges|grouping ) is not good enough. Must consider P(grouping ) also. P(edges|grouping ) × P(grouping )

I

Need a prior on the grouping.

I

Arbitrary choices made so far. Might overhaul the model. Y

FX (G ) =

 s (i,j) 1−Xi,j

qo qi Z

i
×

Y

 s (i,j) Xi,j

1 − qo qi Z

i
× Qz !

1

Y ˆZ 1≤c≤Q

N nc




(N + 1)

(1)

Heuristics

I

Experimenting with various heuristics to maximize the objective I I I

I

I

Seed expansion. Start with an edge. Deletions Iterative update (a la Louvain method)

But many more things should be experimented with to get better results. Easy to efficiently optimize (with the current model) I I

Oklahoma Facebook network: 892,528 edges. 2,088 seconds. A lot more could be done to speed this up even further.

MOSES A

Current seed, C , in black. Frontier nodes in blue. The next node selected will probably be A, as it has the most connections into the seed. But that depends on pin and po and on the values of sz (i, j) for the edges

Evaluation 20-node communities (cliques), po = 0.005

1.0

I

0.0

0.2

0.4

NMI

0.6

0.8

MOSES LFM (default) LFM (last Collection) GCE Louvain method copra 5−clique percolation 4−clique percolation (dashed) Iterative Scan (dashed)

1

2

4

6

8

10

Average Overlap

12

14

15

Evaluation 20-node communities (pin = 0.4), po = 0.005

1.0

I

0.0

0.2

0.4

NMI

0.6

0.8

MOSES LFM (default) LFM (last Collection) GCE Louvain method copra 5−clique percolation 4−clique percolation (dashed) Iterative Scan (dashed)

1

2

4

6

8

10

Average Overlap

12

14

15

Evaluation 20-node communities (pin = 0.3), po = 0.005

1.0

I

0.0

0.2

0.4

NMI

0.6

0.8

MOSES LFM (default) LFM (last Collection) GCE Louvain method copra 5−clique percolation 4−clique percolation (dashed) Iterative Scan (dashed)

1

2

4

6

8

10

Average Overlap

12

14

15

Evaluation LFR k15minc15-maxc60mult1

1.0

I

0.0

0.2

0.4

NMI

0.6

0.8

MOSES LFM2−firstCol LFM2−lastCol GCE SCP−3 Louvain method copra SCP−4

1

1.2

1.6

2

3

4

5

6

Communities per node

7

8

9 10

Evaluation LFR k15minc15-maxc60mult3

1.0

I

0.0

0.2

0.4

NMI

0.6

0.8

MOSES LFM2−firstCol LFM2−lastCol GCE Louvain method copra SCP−4

1

1.2

1.6

2

3

4

5

6

Communities per node

7

8

9 10

Evaluation

I

Facebook I I

Traud et al’s five university networks. Average of 7 communities per node.

Evaluation

70

Counts 1142 1071 999 928 857 785 714 643 572 500 429 358 286 215 144 72 1

Communitiers per node

60

50

40

30

20

10

0 0

200

400

600

Degree

800

1000

1200

0.6

Size of communities found

0.4 0.3 0.2 0.1 0.0

Density

0.5

Oklahoma Princeton UNC Georgetown Caltech

1

5

10

50

Size of community

500

0.3 0.2 0.1 0.0

Density

0.4

0.5

Comms per node

1

2

5

10

20

50

Communities−per−person

100

0.2 0.1 0.0

Density

0.3

0.4

Degree Distribution

1

5

10

50

Degree

500

Acknowledgments

This research was supported by Science Foundation Ireland (SFI) Grant No. 08/SRC/I1407.