Optimal hash functions for approximate closest pairs on the n-cube Daniel Gordon and Victor Miller and Peter Ostapenko IDA/CCR

January 30, 2009

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

1 / 33

Outline

1

Introduction

2

Optimal Regions and Hash Functions

3

Hashing with Projection

4

Hashing with Codes

5

Computing Optimal Regions

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

2 / 33

Introduction

Closest Pair Problem Given a set of n-bit vectors v1 , v2 , . . . , vM , find a pair with minimal distance.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

3 / 33

Introduction

Closest Pair Problem Given a set of n-bit vectors v1 , v2 , . . . , vM , find a pair with minimal distance.

Applications DNA sequence comparison Information retrieval GET MORE EXAMPLES

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

3 / 33

Finding Close Vectors

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

4 / 33

Finding Close Vectors

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

5 / 33

Strategy 0: Check Every Pair

For lists of size M , work is O(M 2 ). Simple, but this becomes too expensive for large M .

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

6 / 33

Strategy 0: Check Every Pair

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

7 / 33

Strategy 0: Check Every Pair

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

8 / 33

Strategy 0: Check Every Pair

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

9 / 33

Strategy 0: Check Every Pair

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

10 / 33

Strategy 1: Projection

Hash on k bits, check for collisions. If there’s an error in those bits, this will fail.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

11 / 33

Strategy 1: Projection

Hash on k bits, check for collisions. If there’s an error in those bits, this will fail.

Work per Success: M · CHash + M 2 /2k+1 · CTest (1 − pk )

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

11 / 33

Strategy 1: Projection

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

12 / 33

Strategy 1: Projection

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

13 / 33

Strategy 1: Projection

01101010100110010110110001100101001101100100100110 01011100010110001001100001100011101001010101100100 10111001110111100100000010001000000010011100111001 01100101100001000010101111011000001001011000111000 11101111000010011101000000000111010111000100110111 10100100010101011000110011010100101110000011010000 00001001000111111101011001110110010000111001111011 00110001011110011101001110100001111001100110011110 11010010110111010111011110000001011110001111010011 11011100000110001001100001100010101001010101110100 10001101000100110000000101101010110100110001001000 01111011111110111010100010010001010100001000011000 11000000001010010010111100100000100010100011000001

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

14 / 33

Strategy 2: Other Hash Functions

Alternate Idea Use a different hash function, such as mapping n bits to codewords of an [n, k] error-correcting code.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

15 / 33

Strategy 2: Other Hash Functions

Alternate Idea Use a different hash function, such as mapping n bits to codewords of an [n, k] error-correcting code.

This uses more bits, but error may not be fatal.

This idea has occurred independently many times, and been patented twice.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

15 / 33

Cost of Hashing

Work per Success: (M · CHash + M 2 /2k+1 · CTest )/Ph where Ph = Ph (p) = Prob(h(vi ) = h(vi + e)

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

16 / 33

Cost of Hashing

Work per Success: (M · CHash + M 2 /2k+1 · CTest )/Ph where Ph = Ph (p) = Prob(h(vi ) = h(vi + e)

The Big Question What hash function minimizes work/success?

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

16 / 33

Example: n = 3, k = 1

Project on one bit Region Q2 maps to a point.

001 000

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

17 / 33

Example: n = 3, k = 1

Project on one bit Region Q2 maps to a point.

001 000

Ph = (1 − p)

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

17 / 33

Example: n = 3, k = 1

Project on one bit

Code C = {000, 111}

Region Q2 maps to a point.

Region B3 (1) maps to a point.

111 001 000

000

Ph = (1 − p)

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

17 / 33

Example: n = 3, k = 1

Project on one bit

Code C = {000, 111}

Region Q2 maps to a point.

Region B3 (1) maps to a point.

111 001 000

Ph = (1 − p)

GMO (IDA/CCR)

000

Ph = (1 − p)(1 − p( 12 − p))

Optimal hash functions

January 30, 2009

17 / 33

Structure of Hamming space around codewords

c2 c1

c3

c0 x

x+e

c5 c4

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

18 / 33

Standard Coding Theory vs. Hashing with Codes I

Coding Theory Correct codewords with errors.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

19 / 33

Standard Coding Theory vs. Hashing with Codes I

Coding Theory Correct codewords with errors.

Hashing with codes Correct anything with errors.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

19 / 33

Optimal Regions

Let S be the points in V that hash to 0.

h(x) = h(x + e) with probability PS (p) =

1 X d(x,y) p (1 − p)n−d(x,y) . |S| x,y∈S

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

20 / 33

Optimal Regions

Let S be the points in V that hash to 0.

h(x) = h(x + e) with probability PS (p) =

1 X d(x,y) p (1 − p)n−d(x,y) . |S| x,y∈S

Definition S is an optimal region if it maximizes this probability for any region of size |S|.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

20 / 33

Standard Coding Theory vs. Hashing with Codes II Definition If S is a code, the probability of undetected error is P(S, p) =

1 X d(x,y) p (1 − p)n−d(x,y) . |S| x,y∈S

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

21 / 33

Standard Coding Theory vs. Hashing with Codes II Definition If S is a code, the probability of undetected error is P(S, p) =

1 X d(x,y) p (1 − p)n−d(x,y) . |S| x,y∈S

Coding Theory S is a code. Minimize this probability.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

21 / 33

Standard Coding Theory vs. Hashing with Codes II Definition If S is a code, the probability of undetected error is P(S, p) =

1 X d(x,y) p (1 − p)n−d(x,y) . |S| x,y∈S

Coding Theory S is a code. Minimize this probability.

Hashing with Codes S is the sphere around a codeword. Maximize this probability.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

21 / 33

Coding Theory Aside Let Ai = #{(x, y) : x, y ∈ S and d(x, y) = i}

Distance Distribution Function A(S, ζ) :=

n X

Ai ζ i

i=0

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

22 / 33

Coding Theory Aside Let Ai = #{(x, y) : x, y ∈ S and d(x, y) = i}

Distance Distribution Function A(S, ζ) :=

n X

Ai ζ i

i=0

PS (p) := =

GMO (IDA/CCR)

1 X d(x,y) p (1 − p)n−d(x,y) |S| 1 |S|

x,y∈S n X

Ai pi (1 − p)n−i =

i=0

  (1 − p)n p A S, . |S| 1−p

Optimal hash functions

January 30, 2009

22 / 33

Projection Pn,k

Project x onto k coordinates S is an n − k subcube. DD function is A(S, ζ) = (2(1 + ζ))n−k Probability of collision is Pn,k

P

(1 − p)n (p) = 2n−k



2 1−p

n−k

= (1 − p)k .

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

23 / 33

Projection Pn,k (cont’d)

For small error rates, projection is optimal:

Theorem Let S be the 2n−k -subcube of V. For any error rate p ∈ (0, 2−2(n−k) ), S is an optimal region, and so k-projection is an optimal hash.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

24 / 33

Hashing with Codes

Perfect Codes A code is perfect if every vertex is distance ≤ e from exactly one codeword.

Perfect Binary Codes [n, n, 1] Repetition Codes [2m − 1, 2m − m − 1, 3] Hamming Codes Hm [23, 12, 7] binary Golay Code G

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

25 / 33

Binary Golay Code

S = 3−sphere The 3-sphere’s DD function is 2048 + 11684ζ + 128524ζ 2 + 226688ζ 3 + 1133440ζ 4 + 672980ζ 5 + 2018940ζ 6 .

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

26 / 33

Binary Golay Code

S = 3−sphere The 3-sphere’s DD function is 2048 + 11684ζ + 128524ζ 2 + 226688ζ 3 + 1133440ζ 4 + 672980ζ 5 + 2018940ζ 6 .

Corollary This beats projection P23,12 for p > 0.2555.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

26 / 33

Hamming Codes

S = 1 − sphere The 1-sphere’s DD function is 2m + 2(2m − 1)ζ + (2m − 1)(2m − 2)ζ 2 ,

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

27 / 33

Hamming Codes

S = 1 − sphere The 1-sphere’s DD function is 2m + 2(2m − 1)ζ + (2m − 1)(2m − 2)ζ 2 ,

Corollary This beats projection for m ≥ 4 and p > αm ≈ (m − 2)/2m

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

27 / 33

Other Linear Codes

p

0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

d=3 d=5 d=7 H4 G H5

0

5

10

15

20

25

30

k

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

28 / 33

Optimal Regions

Alternate Formulation What region of size 2t in F2n has the best P (p)?

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

29 / 33

Optimal Regions

Alternate Formulation What region of size 2t in F2n has the best P (p)?

Previous Results 2n−1 -subcube is optimal for all n, p. 2t -subcube is optimal for t ≤ 3 for all n, p. A subcube is optimal for any t, n if p is small enough.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

29 / 33

Structure of Optimal Regions Definition For x = (x1 , . . . , xn ) ∈ V, let ρi (x) := (x1 , x2 , . . . , xi−1 , 0, xi+1 , . . . xn ) and σij (x) := (x1 , . . . , min(xi , xj ), . . . , max(xi , xj ), . . . xn ).

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

30 / 33

Structure of Optimal Regions Definition For x = (x1 , . . . , xn ) ∈ V, let ρi (x) := (x1 , x2 , . . . , xi−1 , 0, xi+1 , . . . xn ) and σij (x) := (x1 , . . . , min(xi , xj ), . . . , max(xi , xj ), . . . xn ).

Definition A set S ⊂ V is a down-set if ρi (S) ⊂ S for all i ≤ n.

Definition A set S ⊂ V is right-shifted if σij (S) ⊂ S for all i, j ≤ n. GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

30 / 33

Structure of Optimal Regions (cont’d)

111

101

011

110

001

100

010

000

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

31 / 33

Optimal Regions (cont’d)

Theorem If a set S is optimal, then it is isomorphic to a right-shifted down-set.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

32 / 33

Optimal Regions (cont’d)

Theorem If a set S is optimal, then it is isomorphic to a right-shifted down-set.

Computing Right-shifted Downsets We may find all right-shifted downsets, and look for optimal regions. For size 64, there are 4384627. We have compiled tables of optimal regions of up to size 64. Unfortunately, they don’t tile the cube.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

32 / 33

Random Codes

We would expect that for large n, a random code would do well.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

33 / 33

Random Codes

We would expect that for large n, a random code would do well.

Theorem For a fixed error rate p ∈ (0, 1/2), rate R = k/n, and n sufficiently large, a random code of rate R will beat projection.

GMO (IDA/CCR)

Optimal hash functions

January 30, 2009

33 / 33