Semantic Hashing Presented by : Ali Vashaee IFT725 Neural network With Hugo larochelle Hinton, Salakhutdinov 2009

Outline • • • • •

What is semantic hashing Unlabeled data Multilevel autoencoder Results Conclusion

Semantic hashing

Document retrieval • Word-count vector • If we computes document similarity directly in the word-count space, which can be slow for large vocabularies.

Document Retrieval • • • •

N= size of the document corpus V= latent variable size LSA: O(Vlog(N)) with kdtree Semantic Hashing : is not dependent on the size of the document corpus. And linear of the size of the short list that it will produce.

Deep auto-encoder

Word count vector

First layer RBM • ‘‘Constrained Poisson Model” that is used for modeling word-count vectors.

The constrained Poisson model

is the bias of the conditional Poisson model for word i, and bj is the bias of feature j.


Unrolling Word count vector

Word count vector

Fine Tuning • Conjugate gradient for fine-tuning. • Reconstructing the code words. • Replacing the stochastic binary values of hidden units with probability. • Code layer close to binary . • Deterministic noise to force fine tuning to find the binary codes in top layer.

Activities of 128 bit code

• After fine tuning the array codes are threshold to get binary values.

Training • Pretraining: Mini-baches with 100 cases. • Greedy pretrained with 50 epochs. • The weights were initialized with small random values sampled from a zero-mean normal distribution with variance 0.01. • For fine-tuning : conjugate gradients. • After fine-tuning, the codes were thresholded to produce binary code vectors.

Hashing • Hashing : a shortlist of similar documents in a time that is independent of the size of the document collection and linear.

Semantic Hashing Gaurman et al

Screen clipping taken: 11/28/2012, 6:36 PM

Hamming distance • Very little memory is needed for codes. • Fast to find the hamming distance between binary codes.

Experimental Results

• Using a 2000 word counts vector. • Two text datasets: 20-newsgroups and Reuters corpus Volume I (RCV1-v2). • 2000-500-500 (128 ,30 )

Class structure of documents • Semantic hashing with 128 bit code:

20 Newsgroups

• 3.6 ms to search through 1 million documents using 128-bit codes. • The same search takes 72 ms for 128-dimensional LSA

Reuters RCV2

Image retrieval Torralba et al

• nearest neighbors from a database of 12,900,000 images

Thank you

Semantic Hashing -

Conjugate gradient for fine-tuning. • Reconstructing the code words. • Replacing the stochastic binary values of hidden units with probability. • Code layer close ...

876KB Sizes 0 Downloads 93 Views

Recommend Documents

Semi-Supervised Hashing for Large Scale Search - Semantic Scholar
Unsupervised methods design hash functions using unlabeled ...... Medical School, Harvard University in 2006. He ... stitute, Carnegie Mellon University.

Sparse Semantic Hashing for Efficient Large Scale ...
Nov 7, 2014 - explosive growth of the internet, a huge amount of data have been ... its fast query speed and low storage cost. ..... The test time for SpSH is sufficiently fast especially when compared to the nonlinear hashing method SH. The reason i

Hashing with Graphs - Sanjiv Kumar
2009) made a strong assumption that data is uniformly distributed. This leads to a simple analytical eigen- function solution of 1-D Laplacians, but the manifold.

Compact Hyperplane Hashing with Bilinear ... - Research at Google
Compact Hyperplane Hashing with Bilinear Functions. 0. 0.5. 1. 1.5. 2. 2.5. 0. 0.05. 0.1. 0.15. 0.2. 0.25 ..... 2

Linear Cross-Modal Hashing for Efficient Multimedia ...
Oct 21, 2013 - {zhux,huang,shenht}, [email protected] ABSTRACT. Most existing cross-modal hashing methods suffer from the scalability issue in the training phase. In this paper, we propose a novel cross-modal hashing approach with a li

Fast Min-hashing Indexing and Robust Spatio ...
The increase in the number of video copies, both legal and illegal, has become a major problem in the multime- dia and Internet era. In this paper, we propose a novel method for detecting various video copies .... the transformed video derived by fra

Reversible Sketch Based on the XOR-based Hashing
proportional to the sketch length at none cost of the storage space and a little cost of the update ... Using a large amount of real Internet traffic data from NLANR,.