Online VB for LDA in VW Matt Hoffman, Columbia Dept. of Statistics John Langford, Yahoo! Research

LDA (Blei et al. 2003) in a tiny nutshell Dirichlet Allocation (LDA) is a hierarchical • Latent Bayesian model that explains the variation in a set

of documents in terms of a set of K latent “topics,” i.e., distributions over the vocabulary

document is assumed to be a mixture of • Each these topics • Words are drawn by: • Choosing a topic z | per-doc mixture weights • Sampling from that topic z

Example LDA topics topic 0:

game games play ball score points rules first lead played goal card minutes

---------------------------

0.2027 0.1311 0.0525 0.0361 0.0305 0.0256 0.0224 0.0213 0.0211 0.0188 0.0186 0.0173 0.0163

Example LDA topics topic 1:

born career died worked served director member years december joined college january university

---------------------------

0.0975 0.0441 0.0312 0.0287 0.0273 0.0209 0.0176 0.0167 0.0164 0.0162 0.0157 0.0147 0.0145

Example LDA topics topic 2:

university college research professor science studies education degree department study academy sciences

-------------------------

0.1471 0.0584 0.0412 0.0347 0.0259 0.0229 0.0226 0.0210 0.0141 0.0136 0.0125 0.0123

Example LDA topics topic 3:

stage page stages murray mask shadow hearts finger suit min burn arrow bow

---------------------------

0.2467 0.1115 0.0631 0.0603 0.0528 0.0365 0.0320 0.0295 0.0280 0.0227 0.0215 0.0206 0.0201

Example LDA topics topic 4:

fire attack killed battle gun shot fight shooting men enemy attacks fighting weapons

---------------------------

0.0462 0.0392 0.0391 0.0363 0.0194 0.0185 0.0179 0.0171 0.0165 0.0161 0.0152 0.0143 0.0143

Example LDA topics topic 5:

due effects caused found cause reported study damage people result high associated

-------------------------

0.0198 0.0166 0.0132 0.0125 0.0125 0.0125 0.0116 0.0114 0.0113 0.0113 0.0113 0.0108

Example LDA topics topic 6:

california san los mexico francisco santa del mexican city las juan antonio orange american

-----------------------------

0.1872 0.1705 0.1066 0.0865 0.0655 0.0399 0.0394 0.0369 0.0339 0.0245 0.0239 0.0194 0.0188 0.0165

Online VB for LDA (Hoffman et al., NIPS 2010)

• Until converged: • Choose a mini-batch of documents randomly • For each document in that mini-batch approximate posterior over what • Estimate topics each word in each document came from update approximate posterior over topic • (Partially) distributions based on what words are believed to have come from what topics

Online VB for LDA in VW To learn a set of topics: ./vw wiki.dat --lda 10 --lda_alpha 0.1 --lda_rho 0.1 --lda_D 75963 --minibatch 256 --power_t 0.5 --initial_t 1 -b 16 --cache_file /tmp/vw.cache --passes 2 -p predictions.dat --readable_model topics.dat

Online VB for LDA in VW

./vw wiki.dat: Analyze word counts in wiki.dat --lda 10: Use 10 topics

Online VB for LDA in VW Hyperparameters: --lda_alpha 0.1: θd ~ Dirichlet(α) --lda_rho 0.1: βk ~ Dirichlet(ρ) # of documents --lda_D 75963: We’ll analyze a total of 75963 unique documents

Online VB for LDA in VW

Learning parameters: --minibatch 256: Analyze 256 docs at a time --power_t 0.5, --initial_t 1: Stepsize schedule ηt = (initial_t + t)-power_t

Online VB for LDA in VW

-b 16: We expect to see at most 216 unique words

Online VB for LDA in VW To run multiple passes through the dataset: --cache_file /tmp/vw.cache: Where to cache parsed word counts --passes 2: Number of times to go over the dataset

Online VB for LDA in VW

-p predictions.dat: File to print out the inferred perdocument topic weights to --readable_model topics.dat: We print out the topics in human-readable format to topics.dat

Data Format No labels, no namespace | word_id:word_ct word_id:word_ct word_id:word_ct word_id:word_ct … | word_id:word_ct word_id:word_ct word_id:word_ct word_id:word_ct … | word_id:word_ct word_id:word_ct word_id:word_ct word_id:word_ct … …

Output Predictions Format Each line corresponds to a document d Each column corresponds to a topic k γ1,1 γ1,2 … γ1,k … γ1,K 1 γ2,1 γ2,2 … γ2,k … γ2,K 1 … γd,1 γd,2 … γd,k … γd,K 1 …

Output Topics Format Each line corresponds to a topic k Each column corresponds to a word w λ1,1 λ1,2 … λ1,k … λ1,K λ2,1 λ2,2 … λ2,k … λ2,K … λw,1 λw,2 … λw,k … λw,K … λW,1 λW,2 … λW,k … λW,K

Online VB for LDA in VW To learn a set of topics: ./vw wiki.dat --lda 10 --lda_alpha 0.1 --lda_rho 0.1 --lda_D 75963 --minibatch 256 --power_t 0.5 --initial_t 1 -b 16 --cache_file /tmp/vw.cache --passes 2 -p predictions.dat --readable_model topics.dat

LDA from vowpal wabbit - GitHub

born --- 0.0975 career --- 0.0441 died --- 0.0312 worked --- 0.0287 served --- 0.0273 director --- 0.0209 member --- 0.0176 years --- 0.0167 december --- 0.0164.

55KB Sizes 48 Downloads 355 Views

Recommend Documents

Vowpal Wabbit 2016 - GitHub
Community. 1. BSD license. 2. Mailing list >500, Github >1K forks, >1K,. >1K issues, >100 contributors. 3. The official strawman for large scale logistic regression @ NIPS :-) ...

Vowpal Wabbit 5.1 - GitHub
The Tutorial Plan. 1. Baseline online linear algorithm. 2. Common ... example_39|excuses:0.1 the:0.01 dog ate my homework |teacher male white Bagnell AI ate.

Vowpal Wabbit - GitHub
void learn(void* d, learner& base, example* ec). { base.learn(ec); // The recursive call if ( ec-> nal_prediction > 0) //Thresholding ec-> nal_prediction = 1; else.

Vowpal Wabbit - GitHub
QF ellredu™eF „er—s™—le le—rning p—per a most ... vw -c rcv1.train.raw.txt -b 22 --ngram 2. --skips 4 ... ƒolutionX en explor—tion li˜r—ry whi™h r—ndomizes.

Vowpal Wabbit 2015 - GitHub
iPython Notebook for Learning to Search http://tinyurl.com/ ... VW learning to search. 9. Hal Daumé III ([email protected]). Training time versus test accuracy ...

Vowpal Wabbit 6.1 - GitHub
It just works. A package in debian & R. Otherwise, users just type make , and get a working system. At least a half-dozen companies use VW. Favorite App: True ...

Vowpal Wabbit 7 Tutorial - GitHub
Weight 1 by default. – Label: use {-1,1} for classification, or any real value for regression. 1 | 1:0.43 5:2.1 10:0.1. -1 | I went to school. 10 | race=white sex=male ...

Vowpal Wabbit 2015 - PDFKUL.COM
Active Learning in VW: Simulation Mode vw --binary --active --simulation --mellowness 0.01 labeled.data. --mellowness: small value leads to few label queries vw --binary --active --cover 10 --mellowness 0.01 train.data. --cover: number of classifiers

Vowpal Wabbit 6.1 - PDFKUL.COM
What goes wrong? And xes. 2.1 Importance Aware Updates. 2.2 Adaptive updates. 3. LBFGS: Miro's turn. 4. Terascale Learning: Alekh's turn. 5. Common questions we don't have time to cover. 6. Active Learning: See ... (1). 4. Update wi ← wi+ η2(y −

Vowpal Wabbit 5.1 - PDFKUL.COM
wixi clipped to interval. [0,1]. 3. Learn truth y ∈ [0,1] with importance I or goto. (1). 4. Update wi ← wi+ η2(y − ˆy)Ixi and go to (1). ... 1 | 13:3.96e-02 24:3.47e-02 69:4.62e-02 example_39|excuses the dog ate my homework. 1 0.500000 examp

Vowpal Wabbit 7 Tutorial - PDFKUL.COM
General Options. Other Useful Options. • -b n, default is n=18: log number of weight parameters, increase to reduce collisions from hashing. • -q ab, quadratic features between all features in namespace a* and b*. • --ignore a, removes features

Link-PLSA-LDA
Machine Learning Department,. Carnegie ..... ploy the mean-field variational approximation for the pos- .... size of the pruned corpus is quite small compared to the orig- .... business. TOP BLOG POSTS ON TOPIC billmon.org willisms.com.

Perturbation LDA
bDepartment of Electronics & Communication Engineering, School of Information Science & Technology, .... (scatter) matrix respectively defined as follows:.

1D-LDA versus 2D-LDA: When Is Vector-based Linear ...
Nov 26, 2007 - Security, P. R.. China. 4Center for Biometrics and. Security Research & ...... in Frontal view, Above in Frontal view and two Surveillance Views, ...

Reading from SQL databases - GitHub
Description. odbcDriverConnect() Open a connection to an ODBC database. sqlQuery(). Submit a query to an ODBC database and return the results. sqlTables(). List Tables on an ODBC Connection. sqlFetch(). Read a table from an ODBC database into a data

Wabbit online calculator instructions Wabbitemu.pdf
Page 1 of 76. Wabbit TI-84 Plus Silver Edition Emulator Instructions. Go to this website http://wabbit.codeplex.com. Download Wabbitemu. Run Wabbitemu.exe. Select “Create a ROM image Select Calculator Type TI-84 Plus SE. using open source software.

Generalizing relevance weighted LDA
Rapid and brief communication ... Key Lab of Optoelectronic Technology and Systems of Education Ministry of China, Chongqing University, Chongqing 400044 ...

EM for Probabilistic LDA
2 tr(XiXi). ) ,. (7) where Xi = [xi1 ···xini]. 1.3 Likelihood. The complete-data log-likelihood, for speaker i is: p(Mi|yi,Xi,λ) = ni. ∏ j=1. N(mij|Vyi + Uxij,D−1). (8). ∝ exp.

Complexity Anonymous recover from complexity addiction - GitHub
Sep 13, 2014 - Refcounted smart pointers are about managing the owned object's lifetime. Copy/assign ... Else if you do want to manipulate lifetime, great, do it as on previous slide. 2. Express ..... Cheap to move (e.g., vector, string) or Moderate

Wabbit online calculator instructions Wabbitemu.pdf
Go to this website http://wabbit.codeplex.com. Download Wabbitemu. Run Wabbitemu.exe. Select “Create a ROM image Select Calculator Type TI-84 Plus SE.

WABBIT TI84+ SE EMULATOR.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. WABBIT TI84+ ...

Wabbit online calculator instructions Wabbitemu.pdf
Page 1 of 2. 14/11/2012 Página 1 de 9 Profesor: Luís Rodolfo Dávila Márquez CÓDIGO: 00076 UFPS. CURSO: CÁLCULO INTEGRAL. UNIDAD 2 A.

Electric Field from Point Charges - GitHub
N/C. The electric field equation for a point charge is: ̂. To find the electric field from several point charges we can use superposition and just add the E field.

Labeled LDA: A supervised topic model for credit ...
A significant portion of the world's text is tagged by readers on social bookmark- ing websites. Credit attribution is an in- herent problem in these corpora ...