Hidden Permutation Model and Location-Based Activity Recognition

Hung Bui SRI International

Dinh Phung, Svetha Venkatesh, Hai Phan Curtin University of Technology

Talk Outline


Why model permutations?




Distribution of random permutations




Hidden Permutation Model (HPM)




How to estimate HPM parameters?




How to perform approximate inference?




Experiments with location-based activity recognition

Why Model Permutations?


Permutations arise in many real-world problems 




Usually, there is an unknown matching that needs to be recovered    




Data association, information extraction from text, machine translation, activity recognition

Correspondence in data association Field-to-value matching in IR Word/phrase matching in machine translation A permutation is the simplest form of matching

Brute-force computation is at least O(n!)

Permutations in Activity Recognition


Many activities require carrying out a collection of substeps, each performed just once (or a repeated a small number of times) 






Factors affecting ordering between steps:   




AAAI travel = (get_approval, book_hotel, book_air_ticket, register, prepare_slides, do_travel) Ordering of steps is an unknown permutation that needs to be recovered

Strongly ordered: A enables B; A and B follow a timetable Weakly ordered: A performed before B out of habit Unordered: A performed before B by chance

Learning these ordering constraints from data can lead to better recognition performance

Permutations and Markov Models


Standard HMM does not enforce permutation constraints

xn = x1 ? xn = x2 ?. . .







Permutation constraints lead to awkward graphical models, since conditional independence is lost Need a more direct way of defining distribution on permutations

Distributions on Permutations


Let Per(n) = permutations of {1,2,…,n} 




Multinomial over Per(n) requires n! parameters (Kirshner et al, ICML 2003)

Exponential Family

f : P er(n) → Rd : feature function λ ∈ Rd : natural parameters

Very general Few parameters

E.F. distribution on permutations

Pr(x | λ) = exp {f (x), λ − A(λ)} Log-partition function   A(λ) = ln 

x∈P er(n)



exp (f (x), λ)

Expensive

Exponential Family on Permutations (cont.)


What features to use?








Factors affecting ordering between activity steps: Strongly ordered: A enables B; A and B follow a timetable Weakly ordered: A performed before B out of habit Unordered: A performed before B by chance

Does step i appear before step j in x? −1 fij (x) = I{x−1 < x i j }




With no loss of information, keep only fij (x) for i < j

d=

n(n−1) 2

features (also num. parameters)

Exponential Family on Permutations (cont.)


Simplified densityforms  Pr(x | λ) = exp 

Pr(x | λ) = exp










−1 i


l
 λij − A(λ)

λxl xk − A(λ)



Sum over all in-order pairs

Example x = (2 4 1 5 3)

λ2,4 + λ2,5 + λ2,3 + λ4,5 + λ1,5 + λ1,3

Some Properties


Swapping xi and xi+1

x′ = (x1 , . . . xi+1 , xi , . . . , xn )  −λx ,x ′ Pr(x |λ) e i i+1 if xi < xi+1 = Pr(x|λ) eλxi+1 ,xi if xi > xi+1 Cost of switching adjacent (i, j), i < j is eλij


Reverse permutation

x′ = (xn , xn−1 . . . x1 )  exp( i
const(λ)

Hidden Permutation Model


“Graphical Model”

Pr(x|λ)

Pr(ot |xt = i, η) = M ult(ηi )


Joint distribution

Pr(x, o|λ, η) = Pr(x|λ)

n 

t=1

Pr(ot |xt , ηxt )

Max. Likelihood Estimation, Permutation Known


Log-likelihood function: L(λ, η) = ln P (x | λ) + ln P (o| x, η)




Optimize η 




trivial (count frequency)

Optimize λ  

Convex problem Derivative:

i appears before j ?

▽λij (L) = fij (x) −



fij (x)P (x | λ)

x

Pr( i appears before j)

Max. Likelihood Estimation, Permutation Unknown


Log-likelihood function l (λ, η) =

K 

k=1





log

  x



P (ok , x | λ, η)

Need to jointly optimize both λ, η ; Non-convex problem Can we use EM ?  M-step to for λ does not have a closed form 

Can try coordinate ascent:




Fix η and improve λ by one gradient step Fix λ and improve η by EM (now has closed form) Didn’t work as well as simple gradient ascent

Max. Likelihood Estimation, Permutation Unknown


Derivative for λ ▽λij (l) =



fij (x)P (x| o, λ, η) Pr( i appears before j given o)

x

−



fij (x)P (x | λ)

Pr( i appears before j)

x




Derivative for η 

Avoid dealing with constraints by transforming to natural parameter for multinomial ▽ηiv (l) =



∈ o[v]}P (x | o, λ, η) I{x−1 i

x

− Pr(v|ηi )

Pr( i appears at one of v’s position(s) given o)

Approximate Inference via MCMC


Typical “inference” problem requires calculating an expectation.




Expectations can be approximated if we can generate sample from x ∼ Pr(x|λ)




How to draw random permutations? 

Try a well-known MCMC idea




Start with a random initial permutation Randomly switch two positions Accept new permutation with probability



 |λ) min PP(x (x|λ) , 1 ′

Location-Based Activity Recognition on Campus Detection Problem

Student Activity Routines (Permutation with Partial-Order Constraints)

X

Atomic Activities Atomic activities Banking Lecture 1 Lecture 2 Lecture 3 Lecture 4 Group meeting 1 Group meeting 2 Group meeting 3 Coffee Breakfast Lunch

X

Corresponding Locations

Physical locations Bank Watson theater Hayman theater Davis theater Jones theater Bookmark cafe, Library, CBS Library, CBS, Psychology Bld Angazi cafe, Psychology Bld TAV, Angazi cafe, Bookmark cafe TAV, Angazi cafe, Bookmark cafe TAV, Bookmark cafe

GPS “Places”

X

“Places” from GPS


Preprocessing  Removal of points above a speed threshold  Often missing precisely the samples we want! (e.g. buildings)  Interpolation within a day and across days  Clustered into groups to find significant places using DBSCAN

Detection Performance Activity 1

Activity 2

HMM KIR HPM HMM KIR HPM

TP 18.2 18.5 19.1 17.9 18.0 18.8

FP 19.5 2.0 4.1 4.4 0.7 0.4

Precision 48.3% 90.2% 82.3% 80.3% 96.3% 97.9%

Recall 91.0% 92.5% 95.5% 89.5% 90.5% 94.0%

In a long sequence of GPS “places”, detect occurrences of activity routine

Simulated Data, Supervised (Atomic Activities Given) Activity 1

Activity 2

NBC HMM KIR HPM NBC HMM KIR HPM

TP 16.6 18.3 18.3 19.1 17.1 17.7 18.1 18.5

FP 11.1 19.8 8.5 5.1 11.0 3.8 4.7 0.5

Precision 59.9% 48.0% 68.3 % 78.9% 60.9% 82.3% 79.4 % 97.4%

Simulated Data, Unsupervised

Recall 80.3% 91.5% 91.5% 95.5% 85.5% 88.5% 90.5% 92.5%

NBC HMM HPM

TP 6 8.5 9.8

FP 4 5.3 1.9

Precision 60% 61.6% 83.8%

Real Data, Unsupervised

Recall 60% 85% 98%

Conclusion












Modelling permutation is hard, but not impossible A general way to parameterize distribution over permutations using the exponential family If permutation is not observed, use the Hidden Permutation Model (HPM) Demonstrated better performance than other models that do not exploit permutation constraints, as well as naïve multinomial permutation model (Kirshner et al). Future work  

Generalize to permutations with repetitions In supervised mode, a discriminative formulation similar to CRF might work better