! Probability!for!Computer!Science! ! CSCI!3362!/!CSCI!6362! Spring!2017! Lecture!21! ! Prof.!Claire!Monteleoni! ! !

Today! !! Huffman!coding!(finish)! Markov!Chains! ! ! ! ! ! !

! With!credit!to!T.!Jaakkola!and!J.!Wortman!Vaughan.!

Bound!on!the!InformaOon!Rate! The!best!achievable!(minimum)!informaOon!rate,! for!any!code!in!which!each!symbol!is!uniquely! encoded,!is!the!entropy:! ! R(A1 , . . . , An ) H(A1 , . . . , An ) ! X n n X P (Ai )L(Ai ) P (Ai )I(Ai ) ! i=1 i=1 ! ✓ ◆ n X 1 = P (Ai ) log2 P (A ) i i=1

OpOmal!codes! FixedXlength!codes!are!opOmal!when!all!symbols! occur!with!equal!probability.! ! When!the!symbols!have!different!probabiliOes,!the! opOmal!code!will!be!a!variableXlength!code.! X!but!not!any!variable!length!code;!some!achieve!worse!informaOon! rates!than!others.!

Huffman!Code:!InformaOon!Rate!bound! The!informaOon!rate!of!the!Huffman!code!is! upper!bounded!as!follows:! ! R(A1 , . . . , An )  H(A1 , . . . , An ) + 1 ! This!is!opOmal!for!prefix!codes.! ! And!remember,!for!any!code!which!uniquely!encodes! each!symbol,!!

H(A1 , . . . , An )  R(A1 , . . . , An )

Huffman!algorithm!

At!the!end,!codewords!are!read!from!root!to!leaf.!

Credit:!D.!MacKay!2003!

Example!1!

Now:!Exercise!4! Credit:!D.!MacKay!2003!

Markov!Chains!

Example:!Topic!Modeling:!Approach!1!

Approach!2:!A!simple!sequence!model!

Approach!3:!Markov!Chain!

We!model!the!sequence!of!topics!(or!hidden!states!in!the! HMM)!using!a!Markov!Chain.!

Markov!Chains! A!Markov!chain!is!specified!by:! • !The!state!space,!the!set!of!possible!states,!S!=!{1,!…,!m}! !

• !The!transiOons!dynamics:! - !For!each!pair!of!states,!(i,!j)!in!S!x!S,!for!which!a!transiOon! from!state!i!to!state!j!is!possible,!a!transiOon!probability,!! !pij!>!0.!

! The!Markov!chain!is!then!a!sequence!of!r.v.s!X0,!X1,!X2,…!that! take!values!in!S,!such!that!for!all!Omes!n,!all!states!i,j!in!S,!and!all! possible!sequences!of!earlier!states,!i0,…,inX1!,!the!following!holds:! P (Xn+1 = j | Xn = i, Xn

1

= in

1 , . . . , X0

= i0 ) = pij

Markov!Chains!