ISIT 2006, Seattle, USA, July 9 ­ 14, 2006

Error recovery properties of quasi-arithmetic codes and soft decoding with length constraint Simon Malinowski

Herv´e J´egou

Christine Guillemot

IRISA/University of Rennes Email: [email protected]

IRISA/University of Rennes Email: [email protected]

IRISA/INRIA Email: [email protected]

Abstract— In this paper, we propose a method to analyse the error recovery properties of quasi-arithmetic codes. This method is adapted from the one proposed in [1] for variable length codes. The expected number of symbols affected by a single bit error and the probability mass function of the gain/loss [2] of symbols following a single bit error can be computed with this method. A method to estimate this probability mass function when a bitstream is sent over a binary symetrical channel is then proposed. The aggregated state model for soft decoding of variable length codes proposed in [3] is then extended to quasiarithmetic codes, as the synchronisation recovery properties of both kind of codes are similar. The soft decoding results of this scheme reveal high performance with a reasonable computing cost.

I. I NTRODUCTION Entropy coding offers great performance in terms of compression, but it is however very sensitive to channel noise. Indeed, a single error in the bitstream may result in the desynchronisation of the decoder leading to dramatic symbol error rates (SERs). Synchronisation recovery properties of variable length codes (VLCs) have been first studied in [1], where the authors propose a model from which the error span Es following a single bit error is derived (i.e. the expected number of symbols on which the error propagates inside the bitstream). It can be shown that VLCs satisfying the Kraft inequality statistically resynchronize with a probability of 1. However, they do not always resynchronise in the strict sense (i.e. the symbol length of the decoded bitstream may differ from the one of the encoded bitstream). In [2], the authors have extended the model of [1] to calculate the probability mass function (PMF) of the so-called gain/loss. The gain/loss represents the difference between the number of encoded and decoded symbols. It is shown in [4], that E s reflects the performance of a VLC with hard decoding. In [5], the measure of the gain/loss is shown to reflect the performance of VLCs when soft decoding with length constraint strategies are applied at the decoder side. In addition, the synchronisation recovery properties of VLCs have been used in [5], to prove the optimality of the aggregated state model proposed in [3] for soft decoding of VLCs. Recently, arithmetic coding (AC) has drawn the attention of many researchers because of its use in practical applications such as JPEG2000 or MPEG-4. The major interest of AC is that the represention of the information may be arbitrarily

1­4244­0504­1/06/$20.00 ©2006 IEEE

close to the entropy. However, it is also very sensitive to channel noise, which has led many authors to design error correcting schemes involving AC [6][7][8]. Quasi-arithmetic (QA) coding is a simplified version of AC. The compression efficiency of QA coding is lower than the one of AC, but the complexity associated to the encoding and decoding processes is reduced. A QA code can be represented as finite state machines (FSMs), as done in [9]. QA codes and their representation as FSMs are recalled in Section II. Then, we propose in Section III a method to compute Es and the PMF of the gain/loss following a single bit error for QA coding. The computation of these entities are then extended to a binary symetrical channel (BSC). In [10], the authors propose an optimal state model for soft decoding of QA codes. However, the complexity associated to the decoding on this model is too high to be tractable. We will see in the last section that the aggregated state model of [3] can be applied to QA codes to reduce the complexity of the optimal decoding. The optimality of this model for QA codes stems from the synchronisation recovery properties of these codes, studied in Section III. II. Q UASI -A RITHMETIC C ODES In arithmetic coding, the real interval 0 1] is subdivided in sub-intervals. The length of the sub-intervals is proportionnal to the probabilities of the symbol sequence it represents. At the end of the encoding, the arithmetic encoder outputs enough bits to distinguish the final interval from all other possible intervals. One of the main drawbacks of AC is the coding delay and the numerical precision, as the probabilities and hence the length of the final interval fastly tend to be small. In practice, QA coding [11] is more often used. In QA coding, the initial interval is set to the integer interval f0 : : : N g. Bits are output as soon as they are known to reduce the encoding delay and the interval is re-scaled to avoid numerical precision problems. Here, we will consider QA codes as FSMs as in [12]. A QA code can be defined with two different FSMs: an encoding one and a decoding one. The number of states of these FSMs will be denoted N e and Nd , respectively. The sets of state indexes for the encoding and decoding FSM will be denoted Ie = f0 1 : : : Ne 1g and Id = f0 1 : : : Nd 1g respectively. The encoding FSM generates a variable number of bits which depends on the current state and on the source

2338

ISIT 2006, Seattle, USA, July 9 ­ 14, 2006

0=aa

a=0 n0

n1

b=11

0=a

n1

n0 1=b

b=11

b=1

a=0

a=

a=1

1=a

1= 0=aa

a= n2

n3

a=1

n3

0=ab

n2

0=ab

1=

a)

b)

Fig. 1.

Encoding (a) and decoding (b) FSM associated with the code C .

symbol that has been sent. The decoding FSM retrieves the symbol stream from the received bitstream. The number of decoded symbols for each received bit depends as well on the state of the decoding FSM and on the bit received. Let S = S1 : : : SL(S) be a source of length L(S) taking its value into the binary alphabet A = fa bg. The probability of the more probable symbol, P(a) will be denoted p in the following. This source is encoded with a QA code, producing a bitstream X = X1 : : : XL(X) of length L(X). This bitstream is sent over a noisy channel. A hard decision is taken from the received measures. The corresponding bitstream will be noted Y = Y1 : : : YL(X) . Example 1: Let us consider the (0 4) QA code proposed in [10]. This code will be denoted C in the following. The encoding and decoding FSMs are depicted on Fig.1. On the branches, are denoted the input/output bit(s)/symbol(s) associated to the FSM. For this code, N e = 4 and Nd = 5. For example, let us consider the symbol stream aabab. The encoding of this stream produces the bitstream 01010. To ensure that the encoded bitstream is uniquely decodable, rules have to be set for the last symbol. Indeed, encoding the symbol stream aa leads to the bitstream 0. Encoding symbol a would have led to the same bitstream. Hence, the transitions that trigger no bit have to be changed if they are used to encode the last symbol. These changes are represented on the FSM of Fig. 1 by dotted arrows. The representation of a QA code by two FSMs allows the computation of the compression efficiency of the code (as in [9], p.32). The expected length l ni of the output from a state ni (i 2 Ie ) of the encoding FSM is given by:

=
p

oni a

n4

1=a

b=10

l ni

1=b

1=b

+ (1  )
p

oni b

(1)

where oni a and oni b represent the number of bits produced by the transitions from state n i triggered by the symbols a and b respectively. In addition, if the transition matrix corresponding to the encoding FSM is denoted p e , the eigenvector of p e associated to the eigenvalue 1 gives the long term state occupation probabilities P(n i ) for each state of the FSM.

Hence, the expected description length (EDL) of the code is given by l

X

=

i

2Ie

l ni

P(ni ):

(2)

Let us now estimate the PMF of the bit length of the message. This PMF is assumed to be a Gaussian-like PMF whose mean is l
L(S). The variance of this distribution is estimated hereafter. Let us denote by o 0ni a (respectively o0ni b ) the square of the difference between l and o ni a (respectively oni b ). Denoting vni

=
p

0

on a i

+ (1  )
p

0

on b i

(3)

the expected variance of the bit length output for one input symbol is given by v

=

X

2Ie

vni

P(ni ):

(4)

i

The PMF of L(X) is then estimated as a Gaussian PMF of mean l
L(S) and variance v
L(S). This estimation will be used in Section III-D. III. E RROR RECOVERY FOR QAC In this section, a method is proposed to compute the quantity following a single bit error and the PMF of the gain/loss. The gain/loss will be denoted S in the sequel. This method is based on calculating gain expressions on error state diagrams, as in [1].

Es

A. Error state diagrams Let us consider that the bistreams X and Y differ only by a single bit inversion at position i. Let the tuples (N kX NkY ) denote the pair of random variables representing the state of the decoding FSM after having decoded k bits of X and Y respectively. We have :

8

X

k < i nk

=

Y

nk :

(5)

6= nYi , The bit inversion at position i of X leads to n X i which means that the decoder is desynchronized. The decoder

2339

ISIT 2006, Seattle, USA, July 9 ­ 14, 2006

TABLE I

Let us define G(z) as

C OMPUTATION OF THE TRANSITIONS AND PROBABILITY TRANSITIONS FOR THE ERROR STATE DIAGRAM WHOSE INITIAL STATE IS (n0

State

Bit sent

Bit received

0 1 0 1

1 0 0 1

(n1 (n2 (n1 (n3

n2 )

(n1 (n4

n1 )

(n0 (n0 (n1 (n1

n0 )

(n2 (n2

n1 )

0 1

0 1

n4 )

0 1

0 1

n3 )

0 1

0 1

(n3 (n3 (n4 (n4

n0 ) n2 ) n2 ) n1 ) n4 ) n3 )

Next state

(n2 (n0 (n2 (n0

will resynchronize at the bit index

Probability p

n1 )

1

n1 )

n3 )

p

1 p=(1

n2 )

p=(1

n0 )

2

p

n2 ) n0 )

2

1

2

p

+ p) 1=(1 + p) + p) 1=(1 + p)

1 1 2 z 0 z 2 z 0 z 2 z 1 z 2 z 1 z

)

(6)

To calculate Es , one can draw an error state diagram, Y whose states are the possible tuples (n X kY nk ) for k i. The initial state of the diagram is (n X n ) i
1 i
1 (i.e., the state of the decoder just before the error occurs). This means that, depending on where the error occurs (regarding the state of the decoder), the error recovery behaviour of the code will be different. Hence, N d diagrams are drawn, one for each possible value of n X i
1 ). The final states of these diagrams are the states (nk nk ) k 2 Id . When one of these states is reached, the decoder is resynchronized. Example 2: Let us consider the code C whose encoding and decoding FSMs are given in Fig. 1. Table I depicts the Y states (nX k nk ) reached when the error occurs in the state n 0 . The corresponding error state diagram is given in Fig. 2. The transition probabilities are denoted next to the branches. B. Error span To compute E s , the branches of the diagrams are labelled with an indeterminate variable z l , where the superscript l represents the number of symbols decoded by the given transition. For the diagram of Fig. 2, the values of l for each transition are given in the last column of Table I. For each branch of the diagram, the branch probability is set to the probability of the given transition in the decoding FSM times z l . Hence, the gain of the diagram from the initial state (nj nj ) j 2 0 Nd  1] to the synchronisation state S is a polynomial of the variable z :

Gnj (z ) =

X

k2N

gnj
k z k :

(7)

This gain polynomial can be calculated with Mason’s formula [13], or by inverting the matrix (I  H ), where I represents the identity matrix and H is the transition matrix associated to the error state diagram. This matrical expression is explained in more details in [4]. Let us denote by A the random variable corresponding to the number of source symbols on which the error propagates. We have

X 1 = nj ): gnj
k = P(A = kjNi


X

j2Id

X 1 = nj ) Gnj (z )P(Ni


(9)

X1 = where the long term state occupation probabilities P(N i
nj ) are calculated as explained in Section II. Hence, the coefficient of z k in the polynomial G is equal to P(A = k ). The expected span is then given by the average value of the random variable A, i.e. E s = G0 (1).

z

j such that:

.X Y j = i
2

p

n4 )

l z

z p

G(z ) =

n0 )

(8)

Example 3: Let us consider again the previous example. The gain polynomial of the diagram of Fig. 2 is given by G n0 (z ) = (1  p)z 2 + pz 3. Hence, if the bit error occurs in state n 0 , Es = 2 + p. Hence, for this code, higher is p, lower is the EDL of the code, but higher is E s . Note that the error state diagram from the state n0 of the decoder does not include any loop, which ensures that from state n 0 the synchronisation will be achieved after at most 3 source symbols. C. Computation of the PMF of

S

To calculate the PMF of S , the method explained above is used. However, the branch labelling is changed to take into account both the numbers of encoded and decoded symbols for each transition. Hence, the branches are now labelled with 0 a variable y l , where l 0 represents the difference between the number of encoded and decoded symbols along the considered transisition. Note that l 0 can be negative if the decoded sequence has more symbols than the encoded one. Example 4: Let us calculate the value of l 0 associated to the transition between the states (n 0 n0 ) and (n1 n2 ). The transition from n0 to n1 triggers the symbol a when the sequence X is decoded, whereas the transition from n 0 to n2 triggers the symbol b when Y is decoded. Hence, l 0 = 1  1 = 0. The probability of this branch on the diagram is hence set to p  y 0 = p. As in the above section, the gain polynomials on each diagram Gnj (y ) are computed and the overall polynomial G(y) is defined as:

G(y) =

X

X 1 = nj ) : Gnj (y)P(Ni


j2Id of y k in

(10)

The coefficient the polynomial G(y ) is equal to P( S = k ). Hence, we have shown how to compute both E s and the PMF of S by adapting the branch labelling on the same set of error state diagrams. Note that these quantities are valid for a single bit error. D. Extension to the BSC

We propose here to estimate P( S = i) for a symbol sequence of length L(S) that has been sent through a BSC of crossover probability  (equals to the bit error rate). In Section II, we have seen how to estimate the PMF of the bit length L(X) of a L(S)-symbol message. Let E denote the

2340

ISIT 2006, Seattle, USA, July 9 ­ 14, 2006

p2

1

(n1
n2 )

1=(1 + p)

(n3
n4 )

(n0
n0 )

p

p=(1 + p)

p2

(n0
n0 )

1

(n1
n1 )

p2

p

(n2
n1 )

Fig. 2.

1

p

e

= )=

X i

2N

P(E

Note that P(E = ejL(X) = noise ratio and is equal to P (E P (E

ejL(X) = i) = ejL(X) = i) =

(11)

i) only depends on the signal to

=







i

e

=0

e


)
i

(1

e

if if

e
i e > i:

G (y) = (G ::: ? G})(y) = | ? {z e

times

e

X 2Z

a y i e

i

(13)

i

where ? denotes the convolution product. Note that the polynomial G1 = G corresponds to the gain polynomial of (10). Let us now assume that the decoder has already recovered from previous errors when another error occurs. This assumption requires that the probability that an error occurs when the decoder has not resynchronised yet is very low. Lower is E s and higher is E b =N0 , the more accurate is this approximation. Under this assumption, the quantity S is independently impacted by multiple errors, which leads to i e

=

P(S

=

ijE = e):

(14)

With (11), the resulting gain polynomial for this BER can be expressed as

G~ (y) =

X e

2N

G (y) P(E = e)

where only the quantity P(E ~ verify coefficients g~i of G

g~

i

= = =

X e

2N

e

2N

X

a

i e

P(E

P(S

P(S

(15)

e

=

i:

= )

=

e) depends on E =N0 . The b

e

= )

(16)

ijE = e)P(E = e)

(17) (18)

n0 )

This method has been used to compute the PMF of S for the code C . This PMF is depicted on Fig. 3 together with the simulated PMF for E b =N0 = 4 dB. These values have been computed for a 100-symbol message. Note that for this code, the probabilities that S is odd are equal to zero. In the simulated values of S , these probabilities are not all equal to zero. S can indeed be odd if a bit error is located at the end of the bitstream, hence leading the decoder not to resynchronize before the end of the message.

(12)

Let us define

a

p=(1 + p)

Error state diagram for the code C associated to the initial state (n0

ejL(X) = i)P(L(X) = i):

=

S

(n2
n2 )

(n4
n3 )

2

1

1

1=(1 + p)

random variable corresponding to the number of errors in the received bitstream Y. Its probability can be expressed as P(E

1

IV. S OFT DECODING OF QUASI - ARITHMETIC CODES An aggregated state model has been proposed in [3] for soft decoding of VLCs with a length constraint. This model is parameterized by an integer parameter T , which permits trading estimation accuracy against decoding complexity. Provided that the value of T is sufficiently high, this model allows the optimal exploitation of the information conveyed by the termination constraint. This result is proved in [5] for jointly typical source/channel realizations. The states of this model are tuples of the form (N k  Mk ), where the random variables Nk and Mk denote the internal node of the VLC code tree and the symbol clock instant modulo T at the bit clock instant k, respectively. The main argument for the optimality of the scheme stems from the fact that the probabilities P(S = n) vanish as the absolute value of the quantity jnj increases. The same property has been shown in Section III-C for QA codes. Hence, it is reasonable to apply a similar state aggregation technique to QA codes, and to expect the decoding performance to be optimal for T high enough, and this together with a reduced complexity. For this purpose, the state model of the QA has to be extended. This model is initially defined by the internal state N k of the decoding FSM at the bit instant k. It is then augmented to take into account the modulo value Mk of the symbol clock random variable. Hence, the resulting state model is also defined by a tuple of the form (N k  Mk ). This state model exploits an information on the symbol clock provided that m L(S) = L(S) mod T is known at the decoder side. Indeed, this information provides a termination constraint on the decoding trellis: the set of admissible ending states is of

2341

ISIT 2006, Seattle, USA, July 9 ­ 14, 2006

0.6

1

Theoretical values Simulated values

0.5

0.1 Symbol Error Rate

P ( S)

0.4

0.3

0.01

0.2

0.1

0

Fig. 3.

Hard T=1 T=3 T=5 T=7 T=15

Ŧ10

Ŧ5

0

5

S

Theoretical and simulated PMFs of

0.001

10

S : Code C , Eb =N0

= 4 dB

the form (ni mL(S) ), where i 2 Nd . These states correspond to sequences whose symbol length modulo T satisfies the constraint. Thanks to the reduced number of states of the aggregated trellis, the complexity of the overall decoding process is dramatically reduced in comparison with the one of the optimal model. Indeed, this latter integrates the whole information on the number of symbols (see [10]). On Fig. 4, the SER is depicted versus the signal to noise ratio Eb =N0 for the code C . These results have been simulated on an additive white gaussian noise channel for a 100-symbol message. The estimation is made using a Viterbi algorithm [14] on the state model defined above. The error-resilience of the code improves as T increases. In addition, the optimal decoding performance (i.e. obtained with the optimal model of [10]) is reached for a value of T lower than L(S) which ensures that the complexity of the decoding is reduced. For instance, for Eb =N0 = 6 dB, T = 9 is sufficient to reach the optimal performance. Note that on this figure, the decoding performance for even values of T is not depicted. Indeed, according to the study of previous sections, the values of S cannot be odd for this code. Hence, T = 2
t t 2 N does not bring more information about the symbol clock than T = 2
t  1. V. C ONCLUSION In this paper, we have shown that the error recovery properties of QA codes can be modelled by adapting the method proposed in [1] for VLCs. This method allows the calculation of the expected span E s of symbols and the PMF of the gain/loss of symbols caused by a bit error. These quantities reflect the performance of a code for hard and soft decoding strategies, respectively. This study validates the optimality of the state aggregation proposed in [3] and applied in this paper to QA codes. The aggregated model allows the reduction of the decoding complexity without introducing any sub-optimality in terms of error resilience performance, provided that the aggregation parameter is appropriately chosen.

Fig. 4.

0

1

2

3 4 Signal to Noise Ratio

5

6

7

SER results versus signal to noise ratio for the code C

R EFERENCES [1] J. Maxted and J. Robinson, “Error recovery for variables length codes,” IEEE Trans. Inform. Theory, vol. IT-31, no. 6, pp. 794–801, Nov. 1985. [2] P. F. Swaszek and P. DiCicco, “More on the error recovery for variable length codes,” IEEE Trans. Inform. Theory, vol. IT-41, no. 6, pp. 2064– 2071, Nov. 1995. [3] H. Jegou, S. Malinowski, and C. Guillemot, “Trellis state aggregation for soft decoding of variable length codes,” in IEEE Workshop on Signal Processing Systems, Athens, Greece, Nov. 2005. [4] G. Zhou and Z. Zhang, “Synchronization recovery of variable length codes,” IEEE Trans. Inform. Theory, vol. 48, no. 1, pp. 219–227, Jan. 2002. [5] S.Malinowski, H. J´egou, and C. Guillemot, “On the synchronisation recovery and soft decoding of variable length codes,” in Proc. ITG’06, Apr. 2006, munich. [6] G. Elmasry, “Arithmetic coding algorithm with embedded channel coding,” Electronic Letters, July 1997. [7] C. Boyd, J. Cleary, S. Irvine, I. Rinsma-Melchert, and I. Witten, “Integrating error detection into arithmetic coding,” IEEE Trans. Commun., vol. 45, January 1997. [8] J. C. I. Kozintsev and K. Ramchandran, “Image transmission using arithmetic coding based continuous error detection,” in Proc. Data Compression Conf., DCC, Mar. 1998, snowbird, Utah. [9] M. Gormaisch, “Source coding with channel, distorsion, and complexity constraints,” Ph.D. dissertation, Stanford University, 1994. [10] T. Guionnet and C. Guillemot, “Soft and joint source-channel decoding of quasi-arithmetic codes,” EURASIP Journal on applied signal processing, vol. 3, pp. 394–411, Mar. 2004. [11] P. Howard and J. Vitter, “Practical implementations of arithmetic coding,” Image and Text Compression, J.A. Storer, ed., Kluwer Academic Publishers, Norwell,MA, pp. 85–112, 1992. [12] M. Gormisch and J. Allen, “Finite state machine binary entropy coding,” in Proc. Data Compression Conf., DCC, March 1993, pp. 449–459. [13] S. Mason, “Feedback theories : further properties of signal flow graphs,” in Proc. Inst. Radio Eng., vol. 44, July 1956, pp. 920–926. [14] A. Viterbi, “Error bounds for convolution codes and an asymptotically optimum decoding algorithm,” IEEE Trans. Inform. Theory, no. 13, pp. 260–269, 1967.

2342