Covering Arrays of Strength Four and Software Testing Yasmeen Akhtar, Soumen Maity and Reshma C Chandrasekharan

Abstract A covering array t − CA(n, k, g), of size n, strength t, degree k, and order g, is a k × n array on g symbols such that every t × n sub-array contains every t × 1 column on g symbols at least once. Covering arrays have been studied for their applications to software testing, hardware testing, drug screening, and in areas where interactions of multiple parameters are to be tested. We define the coverage measure µt (A) of an array A by the ratio between the number of distinct t-tuples
contained in the column vectors of A and the total number of t-tuples given by kt gt . Given fixed values of t, k, g and n, our objective is to construct an array A of size atmost n having largest possible coverage measure. This problem is called covering arrays with budget constrains. In this article, we present an algebraic construction method for strength four covering arrays with budget constraints.

1 Introduction This article focuses on constructing new strength-four covering arrays with very good coverage measure. A covering array t −CA(n, k, g), of size n, strength t, degree k, and order g, is a k × n array on g symbols such that every t × n sub-array contains every t × 1 column on g symbols at least once. The covering array number t − CAN(k, g) is the smallest n for which a t − CA(n, k, g) exists. For example, a 4 − CA(22, 5, 2) is shown below [4]: Yasmeen Akhtar Indian Institute of Science Education and Research, Pune 411008, India, e-mail: [email protected] Soumen Maity Indian Institute of Science Education and Research, Pune 411008, India, e-mail: [email protected] Reshma C Chandrasekharan Indian Institute of Science Education and Research, Pune 411008, India, e-mail: [email protected]

1

2

Yasmeen Akhtar, Soumen Maity and Reshma C Chandrasekharan 0 0  0 0 0 

1 1 0 0 0

1 0 1 0 0

0 1 1 0 0

1 0 0 1 0

0 1 0 1 0

1 0 1 1 0

0 1 1 1 0

0 0 0 0 1

0 1 0 0 1

1 0 1 0 1

1 1 1 0 1

1 0 0 1 1

1 1 0 1 1

0 0 1 1 1

0 1 1 1 1

0 0 0 1 1

0 0 1 0 1

1 1 1 1 1

1 0 0 0 1

0 0 1 1 0

 1 1  1 1 0

There is a vast array of literature [5, 1] on covering arrays, and the problem of determining small covering arrays has been studied under many guises over the past thirty years. In [5], Hartman and Raskin discussed several generalizations of the problem of creating small covering arrays motivated by their applications in the realm of software testing. When testing a software system with k parameters, each of which must be tested with g values, the total number of possible test cases is gk . For Instance, if there are 20 parameters and three values for each parameter then the number of input combinations or test cases of this system is 320 = 3486784401. A fundamental problem with software testing is that testing under all combinations of inputs is not feasible, even with a simple product [7, 11]. This means that the number of defects in a software product can be vast, and defects that occur infrequently are difficult to find in testing. Software developers cannot test everything, but they can use combinatorial test design to identify the minimum number of tests needed to get the coverage they want. Combinatorial test design enables users to get greater test coverage with fewer tests. A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed. The goal of most combinatorial testing research is to create test suites that find a large percentage of errors of a system while having a small number of tests required. Covering arrays prove useful in locating a large percentage of errors in software systems [2, 13]. The test cases are the columns of a covering array t − CA(n, k, g). This is one of the five natural generalizations in [5]. Covering arrays with budget constraints: A practical limitation in the realm of testing is budget. In most software development environments, time, computing, and human resources needed to perform the testing of a component is strictly limited. To model this situation, we consider the problem of creating the best possible test suite (covering the maximum number of t-tuples) within a fixed number of test cases (fixed number of columns of the array). The coverage measure µt (A) of a testing array A is defined by the ratio between the number of distinct t-tuples
contained in the column vectors of A and the total number of t-tuples given by kt gt . Our objective is to construct a testing array A of size at most n having largest possible coverage measure, given fixed values of t, k, g and n. This problem is called covering arrays with budget constrains. Pair-wise or 2-way interaction testing and 3-way interaction testing are known for its effectiveness in different types of software testing [2, 8, 9]. However, software failures may be caused by interactions of more than two parameters. A recent NIST study indicates that failures can be triggered by interactions up to 6 parameters [11]. Here we consider the problem of 4-way interaction testing of the parameters. In

Covering Arrays of Strength Four and Software Testing

3

this article, we present an algebraic construction method for strength four covering arrays with budget constraints.

1.1 Fractional Linear Group We summarise the results from group theory that we use. Let F be a Galois field GF(s) where s = pm and p is prime. We adjoin to F the symbol ∞: it may be helpful to think of the resulting set X = F ∪ {∞} as the projective line consisting of s + 1 points. Define L(s) = {α : X 7→ X | xα =

ax + b , where a, b, c, d ∈ F and ad − bc 6= 0} cx + d

Here it is understood that the symbol ∞ is subject to such formal arithmetic rules ∞ as x + ∞ = ∞, ∞ = 1, etc. It is easy to verify that L(s) is a group with respect to functional composition: indeed L(s) is isomorphic with projective general linear group PGL2 (F). L(s) is called fractional linear group and |L(s)| = |PGL2 (F)| = (s2 −1)(s2 −s) (s−1)

= (s + 1)s(s − 1). It is known that the group L(s) is sharply 3-transitive on F ∪ {∞} with degree s + 1. For the undefined terms and more details we refer the reader to Robinson [12]; Chapter 7.

2 The Construction Given fixed values of t, k, g, and n we are to construct an array of size maximum n having largest possible coverage measure. Here we only consider t = 4. The elements of X = GF(g − 1) ∪ {∞} are the symbols of covering array. We choose g so that g − 1 is a prime or prime power. Group construction involves selecting a group G and finding a vector v ∈ X k . Set M to be the k × k circulant matrix generated from starter vector v. Here we take G = L(g − 1). For each a ∈ G, let Ma be the matrix formed by the action of a on the elements of M. Let C be the k × g matrix that has a constant column with each entry equal to x, for each x ∈ X. A vector v ∈ X k is said to be a starter vector for a 4 −CA(n, k, g) if any 4 × n subarray of the circulant matrix M has at least one representative from each non-constant orbit of L(g − 1) acting on 4-tuples from X. If v is a starter vector and k|L(g − 1)| + g ≤ n, then the array formed by concatenating C, Ma , a ∈ G is a covering array with coverage measure one. To see this, consider any four rows x1 , x2 , x3 , and x4 . The patterns with all equal entries occur on rows x1 , x2 , x3 , x4 since they occur in C. All other patterns appear in rows x1 , x2 , x3 , x4 since every 4 × k submatrix of M contains at least one representative from each of the orbits 2-5 given below and G is sharply 3-transitive on X = GF(g − 1) ∪ {∞}.

4

Yasmeen Akhtar, Soumen Maity and Reshma C Chandrasekharan

If starter vector is not found, we look for a vector that produces an array with maximum possible coverage measure. Such vector is called vector with good coverage. This group construction follows the technique used in [10]. Since g − 1 is prime power, the group G = L(g − 1) is sharply 3-transitive on the projective line X = GF(g − 1) ∪ {∞}. Under this group action, there are precise g + 11 orbits of 4tuples. These g + 11 orbits are determined by the pattern of entries in their 4-tuples: 1. One orbit of patterns with four equal entries or constant orbit: {[a, a, a, a] : a ∈ X} 2. Four orbits of patterns with three equal entries and one different: {[a, a, a, b] : a, b ∈ X, a 6= b}, {[a, a, b, a] : a, b ∈ X, a 6= b}, {[a, b, a, a] : a, b ∈ X, a 6= b}, {[b, a, a, a] : a, b ∈ X, a 6= b} 3. Three orbits of patterns with two copies of two different entries: {[a, a, b, b] : a, b ∈ X, a 6= b}, {[a, b, a, b] : a, b ∈ X, a 6= b}, {[a, b, b, a] : a, b ∈ X, a 6= b} 4. Six orbits of patterns with two equal entries and two different entries: {[a, a, b, c] : a, b, c ∈ X, a 6= b 6= c}, {[b, a, a, c] : a, b, c ∈ X, a 6= b 6= c}, {[a, b, a, c] : a, b, c ∈ X, a 6= b 6= c}, {[b, a, c, a] : a, b, c ∈ X, a 6= b 6= c}, {[a, b, c, a] : a, b, c ∈ X, a 6= b 6= c}, {[b, c, a, a] : a, b, c ∈ X, a 6= b 6= c} 5. g − 3 orbits of patterns with four distinct entries. The reason is this. There are g(g − 1)(g − 2)(g − 3) 4-tuples with four distinct entries and each orbit contains g(g − 1)(g − 2) 4-tuples as |L(g − 1)| = g(g − 1)(g − 2). We show an example to explain the method. Example 1. Let g = 4, k = 15, and n = 364. Then X = GF(3) ∪ {∞}. For g = 4, there are 15 orbits: 1 orbit of type 1, 4 orbits of type 2, 3 orbits of type 3, 6 orbits of type 4 and 1 orbit of type 5. Let v = (0001102∞∞1∞101∞). Build the following circulant matrix M from v:   0 ∞ 1 0 1 ∞ 1 ∞∞ 2 0 1 1 0 0 0 0 ∞ 1 0 1 ∞ 1 ∞∞ 2 0 1 1 0   0 0 0 ∞ 1 0 1 ∞ 1 ∞∞ 2 0 1 1   1 0 0 0 ∞ 1 0 1 ∞ 1 ∞∞ 2 0 1   1 1 0 0 0 ∞ 1 0 1 ∞ 1 ∞∞ 2 0   0 1 1 0 0 0 ∞ 1 0 1 ∞ 1 ∞∞ 2    2 0 1 1 0 0 0 ∞ 1 0 1 ∞ 1 ∞ ∞    M= ∞ 2 0 1 1 0 0 0 ∞ 1 0 1 ∞ 1 ∞ ∞ ∞ 2 0 1 1 0 0 0 ∞ 1 0 1 ∞ 1     1 ∞ ∞ 2 0 1 1 0 0 0 ∞ 1 0 1 ∞   ∞ 1 ∞ ∞ 2 0 1 1 0 0 0 ∞ 1 0 1    1 ∞ 1 ∞∞ 2 0 1 1 0 0 0 ∞ 1 0   0 1 ∞ 1 ∞∞ 2 0 1 1 0 0 0 ∞ 1    1 0 1 ∞ 1 ∞ ∞ 2 0 1 1 0 0 0 ∞ ∞ 1 0 1 ∞ 1 ∞∞ 2 0 1 1 0 0 0

Covering Arrays of Strength Four and Software Testing

5

Now the 24 elements of L(3) acting on M produces 24 matrices. We also need to add the following matrix, 

0 1 C= 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

0 1 2 ∞

T 0 1  2 ∞

to ensure the coverage of orbit 1. By horizontally concatenating the 24 matrices and C, we build a 15 × 364 testing array with coverage measure 0.834.

2.1 Choice of vector v Recall that we set M to be the k × k circulant matrix generated from a starter vector v. Then, any four rows in the matrix M must have at least one element from each of the orbits 2 − 5. To determine which vectors will work as starters, we define the sets d[x, y, z] for positive integers x, y and z as follows: d[x, y, z] = {(vi , vi+x , vi+x+y , vi+x+y+z ) | 0 ≤ i ≤ k − 1} where the subscripts are taken modulo k. For v to be a starter vector, each set d[x, y, z] must contain a representative from each of the orbits 2 − 5. A covering array of strength 4 satisfies the property that for any four distinct rows all possible 4-tuples of g symbols occur at least
once as a column. For computational convenience, we divide the collection of 4k choices of four distinct rows from k rows into few equivalence classes. We define an equivalence relation ∼ on this collection of choices of four rows out of k rows as (α, β , γ, δ ) ∼ (α 0 , β 0 , γ 0 , δ 0 ) if and only if β − α = β 0 − α 0 mod k and γ − β = γ 0 − β 0 mod k and δ − γ = δ 0 − γ 0 mod k. This equivalence relation induces a partition of the set of all choices of four rows into equivalence classes [x, y, z] given by [x, y, z] = {(i, i + x, i + x + y, i + x + y + z) mod k | i = 0, 1, ..., k − 1} where x = β − α, y = γ − β and z = δ − γ all considered under modulo k. For k = 9, the class [2, 2, 3] = {(0, 2, 4, 7), (1, 3, 5, 8), (2, 4, 6, 0), (3, 5, 7, 1), (4, 6, 8, 2), (5, 7, 0, 3), (6, 8, 1, 4), (7, 0, 2, 5), (8, 1, 3, 6)}. Given k, to generate all equivalence classes without repetition, we give specific choices for x, y and z: x = 1, 2, ..., b 4k c, y = x, x + 1, ..., k − 1 and z = x, x + 1, ..., k − 1 such that 1. x + y + z ≤ k − 1 − x : The reason is this. Let w = k − x − y − z. Then w has to be strictly greater than x. If w = x, then the classes [x, y, z] and [x, x, y] are the same. 2. If y > b k−2x 2 c then z ≥ x + 1: The class [x, y, x] for y > w and the class [x, w, x] for y < w are the same where x + y + x + w = k as z = x. Hence when y ≤ b k−2x 2 c we k−2x allow z ≥ x and for y > b 2 c we require z ≥ x + 1.

6

Yasmeen Akhtar, Soumen Maity and Reshma C Chandrasekharan

3. If k ≡ 0 mod 4 : In addition to the above choices of x, y and z we need to consider one more class with x = y = z = 4k . At this stage, we would like to make few remarks about the size of equivalence classes defined by above choices of x, y and z. 1. k 6≡ 0 mod 2 : If k is an odd integer then each class contains exactly k distinct choices from
the collection of 4k choices and hence there are total l = (k−1)(k−2)(k−3) distinct 24 classes of size k. 2. k ≡ 0 mod 2 : If k is an even integer then 2k can be written as sum of two positive integers a and b where a ≤ b in b 4k c different ways. Case 1 : If k 6≡ 0 mod 4 then class of the form [a, b, a] contains only 2k distinct choices. So there are total b 4k c equivalence classes of size 2k and the remaining classes are of size k. Case 2 : If k ≡ 0 mod 4 then a class of the form [a, b, a] contains only 2k distinct choices and a class of the form [a, a, a] where a = 4k contains only 4k distinct choices. Here we get total 4k − 1 equivalence classes of size 2k , exactly one class of size 4k and the remaining classes are of size k. For k = 9, the equivalence classes are [1, 1, 1], [1, 1, 2], [1, 1, 3], [1, 1, 4], [1, 1, 5], [1, 2, 1], [1, 2, 2], [1, 2, 3], [1, 2, 4], [1, 3, 1], [1, 3, 2],
[1, 3, 3], [1, 4, 2], [2, 2, 2] and each equivalence class is of size 9. Thus 14 × 9 = 94 . A vector v is said to be a starter vector if each set d[x, y, z] has a representation from each of the orbits 2 − 5. Often, representation from orbit of type 1 is taken care by attaching g constant columns, one for each symbol. Once a starter vector is found, the circulant matrix M is constructed and acted upon by the group L(g − 1) and concatenated to form an array of size k|L(g − 1)|. To this array, g constant columns are added to ensure the coverage of orbit 1 to produce a covering array of size k|L(g − 1)| + g.

3 Results The coverage measure of a covering array is always one. For computational convenience, we rewrite the coverage measure of an array A for t = 4 in terms of equivalence classes [x, y, z] and d[x, y, z] as follows:

µ4 (A) =

∑ |[x, y, z]| × number of distinct 4-tuples covered by d[x, y, z] . k
4 4 g

x,y,z

We use computer search to find vectors v with very high coverage measures. Table 1 shows vectors with high coverage, the number of test cases (n) generated by our technique, best known n with full coverage. A comparison of our construction with

Covering Arrays of Strength Four and Software Testing

7

best known covering array sizes show that our construction produces significantly smaller testing arrays with very high coverage measures.

4 Conclusions In this paper, we have proposed construction of strength four covering arrays with budget constraints. In order to test a software component with 25 parameters each having three values, our construction can generate a test suite with 153 test cases Table 1 A comparison of the number test cases (n) produced by our construction with high coverage measure and best known n for full coverage. For g = 5, the elements of GF(4) are represented as 0,1, 2, and 3; here 2 stands for x and 3 stands for x + 1. (g, k) Vector v with good coverage Our Results Best known n n (coverage measure) [3] (3,16) 00001001∞∞011∞1∞ 99 (0.828) 237 (3,17) 0000010∞∞101∞01∞1 105 (0.851) 282 (3,18) 00010∞0∞1001∞111∞∞ 111 (0.864 ) 293 (3,19) 000010010∞01∞0∞111∞ 117 (0.883) 305 (3,20) 0000110101∞0∞10∞∞11∞ 123 (0.892) 314 (3,22) 0000011∞0∞0110∞1∞∞∞01∞ 135 (0.913) 315 (3,23) 0000001∞∞0101∞10∞10∞∞∞1 141 (0.923) 315 (3,24) 00000001∞∞0101∞10∞101∞∞1 147 (0.924) 315 (3,25) 0000000011∞0∞011∞01∞0∞11∞ 153 (0.930) 363 (4,18) (4,19) (4,20) (4,21) (4,22) (4,23) (4,24) (4,25)

00010021∞∞∞21020∞2 0000121011∞01∞0∞221 0000112101202∞0221∞2 0000011021010∞2∞0221∞ 0000001102∞02021∞∞01∞1 00000001210210∞∞20112∞1 00000000121∞011∞02∞0∞112 000000000121220∞011∞2012∞

436 (0.851) 460 (0.866) 484 (0.878) 508 (0.887) 532 (0.894) 556 (0.898) 580 (0.899) 604 (0.901)

760 760 760 1012 1012 1012 1012 1012

(5,21) (5,22) (5,23) (5,24) (5,25) (5,26) (5,27) (5,28) (5,29) (5,30)

110131300∞30010∞∞3203 3∞32011200∞∞00∞0∞10010 0002∞03100∞203021332320 003∞21022212300032302310 ∞200∞0∞∞31020∞300303∞∞33 202002211000∞0121031∞∞2300 ∞∞03002030∞000∞11∞0031301∞3 013333130320∞1∞1003200310300 00012212∞010∞3110031020031010 33001∞0∞000330∞∞010012∞1313001

1265 (0.834) 1325 (0.842) 1385 (0.854) 1445 (0.860) 1505 (0.869) 1565 (0.873) 1625 (0.880) 1685 (0.883) 1745 (0.891) 1805 (0.894)

1865 1865 1865 1865 2485 2485 2485 2485 2485 2485

(6,25) (6,27) (6,28)

000403014003033404320∞1∞∞ 433∞∞01∞∞20∞03020∞∞0∞00401∞ 4023031100232200∞21∞∞2020020

3006 (0.811) 3246 (0.826) 3366 (0.829)

6325 6606 6714

8

Yasmeen Akhtar, Soumen Maity and Reshma C Chandrasekharan

that ensure with probability 0.93 that software failure cannot be caused due to interactions of two, three or four parameters whereas best known covering array in [3] requires 363 test cases for full coverage. The results show that the proposed method could reduce the number of test cases significantly.

Acknowledgements The first author gratefully acknowledges support from the Council of Scientific and Industrial Research (CSIR), India, during the work under CSIR senior research fellow scheme.

References 1. Chateauneuf, M. A., Colbourn, C. J. and Kreher, D. L., Covering Arrays of Strength Three, Designs, Codes and Cryptography, 16, pp. 235-242, 1999. 2. Cohen, D. M. , Dalal, S. R., Fredman, M. L. and Patton, G. C. The AETG system: An Approach to Testing Based on Combinatorial Design, IEEE Transaction on Software Engineering, 23 (7), pp. 437-443, 1997. 3. Colbourn, C.: Covering Array Tables for t=2,3,4,5,6 available at http://www.public.asu.edu/ ccolbou/src/tabby/catable.html 4. Covering Arrays generated by IPOG-F, available at http://math.nist.gov/coveringarrays/ipof/ipof-results.html 5. Hartman, A. and Raskin, L., Problems and algorithms for covering arrays, Discrete Mathematics, 284, pp. 149-156, 2004. 6. Hartman, A. Software and hardware testing using combinatorial covering suites, in Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, Kluwer Academic Publishers, 34, pp. 237-266, 2006. 7. Kaner, C., Falk, J. and Nguyen, H. Q., Testing Computer Software, 2nd Ed. New York, et al: John Wiley and Sons, Inc. pp. 480 pages, 1999. 8. Maity, S., 3-Way software testing with budget constraints, IEICE Transactions on Information and Systems, Vol.E-95-D, No.9, pp.2227-2231, 2012. 9. Maity, S. and Nayak, A., Improved Test Generation Algorithms for Pair-Wise Testing, Proc. 16th IEEE International Symposium on Software Reliability Engineering, Chicago, pp. 235244, 2005. 10. Meagher, K.and Stevens, B., Group Construction of Covering Arrays, Journal of Combinatorial Designs 13(1), pp. 70-77, 2005. 11. Richard Kuhn D., Dolores R. Wallace, Albert M. Gallo, Software Fault Interactions and Implications for Software Testing, IEEE Transactions on Software Engineering, vol. 30, no. 6, pp. 418-421, 2004. 12. Robinson, D. J. S. A course in the theory of groups, Second Edition, Springer, 1995. 13. Yilmaz, C., Cohen, M., and Porter, A., Covering arrays for efficient fault characterisation in complex configuration spaces, IEEE Trans. on Software Engineering, 32(1), pp. 20-34, 2006.