All-pairs testing

From Wikipedia, the free encyclopedia

In computer science, all-pairs testing or pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters by "parallelizing" the tests of parameter pairs.[1]

Computer scientists and mathematicians both work on algorithms to generate pairwise test suites. Numerous exist to generate such test suites, as there is no efficient exact solution for every possible input and constraints scenarios. An early researcher in this area created a short one-hour Combinatorial Testing course that covers the theory of combinatorial testing (of which pairwise testing is a special case) and shows learners how to use a free tool from NIST to generate their own combinatorial test suites quickly.


In most cases, a single input parameter or an interaction between two parameters is what causes a program's bugs.[2] Bugs involving interactions between three or more parameters are both progressively less common [3] and also progressively more expensive to find---such testing has as its limit the testing of all possible inputs.[4] Thus, a combinatorial technique for picking test cases like all-pairs testing is a useful cost-benefit compromise that enables a significant reduction in the number of test cases without drastically compromising functional coverage.[5]

More rigorously, if we assume that a test case has parameters given in a set . The range of the parameters are given by . Let's assume that . We note that the number of all possible test cases is a . Imagining that the code deals with the conditions taking only two parameters at a time, might reduce the number of needed test cases.[clarification needed]

To demonstrate, suppose there are X,Y,Z parameters. We can use a predicate of the form of order 3, which takes all 3 as input, or rather three different order 2 predicates of the form . can be written in an equivalent form of where comma denotes any combination. If the code is written as conditions taking "pairs" of parameters, then the set of choices of ranges can be a multiset[clarification needed], because there can be multiple parameters having same number of choices.

is one of the maximum of the multiset The number of pair-wise test cases on this test function would be:-

Therefore, if the and then the number of tests is typically O(nm), where n and m are the number of possibilities for each of the two parameters with the most choices, and it can be quite a lot less than the exhaustive ·

N-wise testing[edit]

N-wise testing can be considered the generalized form of pair-wise testing.[citation needed]

The idea is to apply sorting to the set so that gets ordered too. Let the sorted set be a tuple :-

Now we can take the set and call it the pairwise testing. Generalizing further we can take the set and call it the 3-wise testing. Eventually, we can say T-wise testing.

The N-wise testing then would just be, all possible combinations from the above formula.


Consider the parameters shown in the table below.

Parameter name Value 1 Value 2 Value 3 Value 4
Enabled True False - -
Choice type 1 2 3 -
Category a b c d

'Enabled', 'Choice Type' and 'Category' have a choice range of 2, 3 and 4, respectively. An exhaustive test would involve 24 tests (2 x 3 x 4). Multiplying the two largest values (3 and 4) indicates that a pair-wise tests would involve 12 tests. The pairwise test cases, generated by Microsoft's "pict" tool, are shown below.

Enabled Choice type Category
True 3 a
True 1 d
False 1 c
False 2 d
True 2 c
False 2 a
False 1 a
False 3 b
True 2 b
True 3 d
False 3 c
True 1 b

See also[edit]


  1. ^ Berger, Bernie (2003). "Efficient Testing with All-Pairs" (PDF). TechWell Corporation. Retrieved 21 November 2023.
  2. ^ Black, Rex (2007). Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional. New York: Wiley. p. 240. ISBN 978-0-470-12790-2.
  3. ^ Kuhn, D. Richard; Wallace, Dolores R.; Gallo, Albert M. Jr. (June 2004). "Software Fault Interactions and Implications for Software Testing" (PDF). IEEE Transactions on Software Engineering. 30 (6): 418–421. doi:10.1109/TSE.2004.24. S2CID 206778290.
  4. ^ Kuhn, D. Richard; Kacker, Raghu N.; Yu Lei (October 2010). Practical Combinatorial Testing. SP 800-142 (Report). National Institute of Standards and Technology. doi:10.6028/NIST.SP.800-142.
  5. ^ IEEE 12. Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. "Test Design: Lessons Learned and Practical Implications. July 18, 2008. pp. 1–150. doi:10.1109/IEEESTD.2008.4578383. ISBN 978-0-7381-5746-7. {{cite book}}: |journal= ignored (help)

External links[edit]