Ordinal Covering Using Block Designs

A frequently encountered problem in peer review systems is to facilitate pairwise comparisons of a given set of documents by as few experts as possible. In (7), it was shown that, if each expert is assigned to review k documents then [n(n-1)/k(k-1) ] experts are necessary and [n(2n-k)/K-2] experts are sufficient to cover all n(n-1)/2 pairs of n documents. In this paper, we show that, if root n <= k <= n/2 then the upper bound can be improved using a new assignnment method based on a particular family of balanced incomplete block designs. Specifically, the new method uses [n(n+k)/K-2] experts where n/k is a prime power, n divides K-2, and root n <= k <= n/2. When k = root n, this new method uses the minimum number of experts possible and for all other values of k, where root n < k <= n/2, the new upper bound is tighter than the general upper bound given in (7).