A comparison of two methods for scoring an in-basket exercise.

The in-basket exercise has been successfully used for decades by a wide variety of organizations for selection and management development in both the public and private sector. (1,2,3,4,5) An in-basket was one of the exercises in AT&T's pioneering Assessment Center. (6) It is now one of

the most commonly used situational exercises, (7,8,9) and is often used outside assessment center programs. (10) However, because each candidate's responses to the in-basket items must be evaluated by trained assessors, the cost of using an in-basket exercise may discourage organizations from using it, despite its success in predicting performance in management jobs. (11,12,13) If an easier-to-score in-basket exercise that still retained the situational test format of a traditional in-basket could be developed, organizations might make greater use of this well-established selection and management development tool.

The purpose of this study was to compare two methods for scoring an in-basket. The first method involved a traditional in-basket exercise during which participants wrote down the actions they would take on each item. The second method consisted of a multiple-choice in-basket test based on the same collection of in-basket materials.

The typical in-basket contains a collection of items of varying importance and priority that managers find in their in-baskets, such as phone messages, memos, and documents, and the candidate must indicate what action they would take on each item. (11,14,15,16) Some of the items may be interrelated to add complexity, and there is also generally a time limit, which puts candidates under some time pressure to handle all of the items. It is a simulated work task designed to measure performance on work that managers typically do, so it has high face validity for candidates. (17) The collection of items in the in-basket are usually targeted to a specific job or they can be made very general, including the kinds of items that any manager might deal with. (18) Trained assessors score the exercise by coming to consensus on ratings on performance dimensions such as prioritization, decision making, delegation, organization, and interpersonal skills, or on some overall measure of performance such as "exercise effectiveness." (19)

A few researchers have experimented with alternative scoring methods primarily designed to make scoring faster and easier to do with large numbers of candidates. Felix M. Lopez, former chairman of the Educational Testing Services' Executive Study Conference, experimented with a 111-item multiple-choice questionnaire for an in-basket developed for the fictional AMA Company, as part of the American Management Association Management Course. (9) Betty Salem, Don Ellis, and Douglas Johnson developed an in-basket for a promotion test for police sergeant (which the Civil Service Commission ruled was legitimate after an official protest over its use), consisting of multiple-choice questions relating to organizational, decision making and administrative skills. (2) A. Ralph Hakstian and Karen P. Harlos, in one of a series of studies on alternative in-basket scoring systems conducted at the University of British Columbia, used a multiple-choice test to score one of the eight performance dimensions measured by the in-basket. (20) Gerald A. Kesselman, Felix M. Lopez, and Felix E. Lopez of Lopez Assessment Services, used a kind of checklist of possible actions (participants could check more than one) to score an in-basket exercise. (21) Richard C. Joines, president of Management & Personnel Systems, originated an "item-by-item" approach to scoring in-baskets. Each item was scored using a detailed scoring key that was supported by criterion-related validation. Some items were designated as priority items and an extra point was awarded if the candidate completed the item. This approach reduced scoring time to less than 30 minutes per in-basket and increased scoring reliability to the 0.90 range. (18,22) Dennis A. Joiner, a consultant who has specialized in the use of assessment center method since 1977, described a similar "item approach" for an in-basket developed for the New York Department of Corrections. Subject matter experts specified "must dos," "nice to dos," and "should not dos" for each item, and each item was scored by comparing the responses to the expert judgments.'" None of these studies reported tests for adverse impact.