African-Americans' Test-Taking Attitudes and their Effect on Cognitive Ability Test Performance:...

This review examined recent research literature which has reported that African-Americans' attitudes toward cognitive ability tests affect their subsequent test performance. In particular, the negative attitudes of African-Americans toward cognitive predictor measures have been shown in some cases to reduce their performance on such tests, even though these tests remain valid predictors of job performance. The implications of the above findings for personnel selection practice are discussed.

The use of tests in modern public personnel selection is premised upon the assumption that an applicant's performance on a predictor measure will be indicative of his or her performance on the job,[1] although in the public sector there is often a competitive aspect to the examination. In response, the Uniform Guidelines on Employee Selection Procedures emphasizes the concepts of test validity and job relevance.[2] A second assumption related to selection test usage is the belief that a test provides an accurate sampling of the knowledge, abilities, skills, or competencies possessed by the applicant.

In the selection literature, there has been a long history of arguments over the fundamental assumptions of job relevancy and accurate sampling in testing African-Americans. Nutshelling a vast body of literature, we find that adverse impact on cognitive ability tests remains a major area of concern, but the search for explanations for the depressed performance of African-American testtakers has largely proved to be unfruitful. Recently, several lines of inquiry have found that African-Americans' performances on tests may be negatively affected by attitudes toward testing, thereby suggesting that, by modifying these attitudes, we might increase African-American performance on traditional tests.[3]

In this paper, we will briefly discuss three relevant studies and their findings. We will then discuss some possible implications of the research findings for public personnel assessment and areas of future research. We should note that the findings from each of these studies require further replication, and that, in some cases, the strength of the findings may be questionable. However, the studies are consistent in suggesting that African-Americans' test-taking attitudes may have the effect of reducing their performances on tests labeled or seen as measures of cognitive ability. It should come as no surprise to those in the testing profession that African- Americans are often contemptuous of the testing enterprise, especially when one considers the social stigma attached to being viewed as intellectually inferior.[4] Further, in the public sector, one often finds a history of challenges and mistrust of testing programs by minority group members.

Arvey, Strickland, Drauden and Martin

The Arvey et al. study was a three-part investigation which sought to examine the motivational components of the test-taking process. In the first phase of the study, the authors developed and validated the Test Attitude Survey (TAS), an attitude instrument designed to measure the opinions of employment testtakers toward the tests they have completed. The TAS was administered to 494 Minnesota state employees. When their responses were subjected to factor analysis, nine factors were derived. The nine factors included motivation, lack of concentration, belief in tests, comparative anxiety, test ease, external attribution, general need for achievement, future effects (consequences of performance), and preparation (see Arvey et al. for greater explanation of the nature of each factor). Of primary interest to the present review, however, are the results reported for Study 2 of their series of studies.

In the second study of their investigation, Arvey and colleagues explored the degree of relationship between scores on the Test Attitude Survey and scores on three employment tests within a sample of 337 applicants for a county financial worker position. The three employment tests included a comparison test (48-item speeded test requiring examinees to indicate whether two columns of information were identical or not), an arithmetic test (30-item test consisting of simple mathematical calculations), and a work sample test (a one-hour, 30-item test, which simulated actual job duties, that required testtakers to interpret a series of arithmetic rules, and consult tables to obtain information elicited by the test items; some of the items required reading comprehension ability).

The findings reviewed from Study 2 of Arvey et al. will be limited to those related to the issue of minorities' attitudes toward tests, and the effect of these attitudes on test scores. Arvey and colleagues examined specifically whether there were black/white differences in motivation toward employment tests, and whether these motivational differences might account for any of the racial gap in test performance. The authors found that applicant race was significantly related to the TAS Motivation factor, suggesting that whites had more test-taking motivation than blacks. More importantly, it was found that test score differences found between black and white applicants (favoring whites) were significantly reduced when the six TAS factor scores were held constant for race. The above finding was significant only for the work sample and arithmetic tests. Thus, the results of Arvey et al. Study 2 suggest that the lack of test-taking motivation among African-American applicants undermined their subsequent performances on employment tests.

Steele and Aronson

The Steele and Aronson study has already been frequently cited in the literature and has been the subject of newspaper columns and editorials. However, the reader should be forewarned that the results of the Steele and Aronson study were not as strong as some of the follow-up publicity might suggest.

Steele and Aronson argued that the test performance of African-Americans may be compromised by their perception of stereotype threat in testing situations. The authors defined stereotype threat as a form of anxiety that results when a person is concerned that his or her performance in some domain may substantiate a negative stereotype that exists about his/her group. According to stereotype threat theory, if African-Americans fear performing poorly on cognitive measures and confirming the negative stereotype concerning their group's mental ability, they will suffer a decrease in the concentration and attention which testing demands. In other words, the anxiety brought about by stereotype threat shifts one's attention from on-task to off-tasks concerns,[5] subsequently depreciating one's test performance. Therefore, the authors hypothesized that, if the influence of stereotype threat could be reduced, then the intelligence test performance of African-Americans would be increased.

Subsequently, the results of Steele and Aronson partially supported the above hypothesis. Test performance was operationalized during the study as the number of Graduate Record Examination-Verbal items answered correctly. Utilizing a sample of 117 Stanford undergraduates, the authors found that when a cognitive ability test was described as a test of mental ability, whites significantly outperformed African-American participants. However, when the same test was framed as a measure of problem-solving ability, and thus nonindicative of intellectual ability, the test performance of African-Americans increased dramatically, and differed only minimally from their white counterparts.

A shortcoming of the above finding is the fact that the race by test description condition (i.e., test framed as diagnostic or nondiagnostic of IQ) interaction initially hypothesized by the authors was not statistically significant. Partial evidence of the stereotype threat phenomenon undermining the test performance of blacks is apparent, however, from the finding that blacks in the nondiagnostic condition significantly outperformed their black counterparts in the diagnostic condition. A study we completed recently also showed some support for the race by test description condition interaction posited by Steele and Aronson within a sample of college undergraduates.[6] After controlling for age and socioeconomic status, McKay, Doverspike, Bowen-Hilton, and Martin found that the African-American/white IQ test score difference favoring the latter group was largest among participants who were informed that the Raven Advanced Progressive Matrices (Raven's; a popular, nonverbal IQ test) was a measure of intellectual ability. An examination of the standardized difference of Raven's scores between African-Americans and whites (d's= .90 and .32 in the diagnostic and nondiagnostic conditions, respectively) showed that the racial disparity in mean Raven's scores diminished by over 50 percent among participants who viewed the Raven's as nonindicative of intellectual ability. Thus, the mere framing of an ability predictor as indicative or nonindicative of one's intellectual ability can alter the performance of African-Americans on cognitive measures. The social stigma concerning the lack of intellectual prowess of blacks[7] has an apparent negative effect on their subsequent test performances.

Chan, Schmitt, DeShon, Clause, and Delbridge

The Chan et al. study explored testtakers' reactions to cognitive ability tests and the interrelationships among race, test performance, face validity, and test-taking motivation. Although a series of hypotheses were examined during the course of the Chan et al. study, we will review the results related to their second hypothesis. The authors posited that the effect of race on test performance during a second test administration (i.e., 2nd parallel version of a test given previously) would be influenced by test-taking motivation assessed prior to taking the test. In other words, any black/white differences in test performance found would be affected to some extent by racial differences in test-taking motivation.

The sample utilized in the study consisted of 180 undergraduate students from Michigan State University. Participants were administered a seven-item scale derived from the Motivation scale of the Test Attitude Survey, a four-item face validity scale, and a cognitive ability test battery developed by the authors. The cognitive ability test battery consisted of six subtests, including verbal comprehension (30 items), grammar (30 items), quantitative reasoning (15 items examining reasoning and problem solving by utilizing mathematical information), deduction (20 items involving rule application and logical reasoning questions), induction (30 items consisting of rule generation, verbal analogy and classification, and series completion questions), and distraction (19 items assessing one's ability to perform several tasks simultaneously). Two parallel versions of the cognitive ability test battery were developed and administered in a test-retest fashion.

As stated previously, the second hypothesis of Chan et al. posited that black/white differences in performance on the cognitive ability test taken a second time would be influenced by racial differences in test motivation. Several findings of Chan et al. are relevant to the current review. First, whites significantly outperformed blacks on both administrations of the cognitive ability test battery (scored as a sumtotal of performance across the six subtests). Second, blacks reported significantly lower test-taking motivation than whites. Third, when test-taking motivation was held constant, the variance in performance on the second cognitive ability test battery accounted for by race was significantly reduced. Thus, the test performance difference between black and white testtakers (favoring whites) was found to be partially explained by the lower test-taking motivation of black examinees.

Implications of Reviewed Findings for Public Personnel Selection Practice

The findings of the Arvey et al., Steele and Aronson, and Chan et al. studies reviewed above are consistent in suggesting that negative perceptions of cognitive ability measures held by some African-Americans, and the fear of substantiating negative stereotypes by performing poorly on cognitive ability tests, may contribute to their (1) lower test-taking motivation, (2) greater anxiety, and (3) poorer performance on cognitive ability tests. This situation is quite unfortunate. Cognitive ability tests are recognized as the most valid predictors of test performance, and appear to be equally valid for African-Americans and other ethnic groups.[8] Furthermore, the use of cognitive predictors results in the higher utility (in terms of cost benefits/savings) relative to alternative predictors currently available.[9] The question that remains is what may be done to reduce the negative attitudes that African-Americans hold toward testing, and to increase their general motivation toward test taking? The research does offer some possibilities. However, it should be noted that all of the following prospects are hypothetical, and additional research is needed.

Use of Test-Taking Skills Training Programs

First, the most obvious approach would be to change the current motivation and attitudes toward testing held by African-Americans. This could be attempted through various types of training programs, including those currently described as test wiseness training, which we discuss here under the topic of test-taking skills training programs. We argue for the need to emphasize, in training, the adoption of positive attitudes toward the test and the importance of working hard and doing well.

One problem we have seen frequently with test wiseness training programs is that they often take a negative attitude toward testing. That is, tests are treated as tricky and something that you can "beat" if you know the secrets. This would seem to create a negative attitude toward tests. While we are not opposed to teaching potential testtakers to be more test-wise, it also seems that test-taking skills training programs should emphasize the value of studying, practice, and hard work on the day of the test. Test-taking skills training programs should encourage an overall positive attitude toward the testing process, and attempt to increase rather than decrease participant motivation.

Another aspect which can be included in pretest training programs is methods for dealing with anxiety. If stereotype threat leads to greater anxiety among African-American testtakers, then they should benefit from instruction on how to handle this anxiety.[10]

Alternative Testing Formats

In the laboratory, simply relabeling the test to make it seem to be noncognitive may have an effect on African-American test performance. Of course, in the real world, this simple type of subterfuge is unlikely to work for long. Nor would one want to describe tests which are obviously being used for selection purposes as nondiagnostic of ability. This leaves the possibility that adverse impact may be reduced using alternative testing formats. This suggestion is supported by Goldstein, Yusko, Braverman, Smith, and Chung[11] who reported evidence that the level of black/white test performance differences derived from a series of predictor measures depends upon the format in which the tests are presented. Among a sample of 633 assessment center participants, Goldstein et al. found that white examinees outperformed blacks to a greater extent on more cognitively oriented tests/exercises (i.e., in-basket exercise, in-basket coaching), while ethnic differences in test performance were minimal on an interactive role-playing exercise. Furthermore, the role-playing exercise was shown to be a valid predictor of supervisory ratings of job performance (r=.12, p [less than] .01).

The findings of a recent study by Chan and Schmitt[12] further support the premise that alternative testing formats to the typical paper-and-pencil test can (1) increase blacks' perceptions of the face validity of the tests, and (2) reduce adverse impact that results from use of the test. Although the authors used a pool of undergraduate students as subjects, the findings of Chan and Schmitt showed that black/white situational judgment test performance differences (favoring whites) were substantially reduced when the test was presented in a video (effect size; d= -.28) versus a paper-and-pencil administration of the test (effect size; d= -1.19). In addition, blacks perceived the video-based test as more face valid than the paper-and-pencil measure. An important limitation of the Chan and Schmitt study, however, was their failure to collect criterion performance information from their subjects. Thus, they were unable to assess the relative validities of the video-based and paper-and-pencil predictor tests.

Chan and Schmitt suggest that the substantial difference in test scores between blacks and whites on the paper-and-pencil test resulted from deficits in reading comprehension among black examinees in their sample (see Barrett, Miguel, and Doverspike[13] for further discussion on this topic). Also, the authors conjectured that blacks were probably less motivated to perform on paper-and-pencil predictor tests, because they perceived them as face invalid. Based on both the Chan and Schmitt and Goldstein et al. studies, it would appear that the perception of face validity may be a critical mediating variable. Thus, we suggest that tests in the public sector should be designed or selected to look as face valid and job related as possible.

Finally, the findings of a recent study show that the use of tests tailored to assess information processing skills requisite for task performance substantially reduces adverse impact. Within a sample of undergraduate students, McKay, Barrett, Doverspike, and Randle[14] found that, among a series of predictor and criterion measures, only the cognitive ability test (Raven Advanced Progressive Matrices) resulted in a significant black/white test score difference favoring whites. However, a tailored, task-specific information processing test did not result in adverse impact against black examinees. Furthermore, the task-specific information processing task was a more valid predictor a computer simulator task performance than the Raven's (study results are available from the first author). At the present time, however, there are practical and logistical problems which would probably limit the use of information-processing-based measures for large-scale testing programs in the public sector.

The series of findings reported above support the notion that the use of testing formats other than the typical paper-and-pencil cognitive measure, or paper-and-pencil measures which look more face valid and job related, can decrease adverse impact against African-Americans. However, more research is needed to determine whether such measures can be developed without a reduction in validity. One limitation is worth mentioning concerning the evidence presented on alternative testing formats. Two of the three studies reported (i.e., Chan & Schmitt, and McKay et al.) utilized samples of undergraduate students as participants. Therefore, the generalizability of the studies' findings to actual work organizations may be questionable. Further research in applied settings is necessary to examine the degree of validity and adverse impact that results from the use of alternative testing formats.

Measure and Control Motivational Differences

In theory, it would be possible to statistically control for racial differences in test-taking motivation prior to examining the cognitive ability test scores of whites and African-Americans. That is, test results could be equalized for the effects of motivational differences using statistical procedures such as multiple regression or partial correlation. There are, however, statistical problems which must be overcome in performing such adjustments. In addition, and more importantly, not only would the legality of this procedure be questionable,[15] it would be unlikely to be met with much support from testtakers themselves.

Conclusion

Recent research suggests that African-American/white differences in cognitive ability test performance may be partially related to test-taking motivational differences between the two races. Additional research is needed to explore whether these motivational differences are real, and to identify the underlying psychological mechanisms and test variables linking these attitudes to test performance. Nevertheless, given the negative experiences that African-Americans have had historically with the testing enterprise,[16] it makes sense that African- Americans hold negative attitudes toward tests, which affects their subsequent test performances. Presently, and pending future research, the most practical solutions to increasing African-Americans' test-taking motivation and test performances seem to be (1) the development of test-taking skills training programs to enhance test-taking attitudes, and (2) further experimentation with the use of alternative testing formats and their potential effects on test performance.

Notes

[1] W.F. Cascio, Applied Psychology in Personnel Management (4th ed.), (Englewood Cliffs, NJ: Prentice-Hall. 1991).

[2] "Uniform Guidelines on Employee Selection Procedures," Federal Register, 43 (1978): pp. 38290-38309.

[3] R.D. Arvey, W. Strickland, G. Drauden, & C. Martin, "Motivational Components of Test Taking," Personnel Psychology, 43 (1990): pp. 695-716; D. Chan, N. Schmitt, R.P. DeShon,, C.S. Clause, & K. Delbridge, "Reactions to Cognitive Ability Tests: The Relationships Between Race, Test Performance, Face Validity, and Test-Taking Motivation," Journal of Applied Psychology, 82 (1997): pp. 300-310; C.M. Steele & J. Aronson, "Stereotype Threat and the Intellectual Test Performance of African-Americans," Journal of Personality and Social Psychology, 69 (1995): pp. 797-811.

[4] S.J. Gould, The Mismeasure of Man, (New York, NY: W.W. Norton, 1996).

[5] I.G. Sarason, "Stress, Anxiety, and Cognitive Interference: Reactions to Tests," Journal of Personality and Social Psychology, 46 (1984): pp. 929-938.

[6] P.F. McKay, D. Doverspike, D. Bowen-Hilton, & Q.D. Martin, "Stereotype Threat Effects on the Raven's Scores of African-Americans," Journal of Applied Social Psychology, (in press).

[7] S. Plous & T. Williams, "Racial Stereotypes from the Days of American Slavery: A Continuing Legacy," Journal of Applied Social Psychology, 25 (1995): pp. 795-817.

[8] J.P. Campbell, "Group Differences and Personnel Selection Decisions: Validity, Fairness, and Affirmative Action," Journal of Vocational Behavior, 49 (1996): pp. 122-158; M.J. Ree, J.A. Earles, & M.S. Teachout, "Predicting Job Performance: Not much more than g," Journal of Applied Psychology, 79 (1994): pp. 518-524.

[9] J.E. Hunter & R.F Hunter, "Validity and Utility of Alternative Predictors of Job Performance," Psychological Bulletin, 96 (1984): pp. 72-98.

[10] R. Hembree, "Correlates, Causes, Effects, and Treatment of Test Anxiety," Review of Educational Research, 58 (1988): pp. 47-77; I.G. Sarason, "Anxiety, Self-Preoccupation, and Attention," Anxiety Research, 1 (1988): pp. 3-7; C.D. Spielberger, W.D. Anton, & J. Bedell, "The Nature and Treatment of Test Anxiety," In M. Zuckerman & C.D. Spielberger (Eds.), Emotion and Anxiety: New Concepts, Methods, and Applications, pp. 317-345, (Lawrence Erlbaum, 1976).

[11] H.W. Goldstein, K.P. Yusko, E.P. Braverman, D.B. Smith, & B. Chung, "The Role of Cognitive Ability in the Subgroup Differences and Incremental Validity of Assessment Center Exercises," Personnel Psychology, 51 (1998): pp. 357-374.

[12] D. Chan & N. Schmitt, "Video-Based Versus Paper-and-Pencil Method of Assessment in Situational Judgment Tests: Subgroup Differences in Test Performance and Face Validity Perceptions," Journal of Applied Psychology, 82 (1997): pp. 143-159.

[13] G.V. Barrett, R.F. Miguel, & D. Doverspike, "Race Differences on a Reading Comprehension Test with and without Passages," Journal of Business and Psychology, 12 (1997): pp. 19-24.

[14] P.F. McKay, G.V. Barrett, D. Doverspike, & S.P. Randle, "Prediction of Book Purchasing Task Performance Using a Task-Specific Information Processing Test and Cognitive Ability: Black and White Differences," Unpublished manuscript, The University of Akron (1998).

[15] "Civil Rights Act of 1991," Publication L. No. 192-1066, 105 Stat. 107.

[16] H.J. Grubb, "Intelligence at the Low End of the Curve: Where are the Racial Differences?," The Journal of Black Psychology, 14 (1987): pp. 25-34; H.J. Grubb & T.H. Ollendick, "Cultural- Distance Perspective: An Exploratory Analysis of its Effect on Learning and Intelligence," International Journal of Intercultural Relations, 10 (1986): pp. 399-414; W.W. Nobles, "Psychometrics and African-American Reality: A Question of Cultural Antimony," The Negro Educational Review, 38 (1987): pp. 45-55.

Patrick F. McKay, Ph.D.
Department of Psychology
The University of North Carolina at Wilmington
601 South College Road
Wilmington, NC 28403-3297

Patrick F. McKay, Ph.D., is an Assistant Professor of Psychology at the University of North Carolina at Wilmington, Wilmington, NC. He has 6 years of consulting experience for both public and private sector employers performing such functions as personnel test development and validation, selection test administration, performance appraisal system development, training program implementation, applicant rating, and litigational support. His research interests include cognitive ability testing, alternative predictor development, test cognitions/test performance relationship, adverse impact reduction strategies, and cultural diversity training.

Dennis Doverspike, Ph.D.
Department of Psychology
The University of Akron
Akron, OH 44325-4301

Dennis Doverspike, Ph.D., ABPP, is a Full Professor of Psychology at the University of Akron and a Fellow of the Institute for Life-Span Development and Gerontology. He holds a Diplomate in Industrial/Organizational Psychology from the American Board of Professional Psychology and is a licensed psychologist in the State of Ohio. From 1982 until 1984, he was a member of the psychology and graduate faculties at the University of Nebraska at Omaha. Since 1984, he has been a member of the psychology faculty at the University of Akron. He has over 20 years of consulting experience in both the private and public sectors, and is the author of two books and over 70 refereed journal publications or book chapters.