This study sought to determine whether self-estimates of work-relevant abilities can improve upon the validity of test estimates (scores). Validity for career planning applications was the focus. The study (final sample of 1,620 college students) used 9 self-estimates and test estimates to predict
**********
As Donald Super noted over 40 years ago, "In choosing an occupation one is, in effect, choosing a means of implementing a self-concept" (1957, p. 196). What might be called "Super's Dictum" has implications for both career counseling and career development, more generally considered. Regarding the former, self-estimates provide a way to make career-relevant self-concepts evident to the counselor and counselee, and thus, open for discussion, clarification, and application. Regarding the latter, self-concepts are relevant to understanding career choice.
The general purpose of this study was to determine whether self-estimates of work-relevant abilities can improve upon the validity of test estimates (i.e., test scores) when both are used to facilitate career exploration and planning, hereafter called "career planning." The study's validity analyses addressed the use of self-estimates of work-relevant abilities in a comprehensive, work-world search for occupations with counselee-compatible work tasks, that is, in career planning "for the many." The use of self-estimates in personnel selection, college admissions, and so forth was not investigated, nor is it proposed.
In what follows, the term work-relevant abilities includes noncognitive abilities in addition to cognitive abilities, and it subsumes "basic and cross-functional skills" (Peterson, Mumford, Borman, Jeanneret, & Fleishman, 1999, p. 51). Many of these abilities are relevant to career planning, but relatively few are routinely assessed by tests. For a number of years, self-estimates have been used to address this problem. However, computer-based literature searches using PsycINFO (http://www.csa.com) and the Social Science Citation Index (http://www.webofscience.com) located only five studies (summarized below) that determined whether self-estimates can improve upon the career planning validity of test estimates. Because both self-estimates and test estimates of abilities are readily available to career counselors, research on this topic would appear to be important to the practice of career counseling.
Estimates of skill self-confidence and self-efficacy were not addressed in this study because no one appears to view them as ability estimates. For example, Betz, Borgen, and Harmon (1996) noted that the Skills Confidence Inventory "is not a measure of actual abilities" (p. 21). Regarding the Campbell Interest and Skill Survey, see Campbell, Hyne, and Nilsen (1992, p. 41). In a study of the relationship between self-efficacy beliefs and self-rated abilities, Brown, Lent, and Gore (2000) concluded that the constructs "are empirically distinguishable.... [They] may, thus, serve complementary, rather than competing, roles relative to ... career choice making" (p. 233). The computer-based literature searches revealed no studies comparing the career planning validity of skill self-confidence and self-efficacy estimates with that of test estimates of work-relevant abilities.
Rationale for the Use of Ability Self-Estimates
Prediger (1999b) discussed general considerations regarding the assessment of abilities for the purpose of facilitating career planning. Much of this section draws on that discussion.
As evidenced by common sense (e.g., Jones, 1996), theory (e.g., Holland, 1997), and research (e.g., Peterson et al., 1999; Prediger, 1989), occupations differ on a wide range of work-relevant abilities. Consider, for example, the different types of abilities needed by sales workers, machinists, commercial artists, actuaries, and career counselors. Table 1 lists the 15 work-relevant abilities used in the study reported here. Their bases in theory and research (beginning with Abe & Holland, 1965) were reported by ACT, Inc. (ACT, 2001). A primary consideration was comprehensive coverage of career clusters paralleling Holland's (1997) typology. Certainly, it would be possible to add abilities to the list. But their intended use with counselees suggested otherwise.
Unfortunately, test estimates for many work-relevant abilities are seldom available (e.g., see Abilities 8-15 in Table 1). Hence, self-estimates are needed if those abilities are to be considered in career planning. Also, the test estimates that are available may have limitations, for example, narrow operational definitions for some abilities and the time and expense involved. In contrast, ability self-estimates can systematically address a wide range of broadly defined work-relevant abilities in a relatively brief amount of time.
The various ways in which ability self-estimates are obtained are illustrated by the Self-Directed Search (Holland, 1997), the Ability Explorer (Harrington & Harrington, 1996), the O*NET Web site (http://online.onetcenter.org) Skills Search, and the Inventory of Work-Relevant Abilities (IWRA), which is used in the Career Planning Survey (ACT, 2001) and DISCOVER (ACT, 1999). Prediger (1999b) emphasized the importance of obtaining self-estimates that are informed by experience (direct or vicarious) and by feedback over the years from parents, peers, teachers, and employers. Test scores, if available, can also be considered. Prediger (1999b) noted the need to use normed scores to report self-estimates in order to address general differences in self-estimate "optimism" across abilities and to facilitate ability level comparisons across abilities.
Evidence regarding self-estimate validity is sometimes sought by correlating self-estimates with test estimates of ability. However, it is important to keep in mind that, even when the ability's name is the same for both types of estimates, they may operationally define different aspects of the ability. In addition, neither self-estimates nor test estimates are perfectly reliable, and they are subject to similar types of distortion (e.g., due to limited experience and response style). Hence, if correlations between the two types of estimates are low, both may be the problem.
Regarding self-estimate and test-estimate validity, studies of validity for career planning applications appear to be much more relevant than correlations between the two types of ability estimates. The results of five such studies (four published) were summarized by ACT (2001). These were the only such studies found in the literature searches. The studies' five samples consisted of 1st-and 2nd-year college students (one sample) and students in Grades 11, 11-12, and 12 (two samples). Sample sizes ranged from 356 to 1,620. In each of the studies, the self-estimates were obtained via an early edition of IWRA.
As documented by Prediger (1998), career planning validity is commonly determined by finding the percentage of occupational group members whose membership is predicted by an assessment's scores, that is, by finding the "hit rate." In effect, this validation model asks whether persons in a given occupational group would have been referred to that group by their scores, which is consistent with the goal of identifying personally relevant occupational options. When there are many occupations, they are often grouped by Holland's (1997) six types. Prediger (1998) also noted that occupational choice "has long been used and defended" (p. 207) as the basis for determining occupational group membership and that sometimes the choices are first screened for certainty.
In each of the five studies, ability estimates were obtained for the 15 abilities listed in Table 1, and occupational group membership was based on certainty-screened occupational choices assigned to Holland's (1997) types (see Table 1). Test estimates were available for Abilities 1-6 in Table 1. All 15 abilities listed in Table 1 were used to obtain composite ability scores for Holland's types. Each composite was based on the four abilities flagged by Holland type in Table 1. In four studies, the six ability composites were based on test estimates for Abilities 1-6 plus self-estimates for Abilities 7-15. In the fifth study, the composites were based on self-estimates for all 15 abilities.
For each study, the average occupational group hit rate for the ability composites was greater than that for the test estimates alone (medians of 42% and 34%, respectively). Thus, the composites had greater validity for career planning applications. The hit rates for the composites were similar to those typically reported in interest inventory validation studies using Holland-type occupational groups (e.g., see Swaney, 1995, for a summary).
One of the studies (Prediger & Swaney, 1992) also compared the validity of self-estimates and test estimates for Abilities 1-6 in Table 1. The untimed self-estimates required about 2 minutes; the test estimate time limits totaled 75 minutes. The initial sample consisted of 529 Grade 11 students from three schools. The ability self-estimates were obtained prior to the test estimates, which were provided by the ability tests used in the study reported here. Students in the final sample (N = 356) were assigned to Holland-type occupational groups on the basis of certainty-screened occupational choice. The self-estimate and test-estimate hit rates for the occupational groups were 41% and 39%, respectively. Hence, results favored the self-estimates. The argument that results of the five studies were obtained because self-estimates of work-relevant abilities simply reflected interests is not supported by the results of correlational analyses and independent validity analyses summarized by Prediger (1999b).
Although several published instruments obtain ability self-estimates, the above studies reporting the career planning validity of test estimates and self-estimates of ability appear to be unique. However, there have been a number of studies comparing the validity of ability self-estimates and test estimates as predictors of school and college grades. In a yet-to-be-replicated review of such studies, Baird (1976) found conclusive evidence favoring self-estimates.
The purpose of the study reported in this article was to compare the career planning validity of test estimates of six abilities typically included in ability test batteries with the validity of the six test estimates plus self-estimates for nine additional abilities. In effect, the study determined whether adding self-estimates of abilities to test estimates resulted in greater validity, that is, higher hit rates for occupational groups. The occupational groups consisted of 1,620 college students assigned to Holland (1997) types on the basis of certainty-screened occupational choice. Study analyses extend those cited above in that the previous analyses involved ability composites rather than the full set of 15 work-relevant abilities listed in Table 1.
Method
Sample
The initial sample consisted of 7,008 1st-and 2nd-year students (66% female, median age of 28) attending 14 colleges (12 were community colleges) in 8 primarily midwestern states. All had completed the ability tests and ability self-estimates in the Career Planning Program (CPP; American College Testing, 1988). Because community colleges have open admissions, they typically use the CPP in orientation programs, career planning courses, workshops, and career centers. Hence, it is reasonable to assume that students were motivated to do well on the CPP tests. It was also in their interest to be insightful and open when they provided self-estimates. Across the six CPP tests, the mean stanine scores of sample members ranged from 4.6 to 5.4. The Grade 12 national norm group has a mean of 5.0.
Prior to completing any of the CPP ability estimates, students were asked to scan a list of 140 occupational titles (ACT, 1989) arranged according to 23 job families. Then they were asked to identify the occupation that came the closest to the one they were considering. Finally, they were asked the following question: "How sure are you that the occupational choice you selected ... will still be your first choice one year from now?" (p. 2). Response options for the question were very sure, fairly sure, and not sure. Students choosing either of the last two responses were excluded from the study analyses. The remaining 1,620 students (the final sample) were assigned to Holland-type groups on the basis of occupational choice. Their distribution across Holland (1997) types was as follows: R (21%), I (21%), A (13%), S (21%), E (11%), and C (14%).
Variables
Test estimates of ability. The CPP obtains test estimates for Abilities 1-6 in Table 1. The number of items and time limits for the tests are as follows: Reading (40, 20 minutes), Numerical (32, 18 minutes), Language Usage (64, 11 minutes), Spatial Perception (35, 9 minutes), Clerical (35, 5 minutes), and Mechanical Reasoning (30, 12 minutes). These time limits total to 75 minutes. Test content and psychometric support were described by ACT (1988).
Internal consistency reliability coefficients for five of the CPP tests (Clerical excluded) ranged from .81 to .91 (median of .89) for a sample of 2,238 12th graders. The test-retest reliability coefficient (2-week interval) for the speeded Clerical Test was .74 for a sample of 590 11th graders.
Two divergent types of validity data are available for the CPP tests: (a) correlations between the tests and high school grades, and (b) the extent to which mean score profiles show expected differences among persons pursuing various occupations. Four samples of 9th graders (N = 932) and four samples of 11th graders (N = 913) completed the CPP in late winter. Across the eight samples, the median correlation between the CPP tests and end-of-year grade-point-average was .42. For a composite score based on the three academic ability tests, the median correlation was .63.
Mean score profiles for the six CPP tests were obtained for 2,021 persons who had been grouped into 15 job families on the basis of the occupations they were pursuing 6 years after completing the tests. Generally, the profiles were in line with expectations. For example, persons in the Clerical and Secretarial Work job family scored highest on the Language Usage Test and the Clerical Test, whereas persons in the Construction and Maintenance job family scored highest on the Mechanical Reasoning Test and the Spatial Perception Test. Taken together, the two types of validity evidence appear to provide good support for the CPP tests.
Self-estimates of ability. As part of the CPP, students provided self-estimates for Abilities 7-15 in Table 1. First, they read the description of an ability. For example, "sales ability" was described as "Influencing people to buy a product, service, or take a suggested course of action"; "manual dexterity" was described as "Making or repairing things easily and quickly with one's hands" (ACT, 1989, p. 9). For each of the abilities, students then indicated whether they considered themselves to rank in the lower 25%, middle 50%, or upper 25% compared to persons their own age. These response categories were assigned scores of 1, 2, and 3 in the analysis. The untimed completion of the self-estimates required about 2 minutes.
Because the nine ability self-estimates are not used, separately, for career planning, reliability data are not available. In any case, validity is not possible without reliability. Validity data were provided by the five studies cited above; each used self-estimates for the same abilities. A different type of validity data is provided by the self-estimate mean scores of occupational choice groups. As reported by ACT (2001), Grade 12 students in two samples (Ns = 1,049 and 1,503) provided self-estimates for all 15 IWRA abilities. The self-estimates were used to obtain ability composites for Holland (1997) types. Holland-type occupational group membership was determined on the basis of occupational choice screened on certainty. In both studies, all six occupational groups scored highest on the ability composite corresponding to the group's Holland type. Finally, Prediger (1999a) found that the factor structure underlying self-estimates for all 15 abilities (two samples) was similar to the Data/Ideas and Things/People Work Task Dimensions underlying the hexagonal arrangement of Holland's (1997) types. Thus, relevance of the abilities for Holland's types was empirically demonstrated.
Analyses
Discriminant analysis (Norusis, 1990) was used to determine occupational group (Holland type) differentiation and to obtain predictions of group membership (hence, hit rates). The group differentiation achieved by the ability estimates was expressed in terms of Wilks's lambda and the more usual "percent of variance accounted for" (VAF), which is one minus lambda. Bayes's rule (Norusis, 1990) was used to obtain predictions of occupational group membership (hence, hit rates) based on the ability estimates.
Discriminant analyses were conducted separately for the test estimates of Abilities 1-6 and for test estimates of Abilities 1-6 together with self-estimates of Abilities 7-15 (a total of 15 abilities). Both analyses were conducted on two random subsamples (1 and 2) stratified by college, educational plans, gender, and Holland-type of occupational choice. A cross-validation design was used with Subsamples 1 and 2. That is, Subsample 1 was used to develop equations to predict occupational group membership, and those equations were used to predict group membership for Subsample 2. Then, equations based on Subsample 2 were used to predict group membership for Subsample 1. This design controls for hit rate inflation, which is crucial when the validities of 6 and 15 predictors are being compared.
Results
Table 2 reports results averaged across the two subsamples. The results were nearly identical for the two subsamples (see Table 2 Note). The Wilks's lambdas for both sets of predictors were highly significant, statistically. The VAF for test estimates plus self-estimates (49%) was substantially higher than the VAF for test estimates alone (23%). The cross-validated occupational group hit rates were 37% and 28%, respectively. Thus, the addition of self-estimates to test estimates resulted in a 32% proportional improvement [(37%-28%)/28%] in the hit rate. The 15-variable hit rate is similar to those typically reported in interest inventory validation studies (e.g., see Swaney, 1995, for a summary).
The Rationale section noted that, in a study comparing the career planning validity of self-estimates and test estimates for the same six abilities (Abilities 1-6 in Table 1), the results favored self-estimates. Because the CPP provided the test estimates, those results suggest that, had self-estimates replaced the test estimates for Abilities 1-6 in the study reported here, the hit rates for the 15 abilities would have been at least as high. By implication, the ability test battery would have made little, if any, contribution to career planning validity. In any case, study results show that assessing a broad range of work-relevant abilities via briefly administered self-estimates can substantially add to the career planning validity of the abilities typically assessed by tests.
In order to obtain perspective on which of the 15 work-relevant abilities did the best job in differentiating Holland types, univariate Fratios (Norusis, 1990) for the abilities were examined. When the Fs were averaged across the two subsamples, the top five abilities were as follows: Manual Dexterity, Mechanical Reasoning, Helping Others, Scientific, and Creative-Artistic. The scores for only one of these abilities (Mechanical Reasoning) were based on a test estimate. In contrast, three of the six test estimates of ability ranked among the bottom five (Numerical, Clerical, and Reading). As on the typical paper-and-pencil test, Clerical was narrowly defined. The other two abilities would appear to be relevant to occupations in all of Holland's (1997) six types. Hence, lack of differentiation would be expected. Two of the nine ability self-estimates (Organization and Leadership) ranked among the bottom five, possibly because they mainly apply to the Enterprising type.
Discussion and Implications
The wide range of work-relevant abilities, the fact that many of those abilities are not easily assessed by tests, the research cited in the Rationale section, and the research reported here all support supplementing test estimates of ability with self-estimates when abilities are used in a comprehensive search for career options. Research results also suggest that, when self-estimates of abilities are available, test estimates may have little of practical significance to contribute to career planning for the many. Of course, test estimates can have a role in counseling individuals with certain types of career-related concerns (e.g., see Prediger, 1999b, for a case study).
In a review of research on self-assessment, more generally considered, Osberg and Shrauger (1990) noted that a "consistent trend among leading assessment theorists has been the call for greater use of self-assessment data" (p. 97). They believed the research indicates that "greater use of a person's own self-knowledge would contribute significant new data to the field of psychological assessment" (p. 97). Regarding ability self-estimates, Goldman (1972) and Prediger (1999b) discussed the psychometric inadequacies of ability tests when used for career planning. In an article titled "Tests and Counseling: The Marriage That Failed," Goldman (1972) noted that "The main contribution of tests in counseling is not making predictions but facilitating the clarification of self-concept" (p. 219). Of course, ability self-estimates are self-concepts. Healy (1990) advocated the use of self-estimates, but he noted that "Appraisals continue to cast counselors as experts and rely on traits for connecting clients and career options.... [This] prompts dependence on tests and inventories" (p. 216).
Counselors who wish to use ability self-estimates in career exploration and planning may want to draw on one of the instruments cited in the Rationale section. All are available via paper-and-pencil assessments and computer. Of course, counselors can also develop their own paper-and-pencil assessments. In either case, it would appear to be important to obtain self-estimates that are informed by experience and by feedback from others. Also, the self-estimates need to be linked to career options.
With respect to IWRA, scores on Holland's (1997) six types are linked to career options via the empirically based World-of-Work Map (WWM), an extension of Holland's hexagon. The recently updated WWM, its 26 career clusters (with examples of typical occupations), and a table for converting scores for Holland's types into "map regions" were provided by Prediger (2002). A brief case study illustrates how the WWM can be used in career planning. (Copies of the WWM, its career clusters, and the table are available from the author.) When time is available, counselors can obtain counselee reactions to their self-estimates and discuss their bases. As a result, self-concepts may be clarified (recall Super's Dictum).
The wide range of work-relevant abilities and limitations of the research reported here certainly appear to warrant further research comparing the career planning validity of self-estimates and test estimates. The limitations include the use of an early form of IWRA. The current form (ACT, 1999, 2001) supplements each of the ability descriptions with examples of activities likely to have been experienced directly or vicariously by persons completing IWRA. Also, only 15 of the many work-relevant abilities were addressed by the study and those cited in the Rationale section. Self-estimates were available for only nine of them, and test estimates were available for only six. Validity studies that include a wider range of ability estimates and different assessment instruments could provide a new perspective on abilities relevant to career counseling "for the many," the context of this study.
TABLE 1 Assignments of 15 Work-Relevant Abilities to Career Clusters
Similar to Holland (1997) Types
Career Cluster and Related Holland Type
Social Business
Technical Science Arts Service Contact
Ability (R) (I) (A) (S) (E)
Abilities for which test estimates are usually available
1. Reading X X
2. Numerical X X
3. Language
usage X X X
4. Spatial
perception X X X
5. Clerical
6. Mechanical
reasoning X
7. Scientific X
Abilities for which test estimates are seldom available
8. Creative/
literary X
9. Creative/
artistic X
10. Manual
dexterity X
11. Meeting
people X
12. Helping
others X
13. Sales X
14. Leadership X
15. Organization X
Career Cluster and Related Holland Type
Business
Operations
Ability (C)
Abilities for which test estimates are usually available
1. Reading
2. Numerical X
3. Language
usage X
4. Spatial
perception
5. Clerical X
6. Mechanical
reasoning
7. Scientific
8. Creative/
literary
9. Creative/
artistic
10. Manual
dexterity
11. Meeting
people
12. Helping
others
13. Sales
14. Leadership
15. Organization X
Note. R = Realistic; I = Investigative; A = Artistic; S = Social; E =
Enterprising; C = Conventional. An X indicates abilities associated with
a career cluster. Adapted from ACT (2001, p. 19) with permission.
TABLE 2 Validities of Six Ability Tests and Tests Plus Self-Estimates
for Nine Additional Abilities
Tests + Self-
Statistics Tests Estimates
Multivariate analysis of variance
Wilks's lambda .77* .51*
Percentage variance accounted for 23 49
Percentage overall hit rate (a) 28 37
Note. These double cross-validation analyses involved certainty-screened
occupational choice groups assigned to Holland (1997) type. Results were
averaged for the two subsamples. The subsample results were nearly
identical. For example, hit rates for the tests were 27.9% and 27.8%.
For tests plus self-estimates, the hit rates were 37.4% and 36.2%.
(a) Chance = 17%.
* p < .0001.
References
Abe, C., & Holland, J. L. (1965). A description of college freshmen: II. Students with different vocational choices (ACT Research Report No. 4). Iowa City, IA: American College Testing.
ACT. (1999). Research support for DISCOVER assessment components. Iowa City, IA: Author.
ACT. (2001). Career Planning Survey technical manual. Iowa City, IA: Author.
American College Testing. (1988). Interim psychometric handbook for the 3rd edition ACT Career Planning Program. Iowa City, IA: Author.
American College Testing. (1989). Career Planning Program Level 2 assessment booklet. Iowa City, IA: Author.
Baird, L. (1976). Using self-reports to predict student performance (Research Monograph No. 7). New York: College Entrance Examination Board.
Betz, N. E., Borgen, F. H., & Harmon, L. W. (1996). Skills Confidence Inventory: Applications and technical guide. Palo Alto, CA: Consulting Psychologists Press.
Brown, S. D., Lent, R. W., & Gore, P. A. (2000). Self-rated abilities and self-efficacy beliefs: Are they empirically distinct? Journal of Career Assessment, 8, 223-235.
Campbell, D. P., Hyne, S. A., & Nilsen, D. L. (1992). Manual for the Campbell Interest and Skill Survey. Minneapolis, MN: National Computer Systems.
Goldman, L. (1972). Tests and counseling: The marriage that failed. Measurement and Evaluation in Guidance, 4, 213-220.
Harrington, J. C., & Harrington, T. F. (1996). Ability Explorer: Level 2. Itasca, IL: Riverside Publishing.
Healy, C. C. (1990). Reforming career appraisals to meet the needs of clients in the 1990s. The Counseling Psychologist, 18, 214-226.
Holland, J. L. (1997). Making vocational choices (3rd ed.). Odessa, FL: Psychological Assessment Resources.
Jones, L. K. (1996). Job skills for the 21st century. Phoenix, AZ: Oryx Press.
Norusis, M. J. (1990). SPSS advanced statistics user's guide. Chicago: SPSS.
Osberg, T. M., & Shrauger, J. S. (1990). The role of self-prediction in psychological assessment. In J. N. Butcher & C. D. Spielberger (Eds.), Advances in personality assessment: Vol. 8 (pp. 97-120). Hillsdale, NJ: Erlbaum.
Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., & Fleishman, E. A. (1999). An occupational information system for the 21st century: The development of O* NET. Washington, DC: American Psychological Association.
Prediger, D. J. (1989). Ability differences across occupations: More than g. Journal of Vocational Behavior, 34, 1-27.
Prediger, D. J. (1998). Is interest profile level relevant to career counseling? Journal of Counseling Psychology, 45, 204-211.
Prediger, D. J. (1999a). Basic structure of work-relevant abilities. Journal of Counseling Psychology, 46, 178-184.
Prediger, D. J. (1999b). Integrating interests and abilities for career exploration: General considerations. In M. L. Savickas & A. R. Spokane (Eds.), Vocational interests: Their meaning, measurement, and counseling use (pp. 295-325). Palo Alto, CA: Davies-Black.
Prediger, D. J. (2002). Abilities, interests, and values: Their assessment and integration via the World-of-Work Map. Journal of Career Assessment, 10, 209-232.
Prediger, D. J., & Swaney, K. B. (1992). Career counseling validity of DISCOVER's job cluster scales for the revised ASVAB score report (ACT Research Report No. 92-2). Iowa City, IA: ACT.
Super, D. E. (1957). The psychology of careers. New York: Harper.
Swaney, K. B. (1995). Technical manual: Revised unisex edition of the ACT Interest Inventory (UNIACT). Iowa City, IA: ACT.
Dale J. Prediger, Iowa City, Iowa. This article was written before the author retired as a vocational research psychologist from ACT, Inc. (ACT). The author thanks Kyle Swaney, an ACT Research Division staff member, for his review of a draft of this article. The study involved a secondary analysis of a database in ACT files. An unpublished study involved analyses of other variables in the database. Results were briefly reported by ACT (2001). Correspondence concerning this article should be addressed to Dale J. Prediger, 444 Mullin Avenue, Iowa City, IA 52246 (e-mail: prediger@inav.net).