Effects of subject matter expert viewpoint on job analysis results.

This study explored the effects of subject matter expert (SME) viewpoint (incumbents/supervisors versus users of work products) on job analysis data in a law enforcement context. As part of a job analysis to develop a content valid test and training program for police report writing skills,

assistant district attorneys and police officers were asked to evaluate the criticality of 16 dimensions of police report writing, incorporating two subscales, grammar and content. Analyses indicated that police officers and district attorneys had similar ratings for items related to report content, but police officers rated grammar as more critical than did district attorneys. District attorneys in this case were more reliable in their ratings. Implications for job analysis, including considerations in choosing SMEs and the utilization of SME data, are discussed.

The importance of job analysis to human resource management cannot be overstated, having been described as "the fundamental building block upon which all later decisions in the employment process must rest." (1) This process is a key activity for other human resource functions such as personnel selection, training, performance appraisal and compensa-tion.

As noted by Landy and Vasey, (2) the Uniform Guidelines on Employee Selection Procedures (3) discusses the use of content validation approaches in personnel selection. In doing so, the importance of job analysis has been underscored, as it forms the basis for drawing inferences between job activi-ties and test content. In addition, the Society for Industrial and Organizational Psychology also addressed content validation issues in the Principles for the Validation and Use of Personnel Selection Procedures, indicating that content validation approaches were acceptable when the abilities assessed are clearly defined. (4)

In conducting a job analysis that demonstrates content validity, the selection of subject matter experts (SMEs) to participate in the process is critical. Goldstein, Zedeck and Schneider suggest that different SME types may be better at providing certain types of information than others. (5) For example, supervisors may be better at providing KSA information, while job incumbents may be more adept at identifying the tasks associated with the job. (6)

However, relatively few studies have focused on the selection of appropriate SMEs for job analysis. The purpose of the present study was to explore potential differences between two samples of SMEs--police officers and district attorneys--in terms of their mean ratings of job analysis facets and their ordering of these facets. These differences were explored within the context of a focused job analysis examining critical aspects of police report-writing skills. We expected that such differences might occur as a result of two self-presentation processes identified in the literature as having an effect on job analysis data: Impression management and demand effects. (7)

Although incumbents and supervisors are the primary SMEs in job analysis, district attorneys (DAs) were also included in this process. As the end-users of police reports, DAs can provide valuable input in the development of a test of report-writing skills, as a criterion of good report writing should be to accurately document details of the crime scene and information provided by witnesses. This, in turn, should lead to convictions and/or charges withstanding court scrutiny. Given that both DAs and police are familiar with police reports, we compared DAs and police in terms of the factors each felt were essential to good report writing.

Selecting SME Samples

An accepted practice in job analysis is to obtain a stratified, random sample of SMEs when collecting job analysis data because, presum-ably, there are systematic differences between the job analysis data collected from various demographic groups. Furthermore, SME demographics have been a consideration in job analysis to address EEO and issues of perceived fairness. SME demographic variables that have received attention in the literature include seniority, (8) education level, (9) race, (10) and job performance level. (11) On the whole, however, the demographic makeup of SME samples has been reported to have little effect on job analy-sis results. Indeed, Schmitt and Cohen concluded that race, gender and tenure should be ad-dressed in job analysis more for social or political reasons than for practical ones. (12)

Customer as SME

Perhaps a more fruitful approach to this problem would be to focus on the customer of the products or services of the job in question. Many quality management approaches stress the importance of the customer in organization-al opera-tions. (13) Total Quality Management (TQM) approaches focus on improving service to customers and meeting customer needs. Moreover, TQM has been applied to the public sector. (14) Improving the quality of police reports can be conceptualized as improving service to DAs, the "consumers" of police reports. Focusing on the customer of the job product in choosing SME samples could direct the job analysis toward those aspects of the job that are actually most necessary or important, rather than on how the job is done by certain incumbents. There-fore, a more useful approach to job analysis for certain types of jobs could be to focus on who ultimately uses the work products of the job. Such a method would concentrate on the SME's relationship to the job.

Research on responses of supervisor versus subordinate SMEs might be relevant to this issue, since these represent two groups with differing viewpoints of the job. In this area, some differences in the responses of SME types have been found. In an early study, Meyer (15) found disagree-ment between foremen and general foremen about the foreman's job responsibili-ties. Hazel, Madden and Christal (16) compared the job profiles given by supervi-sors and subordinates for Air Force positions. They found that while respon-dents agreed on broad job duties, there was disagreement on specific tasks. O'Reilly (17) found that supervisors and subordinates disagreed on both the tasks performed on the job and the level of skills and knowledge necessary to perform those tasks. Though Smith and Hakel (18) found similari-ties in responses among various SMEs (including supervisors and subordinates), the lack of differences were later found to be largely due to methodologi-cal factors (19). Finally, Mueller and Belcher (20) investigated the differences in task and KSA criticality ratings for fire captains and their supervisors in analyzing the fire captain job. Overall, there was moderate agreement in the ordering of tasks and KSAs. However, most of this agreement was in terms of the least important aspects of the job.

In summary, the viewpoint of the respondent in terms of supervisor and incumbent has been found to affect job analysis data. Accord-ing-ly, the importance of certain job dimensions to SMEs has been found to vary as a function of their relationship to the job being analyzed. However, if one assumes that the impor-tance of job dimensions is best determined by the ultimate customers of a job's work products, then differences found between the job analysis data of such customers and that of other SMEs (e.g., incumbents) take on great signifi-cance.

Furthermore, incorporating the viewpoints of customers of job incumbents' work helps to better incorporate organizational goals into the job analysis effort. Schippmann et al. (21) suggest that improvements to job analysis can be made by looking at competency modeling approaches. An area in which the typical job analysis effort is usually lacking compared to competency modeling is the extent to which the job analysis results are linked to organizational goals (22). Therefore, our involvement of DAs in the current study helps to account for a deficiency that is typical in most job analysis processes.

Hypotheses

Morgeson and Campion identified a number of potential issues to consider in conducting job analyses. (23) In general, there may be social and cognitive processes that have the potential to negatively influence the quality of job analysis data. Such effects include biased estimates of task criticality and less reliable data. Moreover, these effects are likely to vary depending on factors such as the method of job analysis, data collection techniques, data sources (e.g., incumbents vs. supervisors), and purpose of the job analysis (e.g., compensation, selection). These biases can have far-reaching effects, given the fundamental nature of job analysis. (24) Clearly, different SME groups may exhibit varying degrees of these biases.

The 16 facets of report writing assessed in the present study consisted of two overarching factors: Grammar (e.g., spelling, punctuation) and content (e.g., accuracy in describing a sequence of events). Based on Morgeson and Campion's (25) review, we identified self-presentation factors, specifically impression management and demand effects, as potential negative influences on police officer criticality ratings. Impression management may underlie police ratings because police will want to present themselves in a favorable light, demonstrating that they are very knowledgeable about aspects of report writing that are crucial to the development of a court case. Also, demand effects may affect police criticality ratings because police may believe that the dimensions presented to them in the criticality survey must be important, based on the job analyst's inclusion of those specific dimensions. Third, DAs had requested that report writing be examined and that aspects of reports that would lead to greater chances of convicting criminals be explored. We therefore believed that there was greater potential on the part of police officers to exhibit some form of impression management and/or demand effects when making their criticality ratings, as they are the ones who generate these reports. These self-presentation processes may lead police officers to consider all aspects of police report writing as highly critical, which may lead to such influences on the data as inflated mean criticality ratings. (26) DAs should not exhibit this same effect because they are not the ones under scrutiny, and should focus instead on what they perceive to be the true criticality of report-writing dimensions.

   Hypothesis 1: Occupation (police or DA) will moderate job analysis
   criticality ratings of grammar and content dimensions.
   Specifically, due to impression management and demand
   characteristics, police will rate both report-writing subscales
   as critical. DAs will rate content as more critical to report
   writing than grammar.

As indicated in Morgeson and Campion's (27) review, self-presentation biases may have an effect on the reliability of the data. However, it is unclear exactly in what way reliability will be affected when comparing police and DAs. Therefore, we propose the following research question:

Research Question: Will the two SME groups in this study differ in terms of the reliability of their job analysis ratings?

Method

Overview

This research was conducted as part of a larger project to develop a test of written communication for the job of police officer to assess incoming academy candidates. In order for it to be content valid, its development relied heavily on a highly detailed, focused job analysis. In conducting the job analysis, police personnel (field training officers, supervisors) and assistant DAs were used as SMEs because these groups are the most likely to know about the critical dimensions of police report writing.

Thus, the two types of SMEs compared in this study were police personnel, who are most familiar with the process of police report writing; and DAS, who must ultimately use the reports to develop and prosecute court cases, one of the overriding goals of writing police reports. Responses to the criticality survey were compared both across SME groups (in terms of differences in mean ratings and in agreement in ordering of facets) and within SME groups (in terms of interrater reliability).

Participants

A total of 105 participants were part of this study. Participants consisted of 72 training officers and academy personnel from a southern, metropolitan police department and 33 DAs from the same city. Questionnaires were distributed to 75 DAs and produced a response rate of 44 percent. All police questionnaires were returned. Complete data was obtained for a total of 101 of the participants.

Procedure

Writing skills survey development. As part of a project to develop a content valid test of police written communication, a questionnaire was constructed to determine which aspects of police report writing to include on the exam. To generate a list of potential written communication dimensions, the human resource department's previous job analysis for the job class of police recruit/police officer was reviewed for those dimensions of written communication used in police work. Other tests of written communication were consulted for standard measurement criteria (e.g., grammar, spelling). Meetings with DAs and police were held to incorporate their views of the key dimensions of report writing. The resulting 16 dimensions of written communication were incorporated into a questionnaire (See Appendix). They were grouped into two subscales. The first subscale, grammar, consisted of eight items that dealt with the more technical aspects of report writing (e.g., "Spelling of names, addresses, and important words."). The second subscale, content, consisted of eight items that focused more on the quality of the information in the report (e.g., "Accuracy in describing a particular scene, situation or sequence of events").

Criticality measure. For each of the 16 dimensions, both SME groups were asked, "How necessary is this area of written communication to ensure that a case stands up in court?" This question was developed to reflect the primary purpose of police reports and to focus SMEs on this purpose in assessing the criticality of the dimensions. Responses to this question were made on a five-point scale, with 1 indicating the dimension was not very important or essential, and 5 indicating the dimension was very important or essential.

Results

Means and standard deviations for the responses to the criticality measure for both SME groups by item are presented in Table 1. Hypothesis 1, that occupation (police or DA) would moderate the criticality ratings of grammar and content job analysis subscales, was tested using a 2 (Occupation) X 2 (Job Analysis Dimension) mixed ANOVA. Results are displayed in Table 2. Although main effects were found for both criticality dimension type and for occupation, these main effects were qualified by a significant interaction. Specifically, in support of Hypothesis 1, occupation moderated criticality ratings for the two report writing subscales, F (1, 99) = 90.24, p < .01, [[eta].sup.2] = .48. This interaction is depicted graphically in Figure 1. As expected, the two groups were significantly different in their mean ratings of the grammar subscale. Police rated the grammar subscale as more important than did DAs (Ms = 4.66 and 3.33, respectively). Police and DAs did not differ in their mean ratings of the content subscale (Ms = 4.90 and 4.72, respectively).

[FIGURE 1 OMITTED]

To explore the research question--whether police and DAs differ in terms of the reliability of their ordering of the items in the criticality survey--an intraclass correlation (in terms of a generalizability coefficient (28)) was computed for each occupation. In this case, the generalizability coefficient represents an index of the agreement of participants in their relative ordering of the criticality of the report-writing facets. Similar approaches have been used for estimating the dependability of cut score estimates. (29) For this analysis, all 16 dimensions of police report writing were examined simultaneously.

The results of the generalizability analysis indicated that both police officers and DAs were very reliable in their ratings of the 16 criticality items ([[rho].sup.2] = .91 for police; [[rho].sup.2] = .97 for DAs). However, a problem in making a direct comparison between these two groups of SMEs lies in the disparate sample sizes, since the larger police sample should lead to a larger intraclass correlation, similar to the way in which the addition of more items to a scale tends to lead to larger estimates of coefficient alpha. (30) Therefore, to better compare the level of reliability exhibited between the two SME groups, we calculated the number of SMEs needed to achieve a [[rho].sup.2] of .80, which is considered a minimally acceptable intraclass correlation. (31) To do this, we took the general form of the generalizability analysis equation (32) and used a form of the Spearman-Brown prophecy formula to determine the number of police versus DAs needed to obtain a [[rho].sup.2] of .80.

Our calculations indicated that to obtain a [[rho].sup.2] of .80, a sample of 29 police officers would be required. However, only four DAs would be required for this level of reliability. Thus, in terms of the Research Question, it appears that there is a difference in the reliability of the data from police officers and DAs. Specifically, DAs were more consistent in their ordering of the items in the police report writing criticality survey than were police officers.

Discussion

It is accepted in personnel research that different types of SMEs should be consulted in collecting job analysis data. (33) These SMEs typically include those from different demographic backgrounds, from different parts of the organization, and both incumbents and supervisors. Therefore, those who depend on the work product of the job such as internal or external customers may also have unique information to contribute about the job in addition to that contributed by incumbents or supervisors.

The present study has focused on the viewpoint of the SME in terms of those who use the job products versus those who supervise or perform the job. This preliminary research has shown differences between the job analysis data of these two SME types as well as differences in the reliability of their responses. Differing perceptions between SME types were found for the two subscales of report writing: Grammar and content. Specifically, police ratings of grammar items were substantially higher than the ratings made by DAs. Ratings for content items were similar for the two groups. All of these differences between DAs and police SMEs were such that police SMEs indicated that the subscale was more necessary than DAs did.

Police may have a more global view of the importance of report writing, believing all dimensions (including, for example, grammar and spelling) are important. In contrast, DAs, with their experience in screening cases and in the courtroom, may have a more complex view of what is and is not important to support court cases and prosecutions. Another potential explanation for the differences between police and DAs is the influence of self-presentation processes, specifically, impression management and demand effects. (34) That is, given that DAs requested an examination of report writing, police officers' work was under close scrutiny. Accordingly, police officers may have attempted to portray themselves as good, conscientious report writers (impression management), and therefore considered all facets of report writing presented to them as being very critical to producing high-quality reports. Or, equally plausible, demand effects may have been at play. (35) Given the police report-writing facets presented to them, police officers may have determined that job analysts had identified these dimensions as ones that are important to quality police reports, and therefore considered all facets to be critical. DAs, on the other hand, were most likely more discerning in their evaluation of the facets of report writing, as their perspective focused them more on the use of police reports rather than producing those reports.

An additional source of differences probably resides in the focus of police SMEs on general acceptability of the reports or the general impression of the department the reports give to outsiders, versus DAs' focus on critical dimensions relevant to prosecution. Such differences could indicate the need to consider the familiarity of the SME with standards associated with particular job dimensions (not just with the job as a whole) in choosing SMEs, since those with less familiarity may rely more on global impressions in making their judgments.

Implications for Practice

One practical implication of these findings is that those who use the work products of a particular job may have somewhat different job analysis responses from those who perform the job. The ramifications of this phenomenon may be significant in certain situations. In this case, for example, DAs rated two of the 16 dimensions below the midpoint of the scale (a frequently used cutoff for retaining dimensions), while police personnel did not. Thus, the use of the different SME groups would have produced different job analysis results. Moreover, the intraclass correlations suggest that the job analysis ratings of DAs were more reliable than those of police.

The consideration of the customer of organizational products and services is central to many quality management approaches. Involving users of a job's work products in relevant areas of a job analysis could have profound effects on other human resource functions, especially training. In training needs analysis, for example, the opinions of those who use or depend on work products of the job would be important to integrating the needs analysis with broader organizational goals and strategies. Additionally, the focus of the needs analysis would then be more on what workers should be doing rather than on what they are doing. Similarly, in developing performance evaluation systems, a focus on a broader perspective in job analysis could lead to performance standards better integrated with the goals of the total organization. And for personnel selection, developing tests that are most relevant to the organization's overall purpose might be accomplished through such a job analysis approach. The incorporation of competency modeling concepts into a task/KSA analysis could aid in a better integration of organizational goals (36), one of which in the present situation would be to aid DAs in prosecuting court cases.

These results underscore the importance of asking multiple SME groups to provide job analysis data beyond the standard recommendation of representatively sampling incumbents across a wide array of demographic and work characteristics. (37) In the present case, it was especially important to incorporate the customers of the job product (i.e., DAs), as they have expert knowledge regarding the utility of the product produced by the incumbent. Police officers may be better at outlining what goes into report writing, while DAs may be better at identifying aspects of report writing that are essential to making the reports more effective for their intended purpose. (38)

Finally, we must note that these issues are most relevant when jobs are interdependent. That is, the consideration of customers of work products in job analysis is only relevant when some aspect of the job has a major impact on the user. Accordingly, users would best be consulted about those areas of the job with which they are most familiar and which affect them directly, as in the present case.

Limitations and Future Research

Because of the limitations of sample size and organizational character, future research should confirm these findings using other jobs and performance areas to determine the pervasiveness of this phenomenon. Additionally, these issues should be studied using other measures of criticality (e.g., importance, necessary at time of hire, time spent), and the phenomenon should be investigated for various types of job analysis methods (e.g., questionnaires, critical incidents) and elements (e.g., tasks and KSAs). Further, our proposition that self-presentation processes may have influenced the data provided by the job incumbents (i.e., police) requires further exploration, including, for example, direct measurement of impression management. Finally, the issue of which job analysis source should be given greater weight (user or incumbent) requires further consideration. In this case, for example, because DAs are considered to be the primary users of police reports in their role as prosecutors, one might weight their views of performance standards more than those of police personnel.

However, if the focus were on the frequency ratings of tasks, incumbents and their supervisors might be given greater deference as SMEs.

It is a truism in job analysis that the method used should be suited to the purpose for which the information is collected. These results indicate that the SME's perspective may also be relevant to the purpose of the job analysis. If this phenomenon is found to be widespread, its practical implications should be studied in terms of its real effects on job analysis results and the personnel decisions based on them.

Appendix

Dimensions of Police Written Communication

1. Vocabulary--proper use of vocabulary in a certain sentence or context.

2. Spelling of names, addresses and important words.

3. Punctuation that enhances understanding of sentences.

4. Grammar (e.g., subject/verb agreement).

5. Logical sentence structure.

6. Combining shorter sentences/clauses into one logical sentence.

7. Accuracy in describing a particular scene, situation or sequence of events.

8. Logical flow in narrative.

9. Accurately relating what was stated by a particular witness.

10. Separating facts observed by witnesses from those that witnesses heard.

11. Attributing statements to the correct witness and not to others (so that identification of source is clear and unambiguous).

12. Including sufficient detail in narrative about facts of case (e.g., names of victim, perpetrator, witnesses, addresses, exact location of crime, precise sequence of events).

13. Including only facts in narrative (i.e., does not make assumptions about facts).

14. Provides description of evidence (e.g., observed events, objects, appearances) upon which inference or conclusion is based.

15. Provides description of perpetrator (e.g., appearance, statements made).

16. Report contains no superfluous information or words that could detract from the meaning of the narrative.

Notes

(1) Casio, W. F. (1987). Applied Psychology in Personnel Management. (3rd ed.). Englewood Cliffs, NJ: Prentice Hall. p. 42.

(2) Landy, F. J., & Vasey, J. (1991). Job Analysis: The Composition of SME Samples. Personnel Psychology, 44, 27-50.

(3) Uniform Guidelines on Employee Selection Procedures. (1978). Federal Register, 43, 38290-38309.

(4) Society for Industrial and Organizational Psychology. (1987). Principles for the Validation and Use of Personnel Selection Procedures (3rd ed.). College Park, MD: Author.

(5) Goldstein, I. L., Zedeck, S., & Schneider, B. (1993). An Exploration of the Job Analysis-Content Validity Process. In N. Schmit & W. C. Borman (Eds.), Personnel Selection in Organizations (pp. 3-34). San Francisco: Jossey-Bass.

(6) Ibid.

(7) See Morgeson, E P., & Campion, M. A. (1997). Social and Cognitive Sources of Potential Inaccuracy in Job Analysis. Journal of Applied Psychology, 82, 627-655.

(8) Mullins, W. C., & Kimbrough, W. W.

(9) Landy, F. J., & Vasey, J.; Mullins, W. C., & Kimbrough, W. W.

(10) Landy, F.J., & Vasey, J.; Veres, J. G., Green, S. B., & Boyles, W. R. (1991). Racial Differences in Job Analysis Questionnaires: An Empirical Study. Public Personnel Management, 20, 135-144.

(11) Conley, P. R., & Sackett, P. R. (1987). Effects of Using High-Versus Low-Performing Job Incumbents as Sources of Job Analysis Information. Journal of Applied Psychology, 72, 434-437.

(12) Schmitt, N., & Cohen, S. A. (1989). Internal Analysis of Task Ratings by Job Incumbents. Journal of Applied Psychology, 74, 96-104.

(13) Forber, L. B. (1991). Quality: American, Japanese, and Soviet Perspectives. Academy of Management Executive, 5, 63-74; Hill, R. C., & Freedman, S. M. (1992). Managing the Quality Process: Lessons from a Baldridge Award Winner. A Concersation with CEO John W Wallace. Academy of Management Executive, 6, 76-88.

(14) Mani, B. G. (1995). Progress on the Journey to Total Quality Management: Using the Myers-Briggs Type Indicator and the Adjective Check List in Management Development. Public Personnel Management, 24, 365-398.

(15) Meyer, H. H. (1959). Comparison of Foreman and General Foreman Conceptions of the Foreman's Job Responsibility. Personnel Psychology, 12, 445-452.

(16) Hazel, J. T., Madden, J. M., & Christal, R. E. (1964). Agreement between Worker-Supervisor Descriptions of the Worker's Job. Journal of Industrial Psychology, 2, 71-79.

(17) O'Reilly, A. P. (1973). Skill Requirements: Supervisor-Subordinate Conflict. Personnel Psychology, 26, 75-80.

(18) Smith, J. E., & Hakel, M. D. (1979). Convergence among Data Sources, Response Bias, and Reliability and Validity of a Struc-tured Job Analysis Questionnaire. Personnel Psychol-ogy, 32, 677-692.

(19) Cornelius, E. T., DeNisi, A. S., & Blencoe, A. G. (1984). Expert and Naive Raters Using the PAQ: Does It Matter? Personnel Psy-chology, 37, 453-464.

(20) Mueller, M., & Belcher, G. (2000). Observed Divergence in the Attitudes of Incumbents and Supervisors as Subject Matter Experts in Job Analysis: A Study of the Fire Captain Rank. Public Personnel Management, 29, 529-556.

(21) Schippmann, J. S., Ash, R. A., Battista, M., Carr, L., Eyde, L. D., Hesketh, B., Kehoe, J., Pearlman, K., Prien, E. P., & Sanchez, J. I. (2000). The Practice of Competency Modeling. Personnel Psychology, 53, 703-740.

(22) Ibid.

(23) Morgeson, F. P., & Campion, M. A.

(24) See Casio, W. F.

(25) Morgeson, F. P., & Campion, M. A.

(26) Ibid.

(27) Ibid.

(28) See Shavelson. R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability Theory. American Psychologist, 44, 922-932.

(29) See for example: Maurer, T. J., Alexander, R. A., Callahan, C. M., Bailey, J. J., & Dambrot, F. H. (1991). Methodological and Psychometric Issues in Setting Cutoff Scores Using the Angoff Method. Personnel Psychology, 44, 235-262.

(30) See Cortina, J. M. (1993). What Is Coefficient Alpha? An Examination of Theory and Applications. Journal of Applied Psychology, 78, 98-104.

(31) Doverspike, D., Carlisi, A. M., Barrett, G. V., & Alexander, R. A. (1983). Generalizability Analysis of a Point-Method Job Evaluation Instrument. Journal of Applied Psychology, 68, 476-483.

(32) See Maurer, T. J., Alexander, R. A., Callahan, C. M., Bailey, J. J., & Dambrot, F. H. (1991). Methodological and Psychometric Issues in Setting Cutoff Scores Using the Angoff Method. Personnel Psychology, 44, 235-262.

(33) See for example: Gatewood, R. D., & Feild, H. S.; Milkovich, G. T., & Boudreau, J. W. (1991). Human Resource Management. Homewood, IL: Irwin.

(34) See Morgeson, F. P., & Campion, M. A.

(35) Ibid.

(36) Schippmann, J. S., Ash, R. A., Battista, M., Carr, L., Eyde, L. D., Hesketh, B., Kehoe, J., Pearlman, K., Prien, E. P., & Sanchez, J. I.

(37) Gatewood, R. D., & Feild, H. S.

(38) See Goldstein, I. L., Zedeck, S., & Schneider, B.

Authors

Donald M. Truxillo

Department of Psychology

Portland State University

P.O. Box 751

Portland, Ore. 97207-0751

(503) 725-3969

Donald Truxillo is an associate professor at Portland State University in Portland, Ore. Previously, he worked in public sector human resources for several years in the design of selection systems. His current research focuses on applicant perceptions and test validation issues. He also consults in the area of selection and training for public safety jobs.

Matthew E. Paronto

Department of Psychology

Portland State University

P.O. Box 751

Portland, Ore. 97207-0751

(503) 242-1955

Matthew Paronto is doctoral student at Portland State University. His current research focuses on applicant perceptions and test validation issues, as well as applicant perceptions of organization-sponsored substance abuse treatment programs.

Michelle Collins

HRD Solutions

3715 Douglas Ave.

Dallas Texas 75219-2941

(214) 559-2599

Michelle Collins has more than 15 years of experience in the field of management assessment and development. In her consulting practice, she works with clients in a variety of industries, including retail, government, entertainment, technology and airlines. Michelle holds an M.S. and Ph.D. in Industrial/Organizational Psychology from Tulane University.

Jefferson L. Sulzer

54560 Hwy 16

Bogalusa LA 70427-1168

(504) 735-0904

Jefferson Sulzer is Professor Emeritus of Psychology at Tulane University. His current work involves public sector personnel selection and test validation issues.

Table 1. Means and Standard Deviations for Criticality Items for Police
and DAs

                                  Police        District Attorney

                                  n = 70        n = 31

Dimension                         M      SD     M      SD

1 .Vocabulary                     4.79   0.47   3.16   1.11
2 .Spelling                       4.57   0.85   3.54   1.18
3 .Punctuation                    4.54   0.73   2.94   1.12
4 .Grammar                        4.61   0.62   3.03   1.06
5 .Sentence logic                 4.80   0.52   3.51   1.18
6 .Combine clauses                4.38   0.86   2.79   1.22
7 .Accurate description           4.96   0.20   4.91   0.29
8 .Narrative logic                4.82   0.42   4.06   0.93
9 .Witness accuracy               4.94   0.23   4.94   0.35
10. Observed vs. heard            4.86   0.39   4.88   0.33
11. Attribute to proper witness   4.96   0.20   4.88   0.55
12. Enough detail                 4.96   0.20   4.76   0.61
13. Only facts                    4.76   0.64   4.21   1.08
14. Description of evidence       4.89   0.36   4.42   0.96
15. Description of perpetrator    4.96   0.20   4.79   0.48
16. No superfluous information    4.78   0.56   3.94   1.05

Table 2. Repeated Measures ANOVA Comparing Mean Criticality
Ratings for Grammar and Content Subscales

                             Sum of         Mean
Source                       Squares   dt   Square   F

Between subjects
Occupation (DA vs. Police)   24.99      1    24.99   86.43 **
Error                        28.63     99     0.29
Within subjects
Facets                       28.87      1    28.87   185.66 **
Facets X Occupation          14.03      1    14.03    90.24 **
Error (Facets)               15.39     99     0.16

Source                       [n.sup.2]

Between subjects
Occupation (DA vs. Police)     .47
Error
Within subjects
Facets                         .65
Facets X Occupation            .48
Error (Facets)

N=101; ** p<.01.