A Methodology for Measuring Engineering Knowledge Worker Productivity

In our society today there is a dangerous mind-set, which engineering managers may be witnessing, that results in behavior harmful to the productivity of knowledge workers. The term "knowledge work" was first coined by Dr. Peter Drucker in the 1960s as any work that requires mental power rather than physical power (Fisher, 1998). It has been further defined as work that involves analyzing information and applying specialized expertise to solve problems, generate ideas, teach others, or create new products and services (Evans, 1993).

It is difficult to define knowledge work in more detail because knowledge work is primarily invisible. It is hidden in the head of the knowledge worker (Fisher, 1998). Because of the difficulty of measuring knowledge worker production, dissatisfied knowledge workers may take advantage of the situation. This dissatisfaction may produce behavior in which personnel seek more financial satisfaction by giving themselves a "stealth raise", i.e., cutting back the effective hours in which they perform knowledge work at the office. They may dedicate more mental effort to another activity that is not job-related that brings them more satisfaction (Barber, 1999). This contradicts Frederick Taylor's main philosophy of a fair day's work for a fair day's pay. Even though there should be no expectation of blind company loyalty as was expected in the past, companies should expect good work and some form of commitment to productivity from their knowledge workers while they are on the job.

Businesses lose $150 billion annually in health insurance and disability claims, lost productivity, and other expenses attributable to burnout, stress-related problems and mental illness (Bassman, 1992). Further quantification of the bottomline impact of indirect cost is demonstrated by the high cost of absenteeism, which is estimated at approximately $40 billion per year in the U.S. (Gaudine, 2001 ). Previous studies on turnover and burnout categorize costs into three groups: direct costs, indirect costs, and opportunity costs. Direct costs include disability claims, worker's compensation claims, increased medical costs, and litigation costs (including wrongful discharge, hiring new personnel, training costs, advertisement for new personnel, and time spent interviewing new personnel). Indirect costs include those associated with poor quality, high turnover, absenteeism, poor customer relationships, or even sabotage. Opportunity costs include those associated with lowered employee commitment, lack of discretionary effort, commitments outside of the job, time spent talking about problems instead of working, and loss of creativity.

Cognitive turnover (CT) is a term developed in this research to describe a phenomenon that is created by a combination of turnover thoughts/cognitions brought about by burnout conditions. While everyone may manifest this mind-set periodically, excessive CT may be detrimental to the individual and the organization for which they work. Subtle acts such as absenteeism, poor quality, and lack of discretionary effort have been related to worker burnout and are common predecessors to quitting and becoming another turnover statistic. Noncommitment type behavior may stem from employee stress and burnout created by management or organizational abuse; hence eCT will lower productivity because of lack of commitment.

Engineering managers may be able to avoid the negative consequences to the organization and employee by identifying the non-productive knowledge workers experiencing eCT; however, it is probably more productive to seek aggregate or group information that will facilitate improvements in attitude, innovation, productivity of the organization, and may prevent ineffective events such as reduced employee productivity and sabotage.

This research demonstrates that an eCT condition occurs when a person is absorbed with the thoughts of turnover created by organizationally driven burnout and provides a methodology to identify personnel with eCT so that the organizations can take actions that may improve the companies' bottom-line.

Understanding Cognitive Turnover

CT is a combination of a turnover thought process and a burned out mental condition. Turnover is defined as voluntary cessation of membership in an organization by an individual who receives monetary compensation for participating in that organization (Mobley, 1982). Turnover has cognitive indicators that predate eventual departure. During this pre-turnover stage, a worker generally has lower productivity. There may be an attempt, prior to quitting, to sabotage the workplace with negative sentiment or other more devious activities. This definition emphasizes voluntary behavior because prevailing turnover models primarily seek to explain what motivates employees to withdraw from the workplace.

Burnout is the mental dissonance from organizational pressure. Burnout, which has been viewed as fairly subjective, is a significant factor in the business world. Cherniss (1980) defines burnout as "a syndrome of inappropriate attitudes toward clients and toward self, often associated with uncomfortable physical and emotional symptoms." Maslach (1976) observed that burnout "appears to be a factor of job turnover, absenteeism, and low morale." He cites other various self-reported indices that indicate burnout may cause personal distress, including physical exhaustion, insomnia, increased use of alcohol and drugs, and marital and family problems. Other researchers present even longer lists of burnout symptoms (Cherniss, 1980).

Similar to pre-turnover thought processes, high degrees of burnout among major proportions of a group suggest low productivity. High burnout implies little slack in a person's coping capacities, and perhaps deficits in them. High measures of burnout are strong indicators of these phenomena, but the inverse, low burnout, does not necessarily indicate high productivity (Golembiewski, 1982). This research focused on the high measures of burnout in conjunction with pre-turnover indicators.

Study Scope and Objectives

Thescope of this research was to develop and test a questionnaire to measure the components of CT in engineering knowledge workers and develop a mathematical model that can be used to measure CT. The researchers have developed a methodology that is being explored as a means to consistently measure knowledge worker CT.SECtCS, or Statistical Evaluation of Cognitive Turnover Control System, is a methodology that attempts to identify, measure, and document CT. The questionnaire developed in this study can be used by engineering managers and will be further described in the results and conclusions. The following are the six phases of the sectCS Research Methodology for Knowledge Workers (note that this article focuses on the first two):

Phase 1 - Develop Test Instrument-(SECtCS Questionnaire)

Step 1: Develop a customized test instrument for the knowledge worker population

Step 2: Administer the questionnaire

Step 3: Collect and record scores

Step 4: Conduct reliability testing on the questionnaire

Phase 2-Develop Mathematical Model-(SECtCS Modeler)

Use the data collected to develop a regression model for a valid CT index score

Phase 3-(not described here)-Statistical Process Control Charts

Use data from the model developed in Phase 2 for the statistical measurement of individuals with respect to all respondents and identify at-risk CT index scores. (SECtCS Evaluator-i) Establish a tracking mechanism for "at risk" and "low risk" respondents. The respondents are required to retake the questionnaire every three months in order to complete the SPC charts.

Phase 4-(not described here)-Intervention

Educate, implement, and monitor the solution (SECtCS intervention)

Phase 5-(not described here)-Intervention Measurement

Remeasure the respondents after they have been subjected to the intervention and compare to the results of Phase 3 (SECtCS Evaluator-r)

Phase 6-(not described here)-Evaluate Intervention

Document the results and conclusions and add to solution database

Although the main objective of the overall research was to develop and test the SECtCS methodology, this article presents only the results of the first two phases of the SECtCS methodology. This involved the development of a questionnaire to measure CT in engineering knowledge workers and the creation of a mathematical model that measures and predicts the CT index levels. The long-term objective of this methodology is to provide a tool for engineering managers to use in assessing the level of CT in engineering knowledge workers. The CT index can then be used by organizations to identify possible low productivity knowledge worker personnel, departments, and groups prior to significant productivity loss. It has the potential to identify CT so that organizations can take appropriate actions and increase the retention and productivity of their knowledge workers.

Traditional models of burnout and aspects of pre-turnover have been constructed, for the most part, on personnel after the events have happened (i.e., the person has quit or has attacked management). The goal of this research was to identify this behavior in a more predictive mode and allow organizations to implement interventions prior to a negative event occurring. In this study, a questionnaire was developed that measures relevant facets of burnout and turnover related to cognitive turnover. The results were then used to develop a mathematical model that predicts the level of cognitive turnover. Lessons learned and future opportunities for using the proposed methodology are discussed.

Research Methodology

Phase 1: Develop Test Instrument (SECtCS Questionnaire)

The summated rated scale methodology was used to create the SECtCS questionnaire. Its invention is attributed to Rensis Likert (1932), who described this technique for the assessment of attitudes. These scales are widely used across the social sciences to measure attitudes and descriptions of people's lives. Summated rated scales have good psychometric properties and, if effectively developed, will demonstrate good reliability and validity. A welldevised scale is usually quick and easy for respondents to complete and typically does not induce complaints. The questionnaire was developed using questions that have been shown to measure burnout constructs and turnover constructs. Multiple questions designed to measure responses are grouped into constructs, or measurable variables.

The questionnaire used in this study was developed from three widely used and accepted questionnaires-the Maslach Burnout Inventory (MBI) (Maslach and Jackson, 1981), the Minnesota Satisfaction Questionnaire (MSQ) (Lofquist and Dawis, 1967), and the Facet-Specific Job Satisfaction Questionnaire (FSJSQ) (Cook, Hepworth, Wall, and Warr, 1989). The MSQ is one of the most widely used measures of job satisfaction. The FSJSQ is commonly used in measuring specific jobs satisfaction items. The items, each measuring a "facet," as indicated in the scale's title, were previously used in a 1973 survey, and a similar measure was employed in 1969 (Cook et al., 1989). The reason both were used is that one measures general job satisfaction while the other measures constructs related to job satisfaction but not specifically job satisfaction. The reasoning for using both in this research is that low job satisfaction leads to turnover, or cognitive turnover.

Burnout is commonly assessed using the Maslach Burnout Inventory (MBI) (Maslach and Jackson, 1981 ). The MBI is a widely accepted questionnaire for numerous burnout studies that has been demonstrated as reliable and valid in case studies. It measures three constructs: depersonalization, personal achievement, and emotional exhaustion, which relate burnout to the respondents' physical well-being. See Exhibit 1 for a description of burnout constructs.

The eight constructs related to turnover are general job satisfaction, goals, comfort, challenge, financial rewards, relationships, resource adequacy, and promotions (Exhibit 1). Applicability to the knowledge worker was a primary criterion for question selection. The initial version of the questionnaire and rating scale was pilot tested and critiqued by other researchers. After feedback from other researchers, ambiguous or confusing items were identified and eliminated.

Respondents completed three questionnaires. The first was comprised of 109 questions concerning the job satisfaction constructs and burnout constructs. A second questionnaire asked for the respondents' more direct appraisal of their level of CT. Respondents were assured that their answers would remain anonymous. Respondents were given both a verbal and written description of CT and the levels of CT. A description of each range is shown in Exhibit 2. The respondent then self-scored their level of CT given the range (1 to 10). Respondents were asked to rate, on a scale of 1 (strongly agree) to 5 (strongly disagree), that indicated how they felt about their employer. An example is: "My employer is concerned about giving everyone a chance to get ahead." They were also asked to rate specific job satisfaction questions on a scale of 1 (very dissatisfied) to 5 (very satisfied). An example is: "On my present job, how do I feel about my pay and the amount of work I do?" The mean values were calculated for each construct.

The questionnaires were distributed to 108 engineering knowledge workers from over 20 companies in the Houston area. Fifty-one questionnaires were returned, representing a response rate of 47.2%. Four incomplete questionnaires were eliminated from the data set, reducing the sample to 47. The data were examined for skewness, kurtosis, and outliers and found to be within the parameters expected in normal distribution. Using SPSS, the data were analyzed to determine the main effects and interactions between the different constructs and CT classifications. An analysis of reliability for each construct was conducted to reduce the number of questions and provide a satisfactory internal consistency. After the elimination of confusing questions, an item factor analysis was performed on the returned questionnaires and the initial set of 109 questions was reduced to 59 questions.

A measurement instrument, if reliable, will give similar results when different people administer it or alternate forms are used. Coefficient alpha (Cronbach, 1951) is a measure of the internal consistency of a scale. The values of coefficient alpha are positive, taking values from 0 to 1.0, where larger values indicate higher levels of internal consistency. Nunnally (1978) and Spector (1992) provide a rule of thumb for acceptability that coefficient alpha should be at least .70 for a scale to demonstrate internal consistency.

Cronbach's alpha was determined for each construct. Questions were deleted in order to attain a desirable coefficient alpha for each of the eleven constructs. Items were successfully deleted from constructs from the original set in order to improve reliability. The final questionnaire was reduced from 109 questions to 59 questions. The results from reliability analysis are shown in Exhibit 3.

The coefficient of determination, R square, is commonly used in research to measure the goodness of fit for regression models. It can also be used as the proportion of variation in the dependent variables "explained" by the model. If the R square is acceptable then model may be considered valid. Stepwise regression was used at an alpha level of 0.05 for our regression analysis. The final regression model was evaluated using R square for the models ability to determine future CT index scores from the questionnaire.

Phase 2: Develop Mathematical Model (SECtCS Modeler)

Multiple linear regressions were used to develop the SECtCS model. They were performed using the stepwise method for each of the different independent variables. The model was examined to ensure that no violations of assumptions occurred including multi-collinearity and heteroscedacity. The dependent variable in this study was respondents' cognitive turnover score. The turnover and burnout constructs measured on the questionnaire are the independent variables that were used to determine the dependent variable, or the CT index.

Cognitive Turnover Results

Exhibits 4 and 5 show results for demographic variables in this study. The ranges along these dimensions were small. The respondents' ages varied from 20 to 64 with a mean of 25 (see Exhibit 4). There were 15 females in the sample representing 29% of respondents as shown in Exhibit 5. The length of service ranged from 1 month to 480 months with a mean of 35 months as shown in Exhibit 6. The CT results, measured on a 10-point scale, are shown in Exhibit 6. There were 34 respondents who, while working, were looking for a better job (noted as LBJ in Exhibit 7), that represents 72% of the sample. There were only eight respondents working a secondary job or activity.

The mean value for each construct of turnover and burnout was determined. The mean scores from the questionnaire constructs were calculated from the values obtained from responses to the individual questions. The mean and standard deviations are listed in Exhibit 8.

The result of the regression analysis is shown in Exhibits 9 and 10. The analysis of variance shows that only four variables had a significant effect on cognitive turnover. The p-values indicate that these variables (see Exhibit 10) have a significant effect on CT at an alpha level of 0.10.

The turnover variables Challenges and Promotions and the burnout variables Depersonalization and Personal Achievement were significant in predicting engineer's CT in this study. Most of the job satisfaction constructs were not shown to be significant (Goals, Comfort, Financial Rewards, Relationships with Co-workers, and Resource Adequacy) and only one of the burnout constructs was not significant in predicting CT the dependent variables (Emotional Exhaustion). These constructs had a weak impact on CT for this group.

Based on these results, the mathematical model for predicting CT for engineers is given by the following equation:

F(x) = 1.199(Challenges) + 1.575(Depersonalization) - 1.712(Personal achievement) - 0.935 (Promotion) + 5.122 (1)

Exhibit 11 summarizes the four variables in the model that were determined to be significant predictors for CT and describes the impact of each on CT.

The function F(x) will be a number between 1 and 10. Scores in the 1-4 range represent low cognitions to leave and generally low burnout indications. Scores in the 5-8 range represent moderate burnout and leaving cognitions. Scores 9 and above represent eCT which may lead to detrimental burnout and possible sabotage if departure is not eminent (refer to Exhibit 2 for definitions).

It is important to note that because only four variables were necessary to determine the CT level, it might be possible for the engineering manager to reduce the 59-question questionnaire to 19 questions. The danger in utilizing the 19-item questionnaire is that these results are based on a small sample size from exploratory research. The engineering manager should evaluate the limitations before using the 19-item questionnaire. The benefit of using the reduced question form is that it is quicker to administer. The researchers suggest using the currently-developed questionnaire and performing Phase 2 for the engineering manager's specific knowledge worker group in order to obtain the most effective measurement instrument.

Study Limitations

Some limitations to this research were the sample size and questionnaire biases. This study used only 51 knowledge workers across multiple organizations for the creation of the mathematical model. Currently, more populations are being targeted for further validation of the mathematical model. This should be taken into account when utilizing the model for possible sample bias when using the questionnaire and regression model. Also, questionnaire biases can occur when implementing the testing of the questionnaire. Respondents may not answer the questionnaire honestly if they feel threatened by what will happen if they score on the high end of the index. The researchers recommend utilizing tools such as a digital simulator or online questionnaire software to offset some of the fears of being identified and possible ramifications. Future research includes the development of a manager's checklist that will allow managers to observe specific behaviors enabling the manager to score the employee for CT. If the 19-item questionnaire is utilized, it is important to note that this research is exploratory and caution should be used before the results are acted upon. Future research may focus on industry specific models.

Lessons Learned and Recommendations

Our findings yielded several lessons learned and many recommendations. First, knowledge worker management is difficult, but crucial to companies' future growth and bottom lines. Kotnour (1999) defined knowledge management as a set of activities that support a firm in creating, assimilating, disseminating, and applying knowledge including tacit knowledge that resides in the minds of people. An organization's success is driven by its effective use of improved products and technology that are created in the minds of knowledge workers. The direct and indirect costs of losing these personnel directly impact a company's profitability.

Second, analyses of the empirical data on the CT indices presented here suggest that companies need to ( 1 ) focus their current practice away from solely financial measures and toward providing challenging work, (2) reduce isolated tasks that cause depersonalization and increase team activities, (3) increase recognition of personal achievement, and (4) provide realistic promotion opportunities. In this study, the high level of depersonalization may suggest that when knowledge workers perform isolated tasks, they can experience higher levels of CT. The engineering manager may be able to improve this component with team-based tasks. Further, the high negative coefficient for personal achievement on CT indices suggests that recognition of knowledge workers can have a strong positive effect on CT. The other two significant variables were promotion and challenges -which may not be under the direct control of the engineering manager.

Finally, one opportunity for improvement that companies miss is giving real feedback to employees. Companies should address the problems with performance by brainstorming and communicating with employees about possible solutions. By using the first two phases of the sectCS methodology, the engineering manager has a method for identifying some of the main components of the CT. The first two phases allow the organization to identify the most significant measures of eCT for the chosen group of knowledge workers. The complete methodology, which is not fully presented in this paper, is designed to measure relevant components of eCT, re-measure implemented solutions effectiveness, document efforts, and provide feedback. As a result of this study, which focused on the measurement phases of sectCS, the following recommendations are suggested:

1. Educate management about the burnout and pre-turnover components that may create high CT indices.

2. Educate management in how eCT can impact the company's bottom line negatively.

3. Be aware of knowledge workers who may be demonstrating some of the general symptoms of CT.

4. Administer the full questionnaire anonymously for evaluation of CT constructs in targeted departments/groups.

5. Create a management checklist for the different constructs that will help a manager identify signs of eCT.

Engineering managers can use the general sectCS questionnaire to collect information and generally evaluate the constructs or areas of CT that may be reducing their knowledge worker productivity. More extensive analysis could be performed for organizations by using the full questionnaire described in Phase 1 and the regression techniques described in Phase 2 in order to create a group-specific questionnaire and mathematical model to measure the CT level of their knowledge workers. The sectCS model created in Phase 2 can be utilized to identify unproductive knowledge worker measures or variables specific to that work group. This group-specific questionnaire and model should be utilized for existing knowledge workers where the organization finds it hard to measure, but requires innovation in order to keep their competitive advantage. Implementation of other phases of the methodology is not recommended without further research as defined here.

Conclusions

In this study, the first two phases of the sectCS methodology were used to develop a questionnaire and a mathematical model for measuring cognitive turnover indices. This questionnaire was developed for full time engineering knowledge workers across different organizations. The results of the study suggest that these participating organizations need to focus their current practice away from mostly financial measures and toward developing more knowledge specific challenging work, increase team activities, increase personal recognition, and provide fair upward mobility opportunities.

Previous research suggests that project managers can influence knowledge workers with organizational influence. Influences like friendliness, bargaining, rational reasoning, assertiveness, coalitions, sanctions, consultation, inspirational appeal, and higher management support were measured for effectiveness. This may be true in certain instances but the discretionary effort that may be required to make a project or an initiative a true success may not come from outside influences but from internal motivators such as the feeling of importance in an organization. This study demonstrates an organized approach to evaluate and possibly improve these work motivations.