A framework for assessing the reliability of software project status reports

Introduction

The seeds of software project failure are often sown with inaccurate status reporting. In spite of this, little is known about the distortion that can occur in project status reporting. The purpose of this research was to build a model to investigate how reported status is impacted by: (1) the fallibility of the project manager in terms of his or her ability to assess the true status of a software project, and (2) the degree to which the perceived status is then slanted as it is communicated upward to executive management.

A project manager may misperceive the true status of a project for many reasons. Although this is a general problem that can occur with any type of project, software projects may be particularly challenging in this regard (Abdel-Hamid, 1988; Brooks, 1975; DeMarco, 1982; Zmud, 1980). The intangible nature of software makes it difficult to obtain accurate estimates of the proportion of work completed, which may promote misperceptions regarding project status.

To make matters worse, project managers may sometimes censor themselves in reporting status as they actually perceive it. If a project is perceived to be performing poorly, the project manager may withhold this information or misrepresent status to senior management. The net result is that executives may often be at the receiving end of status information that has been biased in some fashion. The CONFIRM project (originally estimated at $55.7 million, and cancelled 3.5 years later with expenditures of $125 million) represents a case in which individuals who were close to the project apparently knew that there were significant problems, yet did not disclose the true status of the project to senior management (Oz, 1994). Examples such as this one underscore the need to understand the combined effects of project manager misperceptions (errors) and bias in reporting, which together led to what we call "distortion" in the project status information received by senior executives.

Practitioner Status Metrics and Research Questions

Software projects are widely believed to be difficult to manage because of such factors as continually changing technology and the difficulty of fixing user requirements (Sauer et al., 1999). The intangibility of software means that there are no visible milestones to measure progress as there are for a physical product (AbdelHamid and Madnick, 1991). Furthermore, research suggests that software project managers have a tendency to anchor on their initial perceptions, in spite of subsequent pervasive evidence to the contrary (Abdel-Hamid et al., 1993). This suggests that once project status is misperceived, it is difficult to correct such errors.

Thus, the software project environment contrasts sharply with other, more traditional, project environments such as construction in which project status can be more readily observed. A 1999 study by Sauer, Liu, and Johnston comparing construction projects and software projects suggests that "the performance difference across industries is real, that the degree of task certainty is different, and that some of the performance differences may be attributable to organization and management" (Sauer et al., 1999). With complex software projects involving numerous programmers, there is often a tendency to try to obtain status information from each programmer on his or her particular task, but this can be problematic (Boebert, 1979). Moreover, programmers may have a tendency to hide problems or embarrassing situations (Abdel-Hamid and Madnick, 1991). Thus, obtaining accurate status information remains a challenge for most software project managers.

Traffic Light Reporting as a Status Metric. However challenging, the project manager must have a means of distilling project costs, technical, and schedule considerations into a meaningful status metric. This is a difficult assessment because with software projects, function and performance are not often measurable until final systems integration. While there are a variety of different status reporting techniques in use today, one of the simplest approaches is known as "traffic light reporting." Traffic light reporting probably originated within the construction industry, as it first appears in research literature dealing with costs in civil engineering projects (Coulter, 1990). The definitions of what constitute a "Green," "Yellow," or "Red," state can obviously vary from one context to another. In traffic light reporting, project managers distill all project factors into a discrete judgment on project status, for executive consumption.

Within the information technology area, traffic light reporting has been used to rate business partners in terms of their Y2K readiness (Zerega, 1998), to communicate the status of individual project tasks for various process improvement projects (Burris, 1994), and to track the vital signs (e.g., scope, schedule, and budget) for software projects (Brown, 1998).

The construction literature defines status in a quantitative and qualitative way appropriate for construction projects, where cost and schedule, safety, and quality are assessed in coming to one overall discrete status for a project (Chen, 1996). These aspects are inappropriate for software, as functionality and performance are the elements sacrificed when cost or schedule is overrun. For the purposes of this research, three project objectives are defined as budget, schedule, and functions/performance.

The status definitions used are as follows:

Green: All three objectives substantially met to date;

Yellow: Two objectives substantially met to date;

Red: Less than two objectives substantially met to date.

Other definitions are possible, such as including percent overrun parameters for cost or schedule. Percentage of functions or performance can also be incorporated. Traffic light reporting provides a simple but useful status metric for discussing issues of error and bias in status reporting. For example, a project may be in "Red" state, but the project manager may inaccurately perceive it to be in a "Yellow" state, or the project manager may perceive the project to be in a "Yellow" state but may bias the report as "Green."

Research Questions. As pointed out, research literature indicates it is difficult for project managers to accurately assess software project status. Research also indicates that incorrect status is often reported to executives, with severe repercussions (Keil and Robey, 1999). Too much optimism may cause the executive to continue support of a doomed project, or be surprised by a negative outcome. Also, some organizational cultures may not tolerate failure, resulting in "good-news" only reporting. The motivation of the project manager might be to succeed in spite of any current perceived project adversity.' Perhaps an organization's worst nightmare is an escalating commitment to a failing project (Keil, 1995). Project status reporting distortions may contribute to such failures. This sets the stage for research questions dealing with the effects of project manager status error and bias in status reporting:

How large a difference might there be between true and reported status because of the combined effects of project manager's status errors and bias?

What distorts reported status the most-project managers' status errors or reporting biases?

Are these effects different for different risk projects? Answers to these questions might help organizations enjoy more success in software projects.

Two-Stage Status-Reporting Model

Discrete project status states from traffic light reporting allow the construction of a tractable project status-reporting model. Although project status is continuous over time, periodic discrete state reporting offers an opportunity to consider Markov process and information theoretic approaches, as depicted in Exhibit 1. In this model there are three project statuses, each of which is a random variable: True status (7), Perceived status (P), and Reported status (R). True status is periodically reported, subject to misperceptions and bias. The selected model is a cascaded information channel, where the possible input alphabet (G, YR)is subjected, in turn, to two different noise channels. The input alphabet passes through an error channel, and the intermediately received alphabet then passes through a bias channel. The model assumptions and their plausibility were presented in Snow and Keil (2001).

In such a system, if we know the input alphabet probabilities (chances of the project being in Green, Yellow, and Red states), the error channel characteristics, and the bias channel characteristics, we may determine a number of things, including:

* The probability the project is reported Green,

* The conditional probability, if reported Green the project is truly Red,

* The joint probability, the project is truly Red and reported Green.

These and other closed form analytical results are well known from information theory (Hamming, 1986). In the remainder of this section, we present the model in more detail. For instance, assume random variables T, P, and R take on value t^sub i^ p^sub j^, and r^sub k^, where indices of 1 are Green, 2 Yellow, and 3 Red. We may represent the perceived probabilities as: IMAGE FORMULA 30

Research Methodology

The software project management environment is one that is complex and challenging (Abdel-Hamid and Madnick, 1991). In many cases, significant insights can be gained by constructing models that allow the researcher to simulate the complexities associated with the software project management environment (Abdel-Hamid, 1988; Abdel-Hamid, 1990; Abdel-Hamid and Madnick, 1989). In this research, we developed and tested a simple model based on information theory, which allowed us to explore the ramifications of errors and bias in project status reporting. To ensure that our results would be meaningful, we interviewed five software project risk experts to determine estimates for true status. In the remainder of this section, we describe our methodology in more detail.

Combined Effect of Error and Bias: Distortion. The basic research model introduced earlier (see Exhibit 1) consists of a two-stage model incorporating both status errors and reporting bias. The range of parameters to be investigated using the two-stage model is shown in Exhibit 2.

The discrete state probabilities (i.e., the chance of being in a Green, Yellow, or Red state) form the inputs to the model. Since these probabilities will likely vary depending upon the risk level of the project, we were interested in examining a range of risk types, including low, medium, and high-risk projects. The parameter estimates for true project status (T), which constitute the inputs to the model, were obtained from experienced software project risk experts. For each project risk type, the effects of errors in project management perception and project management reporting bias were investigated by performing sensitivity analyses. To test the model's sensitivity to errors in perception (E), we created various error channels ranging from errorless to error-prone. To test the model's sensitivity to bias in reporting (B), we created various bias channels ranging from no bias to complete bias.

True Status Data Collection. To obtain the initial parameter estimates for true status (T), we conducted a field survey of five software project risk experts, each of whom had extensive experience in evaluating the risks associated with software projects. We chose risk experts instead of project managers because of their years of experience, large numbers of projects participated in or audited, and the breadth of different organizational settings and software applications they have encountered. We did not attempt to collect true status from a sample of project managers because we were skeptical of their ability to do so, the concern being that their perceptions included error. The experience base of the panelists and the exact protocol used to elicit true status is contained in Snow and Keil (2001). True status was elicited at the start of the development phase. It is important to note that the true status is not the predicted status at completion, but rather, the status at this point in the development cycle.

True status predictions from the panel of experts for low, medium, and high-risk projects are presented in Exhibit 3. These results, based on the average of the five experts, are internally consistent within project type and across the risk spectrum. For instance, we would expect low-risk projects to be in a Green state more often than medium- or high-risk projects. Although consistent, these results deserve comment, as they suggest that after project definition and design, the software risk experts judge most projects to be in peril. This is consistent with the literature where we find many software projects get off to a bad start because of such factors as unrealistic budgets and schedules, and poor requirements analysis (Jones, 1994). In addition, from a qualitative perspective, the data exhibited reasonably tight clustering.

The simplicity of the true average estimates beguiles the sophistication of these results. These average results do not mean that the next project state is independent of the previous state(s)in fact we would expect some memory in the process. For instance, in a 1st order Markov process, the next state is dependent only upon the previous state. This means that if a project were truly Red in the last reporting period, the chance of now being in the Red state is dependent on the fact that the last state was indeed Red. The state transition probabilities (the chances of moving from one state to another), shown in Exhibit 4, produce the steady state results for high-risk projects depicted in Exhibit 3. There are other combinations that also produce these results, but these probabilities are shown here for two reasons. First, they show that the data from the experts is consistent with plausible "stickiness" scenarios, where the current state is dependent on the previous state and that, as we would expect, there are some inertial aspects to project status. Secondly, the state transition solutions are used in simulations to verify the analytical results we present later in this article.

Because of the estimate consistency, alignment with the literature, reasonable clustering of the data, and the consistency with a Ist order Markov process, we believe these true status estimates to be a reasonable point of departure for our investigation into reporting reliability.

Error and Bias Channels. The average parameter estimates obtained from the software project risk experts were used as inputs to the two-stage model described earlier. A number of different error channels were then constructed to represent 0%, 10%, 30%, and 50% chance of misperceiving Green as something other than Green, Yellow as something other than Yellow, and Red as something other than Red. The error channels were chosen to test a range of error probabilities. The 10% error channel was chosen to represent a low error probability and to discern whether such low probabilities would be significantly different from an environment in which project managers made no errors. At the other extreme, we selected a 50% error channel, representing a coin toss, as this would seem to represent a very high error rate. In addition, we assume it is three times more likely to make a one state error (e.g., Green to Yellow) than a two state error (e.g., Green to Red). Intuitively, two state errors are more improbable than single state errors.

Bias channels were also constructed to represent a range of bias situations (none, slight, modest, moderate, heavy, and complete). Complete bias means a project is always reported Green, irrespective of perceived status.

Results and Discussion

In this section, we present and analyze the results produced by our analytical model. First, we present and discuss the effect of different error and bias levels on what is reported, and compare results with a discrete time-event simulation in order to verify model results. Next, we present and discuss perceived and reported status of low-, medium-, and high-risk projects. Lastly, we present and discuss the chances of Red projects actually being reported as Red.

Effects of Different Error and Bias Levels on What is Reported. For each project risk type, the state probabilities of Exhibit 3 serve as the input to the model previously introduced in Exhibit 2. Channel equations introduced in section 3 were constructed using the error and bias channels, then solved with matrix algebra. Also for each project type, the model was solved for four different error channels, namely the errorless, 10% error, 30% error, and 50% error channels. In addition, each error channel was concatenated with bias channels ranging from no bias to complete bias channels, as described earlier. The remainder of this section presents selected results. To address the earlier posited research question regarding the effect of project management errors versus the effect of bias reporting on reported status, the Green status probabilities are shown for the low- and high-risk projects, Exhibits 5 and 6, respectively.

On these graphs, B0 represents zero bias, and consequently represents both perceived and reported status. B1-B5 represents the varying gradations of bias and represent reported status. From these results we conclude:

* With increasing project risk, project manager errors affect reporting distortion very little; for the high-risk project, status errors do not affect what is reported;

* The higher the risk project, the higher the amount of reporting distortion for each incremental increase in bias;

* At high levels of bias, project manager errors affect reporting very little, irrespective of project risk; and,

* For low risk project and 30% error, the error is offset by slight bias (B1), resulting in little reporting distortion.

These results suggest that reporting distortion is insensitive to a wide range of project manager perception errors, especially as the risk level of the project increases. This interesting result suggests that in order to receive a more accurate report, an executive manager should focus more on ways to reduce reporting bias, rather than on whether the project manager accurately perceives status.

As these results were derived from our analytic model, we also performed a discrete time event simulation to verify the results shown in Exhibits 5 and 6. The purpose of this simulation was to ensure our model was producing proper results. First, sets of state transition probabilities that produced the predicted true results were found for the low-, medium-, and high-risk projects (an example of which was shown in Exhibit 4). Combining these state transition probabilities with the various error and bias channel representations, we were able to produce reported status results very close to those produced by the analytical results through a simulation program written in C++. In fact the simulation's (mean, maximum, minimum) deviation from the model for BO through B4 for both the low-risk and high-risk project is (-0.26%, +0.82%, and -1.73%). This comparison verifies the analytic results presented in this article.

Perceived and Reported Status of Low-, Medium-, and High-Risk Projects. Discussions with experts indicate that the riskier the project, the more error-prone the project manager will be in perceiving project status. The true, perceived and reported status for low-, medium-, and high-risk projects are shown in Exhibit 7. Here, we assume that a low-risk project encounters a 10% error channel, a medium-risk project a 30% error channel, and a highrisk project a 50% error channel, which were introduced in the previous section. Future research will address quantitative expert assessments of errors for different project risks. From these results, we observe that: (1) project manager perceptions for low- and medium-risk projects are pessimistic, while perception of high-risk projects are optimistic; and, (2) with bias, there is little difference in reported status for medium- and high-risk projects.

The first observation suggests that low- and medium-risk projects may benefit from the project manager's sense of pessimism regarding their status. Specifically, this pessimism may trigger more management attention on these projects. Conversely, project managers may be "lulled" into complacency with high-risk projects.

Chances of Red Projects Being Reported Red. Lastly, we present joint probabilities to investigate how a project is reported when it is actually in the Red state. For any channel, the chance of a project actually being in the Red state, and being reported as Green, Yellow, or Red, all add up to the chance the project is truly Red (0.22 for low risk and 0.44 for high risk). The chance of a lowand high-risk project in the Red state being reported as Green, Yellow, or Red is shown in Exhibits 8 and 9, respectively. These results are self-verifying, as the probability at any point adds up to the truly Red probability. From such results the following observations are made about Red projects:

* When the low risk project is Red, it is almost always perceived as Red;

* When the high-risk project is Red, it is perceived to be Red only about 50% of the time and Yellow about 40%;

* For each project type, a Red project with slight bias (B1) is most likely reported Yellow;

* With even slight bias (B1), there is less than one chance in four that the project is reported Red; and,

* For each project type, when a project is Red and bias is modest (B2), it is most likely reported Green and least likely to be reported Red.

These results indicate that Red projects are infrequently reported Red. This may explain why executives are often surprised when projects fail. Indeed, the results indicate that Red projects-- those that should be on the executive's radar screen-are most often reported Green! Interestingly, this type of reporting distortion can occur with only a modest amount of bias. These results strongly suggest that more attention should be given to creating environments where bad news can be reported.

Conclusions

We have introduced a simple model to assess combined effects of project manager misperceptions and bias on the reliability of software project reporting. This combined effect, that we call reporting distortion, is demonstrated to be greater for higher-risk projects. We also show that if the goal is accurate reporting, minimizing bias is more important than minimizing project manager misperceptions regarding status. This result is significant because much of the previous literature has focused on traditional project management and control techniques for obtaining accurate status information. What our model demonstrates is that more attention needs to be placed on the effect that bias has on the reliability of the reported status.

In summary, our research and model results suggest the following:

* Traffic light status reporting, commonly encountered in software development, allows construction of a simple yet powerful software status reporting model;

* Executives should be skeptical of favorable status reports early in the development cycle;

* Executives should assess the risk level of individual projects, because the reliability of status reporting decreases with increasing project risk;

* Both errors (i.e., project managers' misperceptions of true status) and bias can lead to significant reporting distortion, resulting in unreliable project status reports; and,

* As the level of project risk increases, the impact of project manager bias completely swamps the impact of errors in explaining the distortion that occurs.

These results have promising implications for both researchers and practitioners, and offer new insights for improving software project management.