A Prioritization Process

Why Is Prioritization Important?

To help select a new car to purchase, assume that you will use evaluation criteria of five-year life-cycle cost, horsepower, and safety. The five-year life cycle cost (in U.S. dollars) includes purchase price, taxes, licenses, loan interest, insurance, gasoline, and maintenance. The horsepower is the peak SAE net horsepower. The safety rating is O to 5 stars based on the National Highway Traffic Safety Administration's front and side crash test and rollover ratings. Assume that you have narrowed the field to three cars. One is best on five-year life-cycle cost, the second car is best on horsepower, and the third car is best on safety. Which car should you buy? Your answer obviously depends on which criteria are most important to you. To help you make this decision, it would be nice to have a prioritization process. This article describes such a process.

Development of this prioritization process was stimulated by a need to prioritize requirements, but the resulting prioritization process is not limited to requirements. It can also be used to derive weights of importance for tradeoff studies and to prioritize customer needs, capabilities, risks, activities, and functions. First, we will discuss some of the reasons that requirements should be prioritized. Then, we will discuss prioritization of other items and, finally, we will present the prioritization process.

A requirement is a statement that identifies a capability needed by a system in order to satisfy customer needs. A functional requirement defines what, how well, and under what conditions one or more inputs must be converted into one or more outputs at the boundary in question in order to satisfy the customer's needs. A customer's need might be to solve a problem, achieve an objective, or satisfy a contract, standard, or specification (Bahill and Dean, in press). For small projects the requirements might be put in an Excel spreadsheet; for large projects, the requirements might be put in a requirements database. With either implementation, one of the attributes must be priority.

Why should you prioritize requirements?

1. If the project is budget-constrained, prioritization will help you decide which requirements should be implemented and which should be candidates for elimination.

2. If the project is time-constrained, prioritization will help you decide which requirements should be implemented first. Often the product is delivered in phases. At each delivery, the system must have testable functionality. Prioritization helps you choose the functions to implement in each phase.

3. Prioritizing scenarios and identifying benefits, costs and dependencies will help create the system architecture.

4. Prioritization improves customer satisfaction by increasing the likelihood that the customer's most important requirements are implemented and delivered firstcustomers like to see their funds being used effectively and wisely.

5. Prioritization will allow you to spend more time and effort reducing risks associated with hard technical problems and key performance parameters.

6. You might want to assign your best people to the highest priority requirements.

7. Prioritizing requirements will help you manage requirements creep. If requirements being added are high priority, then they might displace some lowpriority requirements.

8. Prioritizing requirements will reduce discussion time at meetings and reviews.

9. Prioritizing requirements will help identify the highpriority requirements for which you should create Technical Performance Measures (TPMs) (Oakes, Botta, and Bahill, 2006).

Some requirements are more important than others; therefore, requirements must be prioritized. Build the most important features of the system first, making the critical functionality available to the users as soon as possible and leaving the less important features for future releases. Requirements prioritization enables implementation of the highest priority requirements first. To effectively prioritize requirements, you should consider risk, criticality to mission success, customer satisfaction, commitment, architecture, business value, priority of scenarios, benefit, cost, benefit to cost ratio, implementation time, when it is needed, frequency of use, safety, complexity, implementation difficulty, stability, dependencies, infrastructure, and reuse potential. Individual requirements can be prioritized or requirements can be grouped into functional categories and these new categories can be prioritized.

Prioritization is a negotiation process that involves a wide range of project stakeholders, including the customer, user, project manager, chief engineer, architect, maintainer, etc.; however, it is ultimately up to the customer to determine which requirements are more important, but obviously, the contractor is responsible for working with the customer to define the relative importance. This means that the contractor's requirements team cannot prioritize requirements in a vacuum-it must be done in conjunction with the customer. But getting the customer to prioritize requirements may not be as easy as it seems.

The real value in prioritizing requirements comes when you have tight delivery schedules, staffing shortages and/or budget constraints. The value is delivering something that is useful to the customer even if you do not deliver all the requirements, since you have ensured the most important requirements are addressed.

Requirements should be prioritized early in the system life cycle. Setting relative priorities during the requirements development phase:

1. Reveals to the contractor what the customer deems important

2. Helps balance customer expectations against available resources

3. Helps produce realistic schedules

4. Supports design tradeoff decisions

5. Helps formulate the system architecture.

If features are planned for future development, then prioritization facilitates architecture such that adding lowpriority features later will not require redesign of the architecture. The product can be developed incrementally with high-priority requirements in early versions and low-priority requirements in later versions (Hooks and Farry, 2001).

The preceding paragraphs specifically discussed requirements prioritization; however, prioritization is also applicable to deriving weights of importance for the criteria in tradeoff studies and to prioritizing goals, customer needs, capabilities, risks, directives, initiatives, issues, activities, use cases, technical performance measures, features, functions and value engineering activities. For these other tasks, some things will be different. For example, in prioritizing design activities, external constraints or changes in the intended operational environment may make some activities infeasible; therefore, they will be dropped. This is possible but not likely with requirements. In deriving weights of importance for the criteria in tradeoff studies, cost might be treated as an independent variable and hence not be in the criteria set for the tradeoff study. An important purpose in prioritizing requirements and design activities is scheduling the work-this is not true for prioritizing technical performance measures where all TPMs are managed throughout. One purpose of prioritization is to identify features that should be candidates for elimination; this is much more common for goals, customer needs, and capabilities than it is for requirements.

When a project is nearing the end of a development phase (or spiral, iteration, time box) and it appears that you cannot produce all of the features that were scheduled during an iteration, do not slip schedule and delay the review until all the features are finished. Instead, push some low-priority features into the next iteration. Prioritization of features helps determine which features get delayed. This approach obviously depends on the contracted life cycle model. Furthermore, low-priority features should be renegotiated with the customer to see if they are really necessary. Changes to customer funding profiles may force lowpriority features to be deleted.

All features will not be implemented equally-some will get extra-special polish. Companies often seek advice from outside consultants for the highest priority features. Also, they often assign their best people to the high-priority features and contract out the low-priority ones. As a noncommercial example, consider Little League baseball: the most important positions are pitcher and catcher. So where do coaches put their best athletes? Pitcher and catcher. Furthermore, the highest priority features might be subjected to more reviews and more thorough testing. Even if the extra testing is not planned, it will happen because of the nature of regression testing. In regression testing, at the end of a life cycle phase all completed features are tested. In the next phase, more features are added. And at the end of this phase, all competed features are tested-those completed in the second phase as well as those completed in the first phase. Thus, features finished first will be tested in every iteration.

Priorities will change as you talk with your customer and gain a better understanding of your customer's needs, as the environment changes, as the stakeholders change, as various features are implemented, as the system matures as its architecture develops, and as uncertainty is resolved; therefore, the priorities of all features should change with time (GiIb and Maier, 2005).

Criteria That Help Prioritization

Exhibit 1 lists criteria that are useful for prioritization. Cost is obviously an important criterion in most decisions. Cost should include money as well as other resources such as time, labor, finances, overhead, infrastructure, shipping, etc. In purchasing a new car, cost would include purchase price, taxes, licenses, loan interest, insurance, gasoline, and maintenance. Benefit is a measure of the good things that accrue due to acquiring a feature. This would include performance measures such as speed, mean time between failures, requests served per minute, market percentage, quality, convenience, testability, accuracy, etc. In purchasing a new car, horsepower is an important performance measure. Some criteria naturally go together, like peanut butter and jelly; therefore, cost and benefit are often combined into the benefit to cost ratio. Putting cost in the denominator gives low priorities to high-cost, low-value features. Of course, if you are using the benefit to cost ratio, then you should not include cost and benefit as separate criteria.

In an ideal world, criteria used for prioritization would be completely independent; however, for modern complex systems this is not possible. The criteria in Exhibit 1 are meant to be orthogonal and as independent as possible. It is important that you do not look for derived effects. For example, features that affect the system architecture should be given a high priority because, in general, features that are likely to cause a lot of changes in other systems should be given a high priority; however, when assessing features that affect the architecture, do not derive the conclusion that they subsequently increase cost and risk and, therefore, give them low priorities. If you continually look for interactions, you will never finish the prioritization process.

The criteria of Exhibit 1 are not listed in any particular order, although the top of the list has criteria that are more general. Of course, the criteria to be used must be tailored for the particular company and for the type of business. All of the criteria in Exhibit 1 would not be useful in all industries, and other criteria would have to be added for some industries. Some customers may allow no flexibility in requirements or schedule. This article is written for people who have flexibility in when and how well various features are implemented; however, a list should not be prioritized if the cost of prioritizing is not far less than the cost of doing the tasks. For example, for most people, it would not make sense to prioritize a grocery shopping list.

Deriving Values for the Criteria

The prioritization process consists of first deriving values for all the criteria for all the features and then combining the data to reveal the priorities. Common criteria scales include:

a. Low, medium, and high

b. Optional, conditional, and essential

c. Nice-to-have, goal, highly desired, and must achieve

d. Numeric (e.g., O to 10)

It is very important, however, to use the same range for all criteria. You should not use a range of 1 to 3 for one criterion and O to 10 for another (Bahill and Karnavas, 2000). Obtaining a consensus on criteria values might require a group decision support technique, e.g., voting, Delphi, the analytic hierarchy process, or specialized facilities and software.

In an ideal world, to get the criteria values and priorities, you would first talk to the customer, but the following sequence is more realistic: The systems engineer assigns straw man values to all the criteria for all the features. These values are typically numbers (usually integers) in the range of O to 10, where 10 is the most important. The next step is to meet with specialty engineers and domain experts. The systems engineer should lead a discussion of each criterion in Exhibit 1 and try to get a consensus value for each feature. In the first pass, the engineers might evaluate each criterion and its context and then take the average value. After the in-house evaluation, the prioritizations should be taken to the customer (however many people that might be). The chief engineer should lead a discussion of each criterion in Exhibit 1 and try to get consensus values for all the criteria for all of the features; however, if the customer only looks at one or two criteria and says the feature is a 10, then it's a 10. If the customer says that all criteria are very important, just continue with the process, because later on in the process the evaluation data may prioritize the features.

Of course, as with all systems engineering processes, prioritization is not a waterfall process. It is highly iterative and many tasks can and should be done in parallel. In the beginning of a program, no one generally has a good understanding of the complexity, dependencies, and reuse potential. As knowledge about the system is developed, the prioritization process will be refined. Prioritization is a communication tool-the numbers that are derived are not as important as the understandings. There are alternatives to the above procedure:

1. Instead of assigning a number between O and 10, the systems engineer, in conjunction with the customer, could rank all the features. Sometimes this technique works in spite of being methodologically flawed. It is flawed because we are adding the weighted scores; therefore we need cardinal numbers (e.g., if feature A gets a score of 6 and feature B gets a score of 3, then feature A should have twice as much worth or utility as feature B), not ordinal (as in rank order) numbers.

2. The systems engineer can help the customer make pair-wise comparisons of all the features and then use the analytic hierarchy process to derive the values (Saaty, 1980). This would not be a practical approach without a commercial tool such as Expert Choice.

Tools that implement the analytic hierarchy process add value by producing a consistency index that shows how consistent the pair-wise comparisons were. For example, if the domain expert said that A was preferred to B, and B was preferred to C, then we would expect him or her to say that A is preferred to C. The consistency index indicates how consistent the comparisons were throughout the entire matrix.

Normalization

Values for the criteria could come in a variety of formats, for example, (low, medium, high), (0 to 10) or natural units that might run, for example, from one thousand to one million dollars. In order to combine apples and oranges like these, the values must be normalized.

The values can be normalized with scoring (utility) functions (Daniels, Werner, and Bahill, 2001) so that all of the resulting scores are between 0 and 1. Exhibit 2 shows a typical scoring function for the cost criterion: higher cost gives a lower score. A simple program for implementing such scoring functions is available for free at http://www.sie.Arizona.edu/sysengr/slides/ SSEzip. If scoring functions are thought to be too complex, then simple linear normalization (as explained in the following paragraphs) will work.

Deriving Weights of Importance

There are a dozen methods for deriving numerical values for the weights of importance for the evaluation criteria (Buede, 2000; Daniels, Werner and Bahill, 2001; Kirkwood, 1999; Weber and Bor cher ding, 1993). These methods can be used by individuals or teams. If the decision-makers are subject matter experts and simple qualitative comparisons will be made, then it is often sufficient to just ask the decision makers, "How important are each of these criteria? Give each a number between 1 and 10." We would not expect a domain expert to give a weight of O; however, a weight of O can be given to criteria that have no effect on the output-but whose consideration should be made prominent. Later the weights can be normalized so that they sum to 1. When the output values will be used for numerical comparisons in complex high-risk situations, then more quantitative methods might be useful. When the weights are to be assigned using both the decision makers' relative importance and the expected range of input values, then the method of swing weights (as explained in the following paragraphs) can be used. Creating two sets of weights might be useful-one from the customer's perspective and the other from the contractor's perspective.

The Method of Swing Weights

Let us now explain one particular method-the swing weight method-using our example of selecting a new car. As evaluation criteria we will use five-year life cycle cost, horsepower, and safety. The five-year life cycle cost (in U.S. dollars) includes purchase price, taxes, licenses, loan interest, insurance, gasoline, and maintenance. The horsepower is the peak SAE net horsepower. (The Horsepower to Weight Ratio, however, might have been a better criterion.) The safety rating is 0 to 5 stars based on the National Highway Traffic Safety Administration's front and side crash test and rollover ratings. Exhibit 4 has values for some typical cars.

Next, we need to determine the range of each criterion. As mentioned above, there are several choices for the range. For this example, let us use the real data from Exhibit 4. For the three cars that we are examining, the maximum and minimum five-year life cycle costs are $52,000 and $22,000 (Exhibit 5).

Next, let us take horsepower. The three cars that we are examining have a minimum horsepower of 170 and a maximum of 290 (Exhibit 6).

Our third criterion is safety. The three cars have minimum and maximum values of 3 and 5 stars (Exhibit 7).

We now have definitions and ranges, measured from worst to best, for each of the three criteria that matter the most to us for selecting a new car. Other characteristics such as color or type of transmission may also be important considerations in the choice of a car; however, we are assuming that on all these other criteria the differences between the cars from which you are choosing are unimportant. This does not mean that these other characteristics do not matter, but only that, in the context of this choice, they are unlikely to vary sufficiently that you will have to make explicit tradeoffs among them.

Now imagine a hypothetical car that is the worst it can be on all three criteria. In other words, its five-year life cycle cost is $52,000, its horsepower is 170, and its safety rating is 3 stars. Suppose that you can change the value of one (and only one) of these criteria on this hypothetical car from the worst to the best. This means that you can change only one of the following:

* Five-year life cycle cost from $52,000 (worst) to $22,000 (best)

* Horsepower from 170 hp (worst) to 290 hp (best)

* Safety from 3 (worst) to 5 (best).

Which one would you want to change? Suppose you say five-year life cycle cost. That means that you value a $30,000 drop in price (a change from $52,000 to $22,000) more than you do either an increase of 120 horsepower or an increase of 2 stars of safety. This criterion, the one that you most want to change from worst to best, is the one you weight most highly in the context of this problem. Assign it a score of 100 points.

Now, which criterion do you value second? Let us say it is horsepower. Ask yourself, "How much less do I value the 120 horsepower change compared to the $30,000 drop in price?" Suppose that you value it one-half as much. Then you would assign it 50 points, or half the weight you gave to the most important criterion.

Now look at the last criterion-safety. Because this criterion is ranked below horsepower, it should get fewer points. For example, if you value it two-thirds as much as horsepower, then give it 33 points. Note that this also means you are saying that safety, with its 33 points, is only one-third as important for this decision as the five-year life cycle cost. All that remains is to normalize the weights so that they add up to 1, as shown in Exhibit 8.

Combining the Data

Now that we have values and weights of importance for the criteria we must combine them. There are dozens of methods for combining these data (Daniels, Werner and Bahill, 2001). The most common is the simple sum of weights multiplied by criteria values. This additive method is appropriate when the decisionmakers' preferences satisfy additive independence (Keeney and Raiffa, 1976), which is the case for most industry applications we have seen. This method is implemented with Equation 5 to compute the priority of the 1th feature.

This simplistic technique uses very simple mathematics. The weights and the criteria values are all integers between O and 10. Usually this technique gives good results.

The criteria to be included mustbe tailored for the individual decisions. Criteria that do not differentiate between alternatives can be omitted or be given a weight of zero. Other criteria, such as security and resources (in addition to the already included cost and time resources), might be added. IBM has a criterion of, "Can it be implemented in the future as easily as it can be added today?"

Of course, this prioritization process is iterative. The first pass will show the most important features. These features might then be scheduled to be implemented first, which would change their values in the When it is Needed criterion.

This simplistic technique usually works because the purpose of prioritization is communication. The numbers themselves should not be used in numerical calculations. If the priority values are to be used for calculations, then a more sophisticated technique must be used.

Using Normalized Data

If the priority values are to be used in calculations, then the weights of importance must be normalized so that the sum of the weights is 1.0, and the values of the criteria must also be normalized with scoring (utility) functions (Daniels, Werner and Bahill, 2001) so that the scores for the criteria are all between 0 and 1.

The wt^sub Benefit^ and wt^sub Cost^ in Equation 9 are indeed exponents, as they must be for the product-combining function (Daniels, Werner and Bahill, 2001).

Other Fields That Use Prioritization

DARPA's Image Understanding programs use change detection algorithms to prioritize imagery for exploitation by image analysts (Jackson and Pierce, 2002). Cognitive decision models have been used to prioritize e-mails (Lee, Chandrasena and Navarro, 2002). Knowledge management activities have been prioritized using a matrix with levels of potential intervention (goals, knowledge, business processes, and data) and scopes of intervention (individual, team, organization and business environment) (Bornemann and Sammer, 2003). Google prioritizes the search entries that it presents to the user.

Prioritization is used extensively in the medical field. One application is to prioritize prevention strategies. For example, of the four dozen risk factors for cataracts, the highest priority prevention strategies are to avoid tobacco smoke and avoid UV-B by using shade, sunglasses, and brimmed hats (McCarty, Nanjan, and Taylor, 2000).

In the value engineering improvement technique, the value of a function is defined as the ratio of a function's worth to its cost. The goal is to increase value while maintaining quality either by improving the function or reducing its cost. Value engineering should use a prioritization process, to ensure that the most important functions are worked on first.

Other Methods for Prioritization

Other methods that have been used for prioritization include Quality Function Deployment (Bahill and Chapman, 1993; Ghiya, Bahill and Chapman, 1999) and the Analytic Hierarchy Process (Saaty, 1980).

Prioritization is different from performing a tradeoff study. In a tradeoff study, the criteria are specific to the problem domain (Daniels, Werner and Bahill, 2001). In the prioritization process presented in this article, the same criteria (with some tailoring) should be used for all prioritization tasks. The purpose of a tradeoff study is to select one or a few alternatives from a large list (Smith, Son, Piattelli-Palmarini, and Bahill, 2007). The purpose of prioritization is to prioritize the whole list and possibly eliminate a few features. Scheduling the development of features is not a purpose of tradeoff studies.

Completeness of Coverage

In retrospect we asked ourselves if the criteria of Exhibit 1 would give an even coverage of an organization's aspirations and needs. To answer this question we looked at frameworks.

Frameworks help people organize models of their enterprises. This organization helps ensure interoperability of systems and helps control the cost of developing systems. The Zachman framework provides a general schema that can be used to organize and assess completeness of descriptive representations for a complex enterprise. To ensure a complete and holistic understanding of the enterprise, it is necessary to develop models that address the perspectives and aspects that constitute the rows and columns, respectively, of the framework (Bahill, Botta and Daniels, 2006).

Therefore, we mapped the criteria of Exhibit 1 to the rows and columns of a Zachman framework. Criticality to Mission Success is in row 1, the scope of the organization. Business Value and Benefit to Cost ratio are in row 2, the business model. Architecture is in row 3, the system model. Complexity, Implementation Difficulty and Dependencies are in row 4, the technical model. Finally, Reuse Potential is in row 5, the detailed representation. Now for the columns, in a risk analysis, you look at the high-risk entities and ask, "What could possibly go wrong?" Hence Risks are in column 1-What. Implementation Difficulty quantifies how things are done; hence it is in column 2-How. Architecture describes the positioning and interconnection of things, so it is in column 3-Where. Commitments are made to people, therefore Commitment goes into column 4-Who, along with Customer Satisfaction. When it is Needed is in column 5-When. Business Value and Benefit to Cost Ratio have to do with the motivation of people and the organization, so they are in column 6-Why.

So, although the criteria of Exhibit 1 do not fill all 36 cells of a Zachman framework, they do cover each row and each column. Therefore, the criteria of Exhibit 1 should cover the aspirations and needs of most organizations.

Future Work

We would like many intelligent people in many domains to use this prioritization process and then give us feedback, such as the list of criteria that they actually used along with their weights of importance. We want to know the domain in which it was used, such as DoD, commercial aerospace, NASA, medicine, government, or law, and the type of items that were prioritized, such as requirements, functions, or activities. Proprietary data is not desired, such as the actual items that were prioritized or their values. We will use this feedback to expand and refine our criteria set. Perhaps we will generate different criteria sets for different domains.

Summary

Requirements, goals, customer needs, capabilities, risks, directives, initiatives, issues, activities, features, functions, technical performance measures, and weights of importance for the criteria in tradeoff studies should all be prioritized. Prioritization will help with budget, schedule, system architecture, customer satisfaction and risk reduction. Prioritization can be very simplistic using integers from O to 10 for the weights and the values, or it can be robust, using normalized weights of importance and scoring functions for the criteria. Sometimes it is useful to create two sets of priorities-one from the customer's perspective and the other from the contractor's perspective. Exposing these conflicting objectives often explains existing misunderstandings.