A Structured Approach for Assessing the Effectiveness of Engineering Design Tools in New Product...

New product development (NPD) has become increasingly important to the financial success of many organizations (Barclay, 2002). Cooper, Edgett, and Kleinschmidt (2003, 2004) reported that new product introductions in the last three years accounted for 30% of corporate sales. It has also been estimated that expansions in new product introductions will soon account for 40% of corporate profitability (Rahim and Baksh, 2003). Further, the overall number of new product introductions is expected to double in the next five years compared to the last five years (Barclay, 2002).

Over the past 50 years, attention in the NPD literature has shifted from investigating how idea generation can be stimulated, to identifying key characteristics of successful products, to comparing new product introduction successes and failures, and more recently to understanding what separate elements (tools, activities, etc.) make the NPD process successful as a whole (Barclay, 2002). Rahim and Baksh (2003) reviewed some of the NPD frameworks in the literature. These frameworks have a number of common features, but they vary in terms of the level of detail, the extent to which certain NPD tools are prescribed, and the project management activities specified. Companies lack standard assessment instruments to provide detailed feedback on the effectiveness of the NPD frameworks, tools and activities prescribed in the literature. Meanwhile, Bernasco, de Weerd-Nederhof, Tillema, and Boer (1999) stated that companies attempting to improve their NPD process should perhaps put most effort into "developing theories and tools supporting the analysis and re(design) of NPD processes."

The design phase of NPD appears particularly vital to overall project performance in reducing engineering changes, production time, and overall cost (Rahim and Baksh, 2003). Despite the importance of the design phase, Rahim and Baksh (2003) argued that it is often carried out in an ad-hoc manner. Frequently, each individual engineer has his or her own method of applying design tools, which can create undesirable variation in the design process and, ultimately, problems in manufacturing.

This article describes the development of the assessment approach, which can be used to study the effectiveness of NPD activities, and illustrates its application in an engineer-to-order (ETO) manufacturing organization. At the time of this research, the case study organization was deploying a redesigned NPD process and was concerned about how well project teams were using key design tools in the new NPD process. The measurement instruments developed using the approach, as well as results from the assessment of two design tool applications, are presented.

Background on the Case Study Organization

The case study organization is one business unit of a global manufacturing company that designs and manufactures motors for the motion control industry, with customers in the machine tool, medical products, aerospace and defense industries, and numerous other markets. As indicated, the business unit described in this article operates as an ETO organization with extensive NPD activities within the overall order fulfillment process. NPD projects are executed for new designs as well as product redesign and extensions based on either internal needs (e.g., need to reduce product cost) or external needs (e.g., customer requests for additional product functionality). NPD projects vary in size, with the typical major project taking one to two years.

As mentioned at the start of this research, the business unit, which has several facilities dispersed worldwide, had recently undertaken a strategic initiative to create one standard NPD process to utilize within all its design centers. Previously, project teams across different design centers in the business unit followed their own unique version of the NPD process, although these processes did have some common features. Business unit leadership determined that the organization needed to develop a customized NPD process to fit its unique needs, rather than using an "off-the-shelf" process from the NPD literature. This conclusion is logical because most NPD processes in the literature maybe more aligned with the needs of make-to-stock companies than those of ETO organizations (Rahim and Baksh, 2003). The business unit's new NPD process clearly defined the phases (or stages) of the NPD lifecycle, which all projects must complete. In addition, tools to be used in each stage, as well as the desired outputs (deliverables) of each stage, were also clearly defined. For some parts of the business unit, the new process was not a significant departure from current practice, but for others it was a significant change in terms of structure and rigor.

Business unit leadership wanted to ensure that the new NPD process was successfully deployed by all project teams, and in particular expressed concern about the appropriate and consistent application of the engineering design tools prescribed within the process. The organization's experience with some design tools suggested that project teams did not always apply the tools according to the prescribed process. In monitoring the NPD process deployment, leadership wanted to assess whether project teams were using tools correctly, rather than just whether the teams were using a specific tool (which would be a "yes/no" binary assessment approach). In addition, there was a need for assessing the quality of the output of the design tools. Thus, the organization needed a method of assessing the quality of the design tool application process, as well as the quality of output generated through tool application.

Design of Assessment Approach

Different frameworks have been developed to help organizations improve specific parts of their business (Ibrahim, 2002). Within the area of systems engineering, these frameworks can be categorized based on their approach toward improvement as either standards or capability maturity models. Standards such as ISO/IEC 15288:2002, EIA-632, and IEEE 1220-1998 prescribe with varying levels of detail "what to do." Meanwhile, capability maturity models such as the systems engineering capability model and the capability maturity model integration (CMMI) provide specific guidelines on "how to do things better."

This research included a review of different NPD frameworks to identify an appropriate foundation for assessing the effectiveness of design tool use within the case study organization. The framework that seemed to most closely match the organization's needs was the "continuous representation" (or "process capability approach") of the CMMI (CMU/SEI-2002-TR-003, 2001). The continuous representation of the CMMI suggests specific goals and practices for improving individual process areas (such as requirements management, project planning, etc.). This representation also presents a generic approach for assessing the process capability of any given process area. As shown in Exhibit 1, the continuous representation of the CMMI describes several generic goals which the given process areas should achieve at different capability levels, as well as the generic practices which should be used to achieve these goals.

Assessing the capability of any given process area using the continuous representation of the CMMI therefore includes evaluating both the output quality of that process area (assessing the achievement of specific goals through the application of specific practices) and the quality of the process used in applying the specific practices (assessing the achievement of generic goals through the application of generic practices).

The CMMI framework clearly met the case study organization's need to assess both the output and process quality of design tool use, and was used to design the assessment approach presented here. For any activity being assessed, this approach calls for the development of two types of assessment instruments. First, a Process Quality Assessment Instrument is developed to assess the quality and capability of the process used in applying the NPD design tool. This instrument is used to help determine whether the design tool application process prescribed by the organization is being utilized and successfully institutionalized. Second, an Output Quality Assessment Instrument is used to assess the quality of the output(s) generated from applying the NPD design tool. This instrument is used to help determine whether the specific outputs of NPD design tools met their goals. The Process Quality Assessment Instrument was designed to be generic to all design tools applied within NPD, while the Output Quality Assessment Instrument was designed to contain content specific to each NPD design tool.

Example of Use of Assessment Approach

The assessment approach was piloted for two engineering design tools that anecdotal evidence suggested were problematic for the case study organization, as well as vital to NPD success. The two tools are failure mode and effects analysis (FMEA) and production preparation process (3P).

Need for FMEA Assessment

FMEA is a risk management and quality improvement methodology that helps organizations quantify, prioritize and take action against potential causes of failure in products or processes (Ben-Daya and Raouf, 1996). Often, an FMEA application that focuses on product design is referred to as a "Design FMEA" or "DFMEA," while an FMEA application that focuses on the design of the product's manufacturing process is referred to as a "Process FMEA" or "PFMEA."

The basic FMEA methodology includes assembling a cross-functional team of subject-matter experts (SMEs) who have experience in one or more key features of the product or process. The team uses brainstorming and cause-and-affect analysis to identify potential failure modes, effects of each failure mode, and potential causes of each failure mode. For each cause the likelihood of occurrence (S), probability of non-detection (S^sub d^), and severity (S) ave rated. Ratings are generally on a 10-point Likert-type scale, using either standard ratings tables or ratings tables customized by the organization. A risk priority number (RPN) is then calculated from the occurrence, detection and severity ratings as follows (Equation 1).

RPN = S^sub f^ xS^sub d^ x S (1)

Risks are generally ranked for action according to the magnitude of their RPNs (i.e., the risks with the highest RPNs are addressed first), although a number of qualitative rules-of-thumb exist, such as addressing all risks with a severity rating of "9" to "10" or an occurrence rating of "7" to "10" regardless of overall RPN.

It has been suggested that the use of FMEA leads to reduced cost and product time to market, improved product and process quality and reliability, increased safety and, ultimately, increased customer satisfaction (Dale and Shaw, 1990); however, several issues exist in regard to FMEA application. First, Devadasan, Muthu, Samson, and Sankaran (2003) contended that most organizations have not achieved full integration of FMEA into their product/process improvement system (that is, that most organizations have not achieved "maximum" quality of FMEA application). Second, most research on FMEA appears to center on modifications to the FMEA methodology. There appears to be relatively little research on the effectiveness of FMEA application within organizations. Third, although qualitative guidelines for FMEA applications (checklists, etc.) appear inpractitionermanuals and guidebooks (e.g., FMEA Student Guide [DRM Technologies, Inc.]), there appear to be no research-based measurement instruments designed to provide detailed, quantitative feedback on FMEA effectiveness.

In addition to the questions raised in the literature, the need to assess FMEA effectiveness was also evident in the case study organization. Qualitative post-project evaluation sessions suggested that FMEA may have a critical role in warranty returns after product launch. Meanwhile, anecdotal organizational experience suggested that the quality of FMEA application was inconsistent throughout the organization. Thus, FMEA was a natural choice as one of the first design tools that the case study organization targeted for evaluation using the assessment approach.

Need for 3P Assessment

3P is a design quality improvement strategy that uses a structured brainstorming technique to guide a cross-functional team of SMEs in developing and evaluating design alternatives. The 3P process involves brainstorming designs based on examples from nature, developing and testing design mock-ups, and repeatedly evaluating and narrowing down design alternatives until the best alternative is selected. Although 3P is a tool within the well-known Toyota Production System (TPS), 3P has received relatively little attention in either the research or the practitioner literature (Hampton AutoBeat LLC, 2002); however, companies which do utilize 3P claim to have achieved substantial performance improvements through its use (Hampton AutoBeat LLC, 2002; Waurzyniak, 2005). Thus, improving 3P quality would likely have a significant impact on overall NPD quality. Like FMEA, post-project evaluation sessions suggested 3P quality was critical to post-launch quality. Thus, 3P was another natural choice for piloting the assessment approach.

Development of the Process Quality Assessment Instrument

As mentioned previously, the Process Quality Assessment Instrument was defined to contain the same items for all design tools, thus creating a standard approach for measuring the quality of the tool application process across different design tools. The assessment items are based on the generic practices required to achieve CMMI capability level 2 (refer to Exhibit 1), which represents a "managed" process. These assessment items were selected because CMMI level 2 represented the appropriate level of capability improvement for the organization's NPD process, based on evaluation of current practices. The first eight of the generic practices from the continuous representation of CMMI for capability level 2 ("managed" process) were selected as being applicable for inclusion in the Process Quality Assessment Instrument. The last two practices were not included, since they referred to management activities aimed at improving a given process, rather than dimensions of the given process's current quality. The eight selected practices were converted to descriptive survey questionnaire items, which were designed to be answered on a 10-point scale (1=not at all and 10=to a great extent). In the conversion of the generic practices identified in the CMMI at the capability level 2 to survey questionnaire assessment items, additional detail was added whenever needed. For example, the CMMI capability level 2 generic practice "provide resources" (see Exhibit 1) became the survey item "sufficient resources were allocated for executing the process (e.g., materials, facilitation support, sponsor support, etc.)."

The final set of assessment items in the Process Quality Assessment Instrument was pilot tested with internal expert NPD design tool users identified by the organization. Based on feedback from expert users and organization management, six additional non-CMMI items were added that were of particular interest and concern to the organization. For example, an item was added to assess the extent to which key input documents and information were of good quality. Another item was added to assess the extent to which input documents and information were used effectively.

Exhibit 2 shows the assessment items as worded in the Process Quality Assessment Instrument. The Process Quality Assessment Instrument was designed to be completed by all persons participating in a given NPD design tool application because the assessment of process quality would be most accurate with input from the entire tool application team; however, some items that require knowledge of multiple applications of the given design tool are designed only to be completed by participants who are more experienced in the use of the given design tool. Items in the instrument were divided into two sections. Section I contains 11 items (items la through 1k) that ask participants to consider the particular application of the design tool they just completed. All participants are asked to complete Section I. Section II contains four items (items 3a through 3d) that ask participants to reflect on their overall experience in applying this design tool in the company. Only participants who have completed more than one application of the given NPD design tool are asked to complete Section II since these items require broader knowledge of the use of the given design tool within the organization (e.g., "configuration management is conducted effectively for version control of process outputs"). This division of questions was determined based on meetings with organization management. In addition to the items shown in Exhibit 2, the instrument also includes two sets of additional questions to enable further investigation of causal relationships. First, the instrument asks all participants to record their basic demographic information (functional area, job title and number of times applying the targeted tool). Finally, the assessment instrument contains four questions (addressed only to the team leader) that document the organizational resources invested in the tool application (number of people involved, hours per person, and team meeting format-such as a Kaizen event (Melnyk, Calantone, Montabon, and Smith, 1998) or periodic meetings).

Development of FMEA Output Quality Assessment Instrument

The first step in defining a set of potential items to assess the quality of outputs generated by applying FMEA was to review available resources on FMEA. To identify characteristics of effective and ineffective FMEA as practiced in the organization, the resources used as internal training guidelines for the company were reviewed (AIAG, 2000; FMEA Student Guide [DRM Technologies, Inc.]), and employees that the organization described as "expert FMEA users" were interviewed. This combined process resulted in the development of 29 potential assessment items. Because the assessment instrument needed to be feasible in its implementation (i.e., such that using it would not consume a large amount of time), this number needed to be reduced to a smaller set of the most important items. To achieve this, a survey of FMEA expert users was conducted to obtain perceptions on the relative importance of each item for inclusion in the assessment instrument. Each item was rated on a 10-point importance scale (where 1="not at all important" and 10="extremely important"). These ratings, as well as feedback from organization management, were used to define a set of 16 items to assess FMEA output quality. The organization uses both Process FMEA (PFMEA) and Design FMEA (DFMEA) extensively; therefore, it was necessary to identify assessment items that pertained to both DFMEA and PFMEA (nine items), only to DFMEA (four items), and only to PFMEA (three items). Exhibit 3 shows the final list of items in the FMEA Output Quality Assessment Instrument.

The Output Quality Assessment Instrument is intended to be completed by an expert user who is not a member of the FMEA team. This method is used because an external expert user would likely be able to provide more objective and experienced perceptions of output quality than a member of the tool application team. As shown in Exhibit 3, some items were designed to be assessed as a binary scale (yes/no) because they are more straightforward and less subject to varying interpretation. Other items were defined to be assessed on a 10-point scale (1= "not at all" and 10= "to a great extent"). The item ratings can be used to calculate an overall score for the FMEA, which enables aggregation across multiple design tools in the NPD process for a given project and/or aggregation for the same design tool across multiple projects. Detailed ratings (for individual items) can also be used to pinpoint specific areas for improvement.

Development of 3P Output Quality Assessment Instrument

The process used to develop the 3P Output Quality Assessment Instrument was almost identical to the process used to develop the FMEA Output Quality Assessment Instrument. Internal training materials (3P training manuals) were used to identify characteristics of effective and ineffective 3P as practiced in the organization. In addition internal "expert" 3P users were interviewed. From this process, 15 potential assessment items were developed. A survey of 3P expert users to determine their perceptions of the relative importance of the potential items and a follow-up meeting with organizational management were used to define a final set of 12 items. The organization uses both Process 3P and Design 3P. In the Output Quality Assessment Instrument, 10 items pertain to both Design 3P and Process 3P, one item pertains only to Design 3P, and one item pertains only to Process 3R Exhibit 4 shows the final list of items in the 3P Output Quality Assessment Instrument.

Sample Results for Process Quality Assessment Instrument

Following the design of the assessment approach and acceptance by the case study organization, this approach has become part of the ongoing measurement of their NPD process. Example results from the Process Quality Assessment Instrument are described here to demonstrate how the assessment approach, and its supporting instruments, generated valuable insights for the case study organization. At the time of this writing, the organization had used the Process Quality Assessment Instrument to assess the process quality of two PFMEA applications. The first PFMEA was applied to a product that, for the purposes of confidentiality of the case study organization, will be referred to as "PS." The PS PFMEA was completed in February 2005. The second PFMA is referred to as the "CB PFMEA." The CB PFMEA was completed in May 2005.

Both teams were similar in size and had similar response rates. For both applications, team members who had to leave the PFMEA application early (i.e., before it was finished) did not complete the survey. Five out of seven members completed the PS PFMEA assessment (71% response), and six out of eight members completed the CB PFMEA assessment (75% response). The two PFMEA applications were somewhat different in format. The PS PFMEA was completed over two weeks, during which time the team met frequently (twice daily) with breaks in between to complete individual work. PS PFMEA team members invested 50-60 hours apiece. The CB PFMEA was completed in five days in an intensive Kaizen event format. CB PFMEA members invested 30-40 hours apiece.

A comparison of the two PFMEA applications indicates how the Process Quality Assessment Instrument provided useful feedback for distinguishing strengths and opportunities for improvement both within and across teams. The data were analyzed by computing the average response for each item for each team (x^sub avg^), the standard deviation for each item for each team (S^sub x^), and the interrater agreement (James, Demaree, and Wolf, 1993) for each item for each team (r^sub wg^). The interrater agreement is calculated according to Equation 2, where σ^sub E^^sup 2^ is the variance expected due to purely random error (assuming a uniform distribution). This variance is calculated according to Equation 3 (see James, Demaree, and Wolf, 1984), where A = the number of intervals on the measurement scale.

As shown in Exhibit 5, Section I assessed the process quality of the particular PFMEA application, while Section II assessed the process quality of overall PFMEA application within the organization. Because items 3a-3d required that the respondent had participated in more than one PFMEA at the case study organization, the sample size (n^sub 3^) for items 3a-3d is lower than the sample size (n^sub 1^) for items 1a-1k. For the PS PFMEA n^sub 1^ = 5 and n^sub 3^ = 2, and for the CB PFMA n^sub 1^ = 6 and n^sub 3^ = 4. Two exceptions, both for the PS team, are item 1i and item 1k. Only four PS team members answered item 1i, while the fifth team member left the item blank and wrote "none existed." Similarly, only four PS team members answered item Ik, while the fifth team member left the item blank and wrote "too early to tell." Because there were only two raters for the PS team for items 3a-3d, r^sub wg^ was not calculated for these items for the PS team. Instead, for the PS team, "d" is used in Exhibit 5 to indicate the difference between the two PS team members for the given item.

For both teams, the majority of item averages were above 6 on the 10-point scale, indicating a positive orientation for team member perceptions of the presence of most process quality attributes (5.5 is the midpoint on the scale). One exception is item 1 d for the CB team, with x^sub avg^ = 4.20 . This indicates that the CB team perceived that stakeholder involvement in the CB PFMEA was lacking. Conversely, for the PS PFMEA team, x^sub avg^ =8.40 for item 1 d. The difference between the two teams is statistically significant for this item, with p = 0.008 for the Mann-Whitney U-test. For items 1a- 1k, the PS PFMEA had higher responses, except for items 1g and 1i; however, the only statistically significant difference between the two applications at the α = 0.05 level was on item 1d.

For the items in Section II, the teams displayed markedly similar response patterns. For both teams, item 3a had a relatively high average (x^sub avg^ = 7.50 for PS and x^sub avg^ = 8 for CB), indicating that members from both teams seemed to believe that the organizational policy for when to conduct PFMEA is largely clear; however, for items 3b-3d, both teams had low averages (i.e., x^sub avg^ less than 5.50 except for the CB team on 3b, where x^sub avg^ = 5.75) and there was also strong disagreement within the team (which will be further discussed in the next paragraph). This indicates that opportunities for improvements in organizational policies exist in relation to the definition and communication of a standard procedure for PFMEA (item 3b), placing more emphasis on updating the PFMEA throughout the product lifecycle (item 3 c), and using the assessment instruments for ongoing monitoring and improvement of the PFMEA process (item 3d).

Besides indicating areas of strength and opportunities for improvement based on average scores, the Process Quality Assessment Instrument can also be used to identify items for which there is disagreement within the team (due to confusion about organizational policy, etc.). This can be assessed by examining r^sub wg^. The r^sub wg^ values ranee from 0.00 to 1.00. The closer the value is to 1.00, the stronger the interrater agreement, while values close to 0.00 indicate a strong lack of agreement. For instance, the analysis revealed that both teams displayed internal inconsistency in their answers to item 1i (r^sub wg^ = 0.00 for both groups), with at least one member on each team noting that "no formal process exists." This pinpointed communication and training issues within the organization. While top management acknowledged that there was no standard organizational process for PFMEA, it appears that the majority of team members in each application did not realize that the training they had received did not in fact reflect an organizational standard. This confusion and lack of standardization could lead to difficulties on future PFMEAs if a different process is followed. Similarly, both groups had low interrater agreement for items 3b-3d. This is indicated by r^sub wg^ = 0.00 for the CB team and large absolute differences (d) between raters for the PS team (as mentioned, r^sub wg^ was not calculated for the PS team since there were only two raters for items 3a-3d). Another area of low interrater agreement occurred on item If for the CB team (r^sub wg^ = 0.00). In particular, three members of the team perceived a weakness in the team composition-two members gave this item a "4," while a third gave it a "5" (meanwhile, the scale midpoint is 5.5). The other three members rated the team composition positively-two members gave this item an "8," and one gave it a "10." Due to the disagreement within the team, it may be useful for management to investigate the characteristics of this team that may have led to the unfavorable rating by half the team.

Conclusions and Future Work

The contribution of this article has been to demonstrate a generalizable assessment approach to measure the effectiveness of NPD design tool application. This approach was utilized to define two supporting assessment instruments-the Process Quality Assessment Instrument and the Output Quality Assessment Instrument. While the Process Quality Assessment Instrument content was defined to be standard across all design tools, the Output Quality Assessment Instrument is specific to each design tool. This article illustrated the application of the assessment approach for FMEA and 3P applications within the case study organization. As indicated by the example results provided in this article for the Process Quality Assessment Instrument, this assessment approach can provide project managers and other NPD decision-makers with useful and detailed feedback. The approach and supporting assessment instruments can be used to target specific areas for improvement in the process used to apply the design tool (e.g., involving stakeholders, adhering to the prescribed process, etc.) and quality of the outputs of the design tool (e.g., comprehensiveness, etc.). This information will allow the organization to increase the overall effectiveness of applying the given design tool within the organization. Assessment results can also provide essential information to support risk-based go/no-go decisions during the NPD project lifecycle.

The case study organization is currently deploying the assessment approach and supporting assessment instruments throughout all engineering design and development projects in the business unit. Key implementation issues being addressed include defining clear roles for administering the instruments, further refining the instrument content based on initial applications, developing methods for aggregating data across projects, developing methods for integrating assessment results with other existing project evaluation measures, and developing methods for analyzing data over time and across projects.

There are other opportunities for future work in this area as well. First, similar output quality assessment instruments can be developed for other NPD tools, as needed. Second, using these structured assessment approach and supporting instruments will enable the investigation of the impact of NPD design tool effectiveness (i.e., both process and output quality) on NPD downstream outcomes (i.e., product launch effectiveness). In other words, research can be done to examine how design tool effectiveness relates to project launch performance, such as project lead-times (or schedule performance), new product quality (such as defect levels and warranty returns), and product cost. Thus, the premise that using a structured NPD approach, with prescribed design tools, will have a positive impact on product launch success can be empirically investigated. In addition, this feedback will lead to further evolvement of the assessment tools, as they are refined to capture the key elements affecting output quality, as well as additional elements the organization wishes to improve. This feedback is ultimately expected to lead to improved utilization and usefulness of the NPD tools.

The case study organization is excited about using the feedback from the assessment instruments to improve both the structure of its NPD process and its adherence to the process. One of the business unit managers described the situation facing the organization in this way:

"Having spent years on 'Process Improvement' refinements in the [NPD] process, we are now shifting attention to the softer side such as training, process discipline, and review and feedback processes. Evermore, overcoming these execution issues represents the most significant opportunity for additional improvements. Most people attribute deficiencies in NPD to inadequate process definition. Others categorize deficiencies as implementation issues such as insufficient training or adherence to the process. The reality is, just as in football or any other sport, academic, or business endeavor, there must be a reasonable game plan along with matching capability and motivation in order to achieve success."