Account planners' views on how their work is and should be evaluated.

A relatively recent development in the U.S. advertising industry, account planning is aimed at generating strategic insights during three key phases of the campaign process: strategy generation, creative development, and campaign evaluation. Account planners function as a liaison both between

the account executive and the creative department, and between the creative department and the consumer. The structure of planning across agencies varies somewhat: In some agencies the planning function leans more toward account management, whereas at others planners are considered members of creative departments (Kover and Goldberg 1995). Nevertheless, it is generally agreed that planners are the point people in the process of developing message strategy, the "primary contact with the outside world; the person who, through personal background, knowledge of all pertinent information, and overall experience, is able to bring a strong consumer focus to all advertising decisions" (White 1995, p. 18). Kover, James, and Sonne r (1997) characterize planners as "consumer advocates" who represent the consumer perspective during strategy development. It is this intensive interaction with the consumer and creative strategy development that delineates account planners from more traditional market researchers (Barry, Peterson, and Todd 1987; Capon and Scammon 1979).

Account planning, after originating in the United Kingdom in 1965, spread to the United States in the early 1980s--and grew. In 1995, approximately 250 people were employed in U.S. account planning; only four years later, this figure had quadrupled to over 1,000 (Taggart 1999). Account planning departments are now fairly common in the U.S. advertising industry and are found in large full-service agencies as well as medium-sized agencies and smaller creative boutiques (Nelson and Kent 1999). At the 2000 Account Planning Group--U.S. (APG-U.S.) conference held in Miami Beach, organizers commented that attendance at the conference had grown from less than 50 to more than 950 over the course of 6 years. In addition, organizers stated that planning has grown beyond just bringing the consumer's voice to the table; planning is now used to understand all aspects of a client's business, such as corporate culture, management's values, the meaning of products to the workers who make them, and so forth.

In this sense, planning in the United States has evolved beyond the original British concept of planning. According to Steel (1999, p. 36), the British planners of the 1960s and early 1970s "made it their mission to challenge the methodology of the research industry that they believed was counter-productive to the development of great advertising." According to Barry, Peterson, and Todd (1987, p. 16), account planning arose because of a perception that U.K. advertising agencies were not "adequately tuned in to British consumers." Further, the people in the agency who had the most contact with consumers--the agency researchers--were not being used in a problem-solving capacity or integrated into the process of advertising development. Account planning "solved" this problem by making the researcher (i.e., the planner) a full partner with input into advertising decisions.

The planning function in U.S. agencies has progressed beyond advertising, which was its original focus in the United Kingdom. Steel (1999) notes that planners are most useful when they go beyond advertising solutions and into broader business solutions. He states that planners "should be telling a client how demographic, cultural, economic, competitive and attitudinal change is affecting their overall business, and how changes in fundamental business practices are required if they are to succeed" (1999, p. 36). Others feel that insights derived from planning should not only inform message-strategy development, but also media planning, promotions, packaging, and other types of marketing decisions (Boyko 1999). Structurally, planning has achieved status as an independent department in major U.S. and international advertising agencies.

Along with the transition in function have come heightened expectations of planning from top agency management and clients. Many U.S. agencies are under tremendous pressure from shareholders, market analysts, and mega holding companies to show both profit and growth. Many people on the client side are under similar pressure to show bottom-line accountability. In light of these pressures, agencies and clients may see planning as a "cure-all" that can make up for poor creative work, the inability to sell work to clients, unstable client relationships, or poor new business performance (Steel 1999). According to Steel, these are problems that planners have no business trying to fix. "Against admittedly unrealistic expectations," he claims, "the discipline (planning) is overpromising and underdelivering" (p. 36).

The value of account planning has been addressed in several books: Fortini-Camphell's Hitting the Sweet Spot (2001; 1992), Bond and Kirshenbaum's Under the Radar: Talking to Today's Cynical Consumer (1998), and, most notably, Jon Steel's work, Truth, Lies and Advertising: The Art of Account Planning (1998). The current study began with the premise that planning is a valuable part of today's advertising agency landscape; therefore, it is not the purpose of this paper to argue for the value of planning. Rather, given the evolution of the planning discipline in the United States and the ever-increasing pressures for accountability, this study seeks to understand how agencies evaluate the work of account planners and whether evaluation techniques are appropriate in the eyes of planners.

This article reports the results of a national survey of account planners and is part of a larger study aimed at assessing the attitudes and opinions of planners in the United States. Specifically, the study reported here focuses on two questions: (1) How is the work of planners evaluated in today's advertising agencies? and (2) How do planners think their work should be evaluated? In addition to addressing the accountability aspect of planning, this article is important in that it can contribute to agencies' thinking about evaluating the effectiveness of planning. Furthermore, because it marks the first national survey of account planners, the study will provide a benchmark from which to track changes in the industry's attitudes about evaluating the work of U.S. planners.

LITERATURE REVIEW

There is little in the industry literature regarding how to evaluate the work of account planners. For example, the major books that either discuss or focus on planning, such as Steel (1998), Bond and Kirshenbaum (1998), and Fortini-Campbell (2001; 1992), stress the benefits of planning but do not address evaluation of planners. The Account Planning Group--U.K. (APGU.K.), in a document titled What Is Account Planning? (1987), suggested that traditional measures, such as message recall and tracking the results of advertising in terms of sales, awareness, and image can be used to measure the outcomes of the planning process, that is, the advertising campaign. Using these measures may be justified in light of Newman's (1998) view that "unlike traditional research, which is a staff function, account planning ... has a line responsibility to insure that the advertising is relevant and motivating to the consumer and ultimately is accountable for its effectiveness." However, these measures do not account for what pl anners have said are the unique benefits of engaging in planning (e.g., better team work; more integration within the agency; added dimension to understanding the process of developing ads by stimulating discussion about purchasing decisions, the brand relationship, and how advertising works in specific circumstances; helping to win new business by instilling confidence in the prospective client as a result of a comprehensive and disciplined approach; stimulating more productive contact between the creative department and the consumer; and so forth [APG-U.K. 19871). Some of these benefits are rather intangible and difficult to measure, which may be the reason why some recommend relying on traditional campaign measures.

A review of the academic literature reveals only four studies of account planning. Three of the four focused on the planning process (Barry, Peterson, and Todd 1987; Maxwell 1996; Schofield 1990) and are not relevant to the current study. For example, Schofield examined the different rates of adoption of account planning among advertising agencies and direct-marketing agencies in the United Kingdom, and Maxwell (1996) investigated how account planners differed from traditional researchers. Only one existing study--Maxwell, Wanta, Sheehan, and Bentley 2000--considers any aspect of planning evaluation.

Using a national mail survey of U.S. advertising executives (mainly account managers), Maxwell, Wanta, and Bentley (2000) attempted to link planning effectiveness to creative awards won by advertising agencies, growth in gross billings, and number of client accounts by comparing these fore mentioned statistics between agencies that did and did not employ account planning. The study found that planning agencies won more creative awards than non-planning agencies and that planning agencies had a greater increase in billings and number of accounts than non-planning agencies. The authors suggested that these findings point to the overall effectiveness of the account planning position.

The Maxwell study serves as a good starting point for exploring further how planners' work should be evaluated. Given that many variables within a year's work at an agency can lead to changes in billings, number of accounts, and receiving creative awards, it can be difficult to isolate planning as the reason for notable increases or decreases in these measures. It would be interesting to see how account planners today evaluate the criteria suggested by Maxwell in the context of other types of job performance criteria. Such job performance criteria could be traditional advertising effectiveness criteria such as message recall, effects on sales, awareness, and branding, which the APG-U.K. (1987) and planners like Newman (1998) suggest should be factors in evaluating planning. Other criteria may be important as well. To advance understanding of how planners' work should be evaluated, it is important to understand what criteria those working in account planning feel are valid indices of their work. This aspect of understanding account planning is significant because once the criteria for evaluating the work of planners are identified, future research can use them to focus on finding better ways to evaluate the performance of planners.

METHOD

Survey Construction

Based on a review of the limited existing literature, a survey instrument consisting of 13 measures of evaluation was drafted. These included some traditional measures of campaign effectiveness (for example, "achievement of advertising objectives," "target-marker awareness of advertising message," "purchase intention of target market, increase sales or market share," etc.) as suggested by the APG-U.K. (1987) and Newman (1998). Several nontraditional measures (for example, "feedback from the creative team," "creative awards," etc.) were also included, based on Maxwell, Wanta, Sheehan, and Bentley (2000), other industry writings about account planning, and preliminary interviews with account planners. To assess what measures are currently being used to evaluate the work of account planners, each item was evaluated using a five-point Likert scale with endpoints of never used/always used. To facilitate comparison, planners were asked to evaluate the same items again using a five-point Likert scale with endpoints of should not be used/should be used.

The survey began with a working definition of what an account planner is ("Persons holding the job of account planner may also be called 'strategic planners,' 'strategic researchers,' or 'creative researchers,' or have another title. Regardless of title, these people are characterized by their involvement in research to develop brand/creative strategy and creative tactics, or research to evaluate the creative aspects of a communications campaign"). The first question of the survey queried whether potential respondents felt qualified to participate based on the definition. Respondents were next asked whether there was an account to which they were currently assigned that demanded substantially more of their time than their other accounts (i.e., a "primary" account). If respondents indicated that they worked on a primary account, they were instructed to answer the rest of the questions with this account in mind. If no primary account was indicated, respondents were asked to answer the rest of the questions with their overall account assignments in mind. Included among the remaining questions were standard demographic measures such as gender, age, years in planning, title/position, and agency size (in terms of revenue).

Pretest

To strengthen the instrument and make sure it was grounded in the experience of planners, the initial draft was reviewed by the board of APG-U.S., the largest national organization exclusively devoted to account planning. The instrument was then pretested with six additional individuals who worked in account planning. Pretest participants were recruited word-of-mouth and their cooperation was obtained in advance of a telephone interview. The six individuals varied in title from account planner to executive vice president of strategic planning, and their account planning experience ranged from 6 months to over 10 years. This helped to assure that perspectives of both veteran planners and those new to planning were represented. Copies of the survey were sent to the participants and they were asked to review the survey in advance of a telephone interview to assess the validity of the instrument and see whether any measures of evaluation should be added. Phone interviews ranged from 20 to 45 minutes in length. As a result of the interviews, minor changes were made to the instrument. For example, one of the initial evaluative criteria was "creative awards." Participants suggested that this item should be adjusted to indicate the type of awards and also recommended that "planning awards" be added to the list. The final survey included 15 items of evaluation (see Table 2) in each of two different five-point Likert scales (never used/ always used and should not be used/should be used). To be sure that we gave planners the full chance to express their thoughts, an open-ended question asking respondents to identify and describe any means used to evaluate their work that were not listed in the scales was included.

Sample Frame: Generation and Refinement

A mailing list of 2,286 APG-U.S. members and their affiliated companies was obtained. Unfortunately, this list was somewhat dated and included many entries not of interest to the researchers. For example, many on the list were from agencies outside the United States, or were not from recognized agencies (defined as being listed in the Standard Directory of Advertising Agencies [January 20001). The majority of entries also did not include a title, and since being a planner is not a qualification for belonging to the APG-U.S., it was suspected that many on the list did not work in account planning. To create a list that was more "planner specific," the researchers first sorted the original list and took out those members who were outside the United States. Members who remained were then cross-referenced with the Standard Directory of Advertising Agencies (January 2000) to verify that they were, in fact, employed by a recognized agency. Because of the large amount of turnover associated with the industry, the re searchers then called every agency that remained on the list to verify that (1) the agency did have an account planning department or at least one individual who acted in an account planning capacity, and (2) those on the list did work at the agency in account planning. Given that there is tremendous turnover in the industry, names of agency employees who were not on our initial list but who currently worked at the agency in the planning function were added. Similarly, members who were no longer employed at the listed agencies were removed from the sample frame. All agencies were contacted at least two times.

To call attention to the survey and enhance response rate, a letter explaining the nature of the study and encouraging members to participate was placed in the registration packet that attendees received at the 2000 APG-U.S. conference (held two weeks before the survey was mailed). This letter was written by the researchers and the APG-U.S.'s administrator, and was signed by the administrator on APG-U.S. letterhead. Nine-hundred and fifty-eight persons attended the conference. In addition to names verified through phone calls, the researchers also included names of the conference attendees if the individuals were identified in the conference program as being affiliated with a recognized agency. After the sample was verified and the names of conference attendees were added, the sample contained 1,161 names from 245 different agencies.

Response Rate

Three hundred forty-five surveys were returned, yielding an overall response rate of 29.7%. Two hundred sixty-seven (23%) questionnaires were returned from the first mailing and 76 (6.5%) from the second. In two cases (.2%) the code used on a survey to identify the mailing had been tampered with. To estimate nonresponse bias, first-wave and second-wave data were compared on all the relevant measures of the study; no significant differences were apparent between the two groups.

The 345 completed surveys emanated from 147 different agencies. Of the surveys returned, 21 respondents indicated that they were not qualified to participate based on the working definition of an account planner provided in the introduction of the survey. These names were removed from the sample, leaving an overall response rate of 28.4% (324/1,140). This response rate is similar to other published surveys of advertising professionals (Reid, King, and DeLorme 1998). For example, Kendrick, Slayden, and Broyles (1996) reported a 30% response rare from their survey of agency creatives. More recently, Spake, D'Souza, Crurchfield, and Morgan (1999) reported an 18.9% response rate for a national survey of U.S.-client advertising managers.

Sample Characteristics

Participants ranged in age from 21 to 62, with an average age of 34.7 (see Table 1). In terms of gender, the sample was skewed toward women (5 6.2%). On average, respondents had worked in planning for six years. Though by nature the job of an account planner is a management-oriented position, it is useful to think of these management positions as either "senior" or "junior" level. Most of the sample (52.5%) had senior management titles (for example, vice president, manager, or director), whereas the remaining 47.5% listed more junior-level management titles (for example, assistant planner, senior strategic planner). A little over 94% of the respondents were from full-service agencies, as is noted in Table 1.

FINDINGS

Factor Analysis

To reduce the data set to related evaluative criteria, a factor analysis with varimax rotation was performed on the ratings of the 15 scale items for each question ("How is work of planners currently being evaluated?" and "How should the work of planners be evaluated?"). Results for each question are presented separately below, followed by a comparison of the factor solutions.

"Currently Used" Criteria Factor Solution

The factor analysis of currently used items yielded a three-factor solution based on the criteria eigenvalue greater than one. The three factors were (1) standard campaign measures, (2) awards and press, and (3) feedback (see Table 2). These three factors accounted for 67.9% of the variance in ratings.

To determine that the three factors were indeed distinct aspects of the evaluation process, a within-subjects MANOVA was performed. A pairwise comparison of factors confirmed that the three factors were significantly different from one another, Wilks's [lambda], F(2, 313) = 440.746, p < .001.

"Should Be Used" Criteria Factor Solution

The factor analysis of should be used items also yielded a three-factor solution based on the criteria eigenvalue greater than one. Again, the three factors were (1) standard campaign measures, (2) awards and press, and (3) feedback (see Table 2). These three factors accounted for 65.1% of the variance in the measures. A within-subjects MANOVA was run on these factors as well. The pairwise comparison of factors confirmed that the three factors were significantly different from one another, Wilks's [lambda], F(2, 307) = 274.328, p < .001.

Cronbach's [alpha] reliability testing on both of the above factor structures produced moderate to high scores ranging from .75 to .94. Based on the exploratory nature of the study, these reliability coefficients were deemed acceptable (Peterson 1994).

Comparison of Factor Solutions Across Conditions

Across the two scales, identical items loaded on the same factors. To further investigate whether the factor solution for currently used was similar to or different from the factor solution for should be used, a repeated-measures MANOVA was performed on the factor solutions across the two conditions. The results indicated that true differences existed in some of the measures, Wilks's [lambda], F(2, 302) = 3.536, p < .03.

Subsequently, paired t tests were used to test for significant differences between the factors for each scale (see Table 3). A comparison of the means between the two scales revealed significant differences for two of the three factors: standard campaign measures(t = -4.443, df = 309, p < .001) and awards and press (t = -2.621, df = 309, p = .009). Both of these factors were rated significantly higher as measures that should be used to evaluate the work of planners versus how they are currently used.

Insights from Open-Ended Questions

Seventy-two respondents (22%) answered the open-ended question, "Are there any means used to evaluate your work that are not included in the list above?" Many respondents' answers directly supported the survey items, either by providing clarity or emphasizing the importance of a listed item. One response, for example--"ultimately, sales/market share is the single most important evaluation"--highlighted the importance of the survey item "increased sales or market share." The open-ended comments were counted and grouped relative to the three-factor solution found in this study (Table 4).

Themes related to the standard campaign measures factor dealt with ways to quantify the outcomes of planning and advertising effectiveness. Nearly a third of all responses were related to this factor; of these, over half the items encompassed four themes (sales, meeting objectives, copy testing, and other traditional marketplace measures). The remaining three themes related to the factor were not accounted for in the scale items. These were Web site measures, customer acquisition and retention, new business for the agency, and effect on share price. Although these responses identify measures that can be quantitatively used to measure campaign outcomes, they differ from more traditional campaign measures. Nevertheless, their quantitative nature suggests that they are related to the standard campaign measures factor. Questionnaires from respondents who suggested these items were pulled and examined more closely. The nine respondents who suggested effects on new business and effects on share price were from gene ral full-service agencies working on consumer accounts. However, those eleven respondents who suggested Web site measures and customer acquisition and retention mainly worked on either business-to-business accounts or in an interactive agency. Web site hits were also important to two entertainment-oriented accounts.

Another quarter of the responses related to scale items that grouped under the feedback factor. Of these, nearly all were directly related to scale items. However, a few additional responses suggested that evaluation from peers and external feedback (such as from the target market or in the press) are also dimensions of feedback.

Responses related to the awards and press factor were few in number. This factor encompassed both awards and media coverage of advertising. Three of the four comments directly related to this factor suggested that EFFIE awards are currently used to evaluate the work of account planners. The other comment called for media coverage of prominent planning or opinion leaders.

The remaining categories of comments were not directly accounted for in the existing scale. Three themes--innovation in the way the job is performed creatively, innovation in the way the job is performed methodologically, and job performance skills--reflected performance enhancement skills. These included finding new methods to do a job better, finding more creative ways to express ideas, and basic job performance skills (such as improved presentation skills or meeting deadlines). Two comments related to the cultural impact of the work and the progression of planning within the agency. The five comments related to brand equity considered issues such as brand influences and the overall contribution of planning to the growth and maintenance of a brand. Seven responses related to the acquisition of new business for the agency. Relative to this theme, it is important to note that although new business wins is a potentially important measure of planning evaluation and has been documented (Maxwell, Wanta, Sheehan, and Bentley 2000), its presence was not appropriate for the current study because planners were asked to answer questions with their primary or current account assignment in mind. The remaining seven responses were grouped into a category called "other" because they did nor obviously fit with the major ideas present in the other comments or have any identifiable relationship with each other.

DISCUSSION AND IMPLICATIONS

The three-factor solutions derived from the analysis of account planners' responses in this study suggests that there are three general types of evaluation important to understanding the job of an account planner: standard campaign measures, feedback and awards, and press. Conceptually, these can be grouped into two general stages of evaluation: (1) evaluation of the process (i.e., feedback during the process), and (2) evaluation of outcomes (i.e., traditional campaign measures and attention from media, ad industry, and other planners via awards or press coverage).

The importance assigned by planners to each of these types of evaluation reflects the philosophy of planning in the United States. According to the literature, it is generally agreed that planners are the point people in the process of developing message strategy, and serve as the agency's primary contact with the outside world. Planners are also charged with bringing a strong consumer focus to all advertising decisions (White 1995). Similarly, the APG states that the unique benefits of engaging in planning include better team work, more integration within the agency, added dimension to understanding the process of developing advertising, and stimulating more productive contact between the creative department and the consumer (APG-U.K. 1987).

Therefore, it should not be surprising that planners believe feedback during the process of planning and message-strategy development is the most important factor in evaluating their work (averaged factor mean 4.16). Three items comprised the feedback factor: feedback from the client, creative team, and account team. Feedback from the creative team is the single item that planners believe should be used most in evaluating a planner's job performance. This reflects the core focus of planning on generating quality creative insights.

But just behind feedback from the creative team comes feedback from the client. Communication between the agency and the client has been documented as an extremely important aspect of a good client-agency relationship (Fam and Waller 1999; Hotz, Ryans, and Shanklin 1982; Murphy and Maynard 1996; Wackman, Salmon, and Salmon 1986). This finding supports the importance of agency--client communication and reflects the evolution of planning in the United States. Steel (1999, p. 36) notes that planners, in addition to providing message-strategy insights, "should be telling a client how demographic, cultural, economic, competitive and attitudinal change is affecting their overall business, and how changes in fundamental business practices are required if they are to succeed."

Although feedback during the process is most important to planners, planners in the United States are also aware that standard advertising campaign measures are being used to evaluate their work. In fact, the majority of planners in this study seem to feel that standard campaign measures should be used more than they are now being used. Analysis of the open-ended responses also offers support for the use of "quantifiable" measures as evaluative criteria. Twenty-seven percent of the responses dealt with quantified measures; of these, half were directly related to scale items and the remainder were indirectly related to the scale items. The use of standard advertising campaign measures reflects Newman's (1998) view that "unlike traditional research, which is a staff function, account planning has a line responsibility to insure that the advertising is relevant and motivating to the consumer and ultimately is accountable for its effectiveness." The focus on advertising performance as a way to evaluate a planner' s job performance may also reflect the growing pressure on top agency management and clients to show both profitability and continued growth to shareholders, market analysts, and mega holding companies. The importance placed on standard campaign measures points to the evolution of planning in the United States from the original British conceptualization. That British idea of the mission of planning in the 1960s and 1970s was described thus by Steel (1999, p. 36): "to challenge the methodology of the research industry that they believed was counter-productive to the development of great advertising." Now planners not only challenge research methodology; they are themselves challenged to show their positive impact on their client's bottom line.

The outcome factor, awards and press, is the least important factor for planners in evaluating their work. In fact, the mean value assigned to this factor by planners suggests that awards and press are not important or useful indicators of a planner's job performance. Planning awards such as those given by the APG figure less into how planners' work is currently evaluated than do more traditional awards such as the Clio. However, planners seem to feel that planning awards should be given more weight in evaluating account planning and may be a better gauge of the work they do than the more established creative awards like the One Show or the Clio. This may be due in part to the nature of the APG planning awards. The APG awards are based on case studies of campaigns and include creative briefs and descriptions of research in addition to multiple advertising executions. In this sense, they are similar to the EFFIEs, which are awarded to advertising campaigns based on effectiveness and not creativity (Kover, Gold berg, and James 1995; Moriarty 1996).

The ultimate implication of this study is that U.S. planners feel that their work is currently being evaluated as it should be, with minor exceptions. Overall, the relative importance of each of the three evaluation factors remains the same across the two conditions of currently used and should be used. Feedback is most important, followed closely by campaign outcomes. Also, there appears to be agreement that all three factors should be used a bit more than they are currently used. Planners and agencies may wish to take a more detailed look into the evaluation of the items within each factor to see how current evaluation methods can be altered to reflect better the way planners feel their work should be evaluated.

This study provides a foundation of important criteria for evaluating the work of account planners, a foundation that is grounded in planners' perceptions. Future research may expand on this knowledge in the following ways.

Agency Context and Evaluation of Job Performance

The majority of respondents in the present study were from full-service general consumer advertising agencies. Future research may build on the present study by replicating the study with planners who work in different types of agencies. For example, open-ended responses in this study indicate that those in interactive agencies or in business-to-business planning seem to be using direct-marketing criteria to evaluate planners' job performance. If future studies find that interactive agencies and business-to-business planners use a direct-marketing communications paradigm, whereas general consumer agencies are using a traditional advertising paradigm, then it follows that account planners' job performance criteria may differ across the two paradigms. Evidence of interactive agencies employing a direct-marketing oriented paradigm is offered by Kim and Leckenby (2001) in their study of how media directors view traditional versus interactive media planning.

Also, future research may build on this study by examining how various factors such as agency size, revenue, structure, and culture correlate with criteria used to evaluate planners' work. Such questions could include, Do the criteria used to evaluate planners' work differ from large to small agencies? From planning-driven agencies to traditional agencies? From agencies with a departmental structure to agencies that use integrated account teams?

How to Measure Job Effectiveness

The present study did not attempt to create a metric for measuring planning effectiveness. Rather, the study uncovered the basic criteria that planners believe are important in assessing their job performance. Future research may explore how best to measure the criteria identified in this study. Criteria such as market share, meeting objectives, and so forth, are fairly easily measured. However, the criteria related to feedback--the most important factor to planners--are more problematic when it comes to measurement. Qualitative research with planners may be best suited to explore how feedback should be given to planners from management, the account team, creatives, and the client. Should feedback come informally or formally? Face-to-face or via an instrument? Philosophically, the question of whether feedback effectiveness or the quality of a creative insight can or should be quantitatively measured should also be debated.

The findings in the present study bring up questions about the perceived validity of awards as measures of planning effectiveness and suggest that they do not factor heavily into the evaluation process. Although it is possible that the planners who responded to the study were not "award winners," and thus did nor place much value on winning awards, it is also possible that planners simply do not perceive winning awards as a valid measure of their work. Evidence exists that casts doubt on the credibility of awards, particularly awards for "creativity." A study by Kover et al. (1997) found that award-winning creatives write for their peers rather than consumers, suggesting that creative advertisements are not necessarily relevant to consumers. Creative awards have also been cast in a negative light. Recent press coverage of the Cannes creative competition raised concerns about the prevalence of "ghost" ads (vanity ads created specifically for a competition and never used in a client campaign) (Wentz 2001). Give n the planner's role of representing the consumer, and the evidence of what creative awards actually measure, it is possible that planners do not see these awards as important as awards based on effectiveness.

If awards are nor a good way to measure planning, the same might be true for other jobs in advertising, which raises the issue of the purpose chat awards serve. For example, award-winning creative directors and copywriters are often sought after on the basis of their award history. Award-winning advertising is not necessarily effective advertising, however. Future research to investigate how creatives feel about the evaluative potential of awards is needed. Do creatives see awards as a valid measure of their worth as advertising professionals? If not, what do creative awards really indicate?

Evolution of the Field

Finally, given that advertising is an ever-changing discipline, periodic studies of how to evaluate planners' work and that of other advertising professionals should be done to track the evolution of what the industry values. The literature on planning has already documented an evolution in the discipline from the original British conceptualization to the current U.S. incarnation. Although there is not a study similar to the present one to compare differences in planners' job performance criteria from the original British system to the current U.S. system, the present study is one benchmark against which future evolution can be measured. By understanding how job performance criteria and procedures evolve, future studies can provide the industry and researchers with ways to trace what advertising professionals emphasize and value as measures of a job well done.

TABLE 1

Characteristics of Sample

                     N *  Average

Age                  317   34.7
Gender               322
Female                     56.2
  Males                    43.8
Years in planning    323   5.96
Title                322
  Senior management        52.5
  Junior management        47.5

                          N *  Percent

Agency type               321    100.0
  Full service            303     94.4
  Boutique (specialized)    9      2.8
  In-house                  2       .6
  Other                     7      2.2

* Total sample size = 324; missing cases not included in these
calculations.

TABLE 2

Summary of Factor Scores and Eigenvalues (Varimax Rotation)

                                       Factor                Variance
                                       score   Eigenvalue  explained (%)

Currently Used to Evaluate                                     67.89

Factor 1: Standard Campaign                       5.69         37.93
Measures (a = .94)
  Target-market recall of               .933
   advertising message
  Target-market recall of               .924
   advertisements
  Target-market awareness of            .913
   advertising
  Target-market awareness of brand      .893
  Attitude change by target             .825
   market toward product
  Purchase intention of target market   .811
  Achievement of advertising            .640
   objectives
  Increased sales or market share       .618

Factor 2: Awards and Press (a = .76)              2.46         16.41
  Other awards                          .816
  Planning awards                       .811
  Creative awards                       .810
  Media coverage of the advertising     .515

Factor 3: Feedback (a = .75)                     2.033         13.55
  Feedback from the account team        .883
  Feedback from the creative team       .854
  Feedback from the clinet              .689

Should Be Used to Evaluate                                     65.13
Factor 1: Standard Campaign                       5.29         35.24
Measures (a = .91)

  Target-market recall of               .898
   advertisements
  Target-market awareness of            .891
   advertising
  Target-market recall of advertising   .878
   message
  Target-market awareness of brand      .858
  Purchase intention of target market   .785
  Attitude change by target market      .755
   toward product
  Increased sales or market share       .528
  Achievement of advertising            .505
   objectives

Factor 2: Awards and Press (a = .80)              2.51         16.73
  Creative awards                       .898
  Other awards                          .862
  Planning awards                       .842
  Media coverage of the advertising     .454

Factor 3: Feedback (a = .82)                      1.97         13.16
  Feedback from the account team        .918
  Feedback from the creative team       .822
  Feedback from the client              .814

TABLE 3

Results of Paired t Tests on Factor Structures

                                                  Signif.
                             Mean  df     t     (two-tailed)

Factor I: Standard Campaign        309  -4.443     .001 *
Measures
 Currently used              3.58
 Should be used              3.80

Factor 2: Awards and Press         306  -2.621    .009 **
 Currently used              2.19
 Should be used              2.36

Factor 3: Feedback                 308  -.603       .547
 Currently used              4.16
 Should be used              4.19

* Significance < .001.

** Significance at the .05 level.

TABLE 4

Themes from Open-Ended Questions Related to Factors


Factor I: Standard Campaign Measures (29 total)
Directly related to items in scale
  Sales-related (8)
  Meeting objectives (4)
  Copy testing (2)
  Marketplace measures (2)

Unaccounted for but related to current scale
  Web site measures (7)
  Customer acquisition and retention (4)
  Effect on share price (2)

Factor 2: Awards and Press (4 total)
Directly related to items in scale
  EFFIE awards (3)
  Attention of press to planning/opinion leaders' status (not ads) (1)

Factor 3: Feedback (26 total)
Directly related to items in scale
  Internal feedback from management (9)
  External feedback from client (8)
  Internal feedback from team (5)

Unaccounted for but related to current scale
  Internal feedback from peers (2)
  External feedback other (2)

Themes not related to items in the scale
  New business for agency (7 mentions)
  Innovation in the way job is performed creatively (7)
  Brand equity (5 total)
  Job performance skills (3 total)
  Innovation in the way job is performed methodologically (3)
  Internal and external cultural issues (2 total)

Other (7 total)

REFERENCES

APG-U.K. (1987), "What Is Account Planning?" Account Planning Group-U.K. Web site, www.apg.org.uk/whatis.html (accessed June 25, 1998).

Barry, Thomas E., Ron L. Peterson, and W. Bradford Todd (1987), "The Role of Account Planning in the Future of Advertising Agency Research," Journal of Advertising Research, 27 (1), 15-21.

Bond, Jonathan Bond, and Richard Kirshenbaum (1998), Under the Radar: Talking to Today's Cynical Consumer, New York: Wiley.

Boyko, Rick (1999), "The Evolution of an Idea," presentation made at the 1999 Account Planning Conference, San Diego, July.

Capon, Noel, and Debra Scammon (1979), "Advertising Agency Decisions: An Analytic Treatment," Current Issues and Research in Advertising, 25-52.

Fam, Kim Shyan, and David S. Waller (1999), "Factors in Winning Accounts: The "Views of Agency Account Directors in New Zealand," Journal of Advertising Research, 39 (3), 21-32.

Fortini-Campbell, Lisa ([1992], 2001), Hitting the Sweet Spot: How Consumer Insights Can Inspire Better Marketing and Advertising, Chicago: The Copy Workshop.

Hotz, Mary R., John K. Ryans, Jr., and William L. Shanklin (1982), "Agency/Client Relationships as Seen by Influentials on Both Sides," Journal of Advertising, 11 (1), 37-44.

Kendrick, Alice, David Slayden, and Sheri J. Broyles (1996), "Real Worlds and Ivory Towers: A Survey of Top Creative Directors," Journalism and Mass Communication Educator, 51 (Summer), 63-74.

Kim, Hyo-Gyoo, and John D. Leckenby (2001), "How Media Directors View Traditional and Interactive Media Planning," paper presented at the 2001 conference of the American Advertising Association, Salt Lake City, April 1.

Kover, Arthur J., and Stephen M. Goldberg (1995), "The Games Copywriters Play: Conflict, Quasi-control, a New Proposal," Journal of Advertising Research, 35 (4), 53-61.

-----, Stephen M. Goldberg, and William L. James (1995), "Creativity Vs. Effectiveness? An Integrating Classification for Advertising," Journal of Advertising Research, 35 (6), 29-38.

-----, William L. James, and Brenda S. Sonner (1997), "To Whom Do Advertising Creatives Write? An Inferential Answer," Journal of Advertising Research, 37 (1), 41-53.

Maxwell, Ann (1996), "What Do Account Planners Do?" presented on panel "Account Planning: Add the Consumer's Voice and Stir," in Proceedings of the 1996 Conference of the American Academy of Advertising, Gary B. Wilcox, ed., University of Texas at Austin, 252-255.

-----, Wayne Wanra, Kim Sheehan, and Clyde Bentley (2000), "The Effectiveness of Account Planners at U.S. Advertising Agencies," in Proceedings of the 2000 Conference of the American Academy of Advertising, Mary Alice Shaver, ed., Michigan State University, 149-154.

Moriarty, Sandra E. (1996), "Effectiveness Objectives and the EFFIE Awards," Journal of Advertising Research, 36 (4), 54-63.

Murphy, Priscilla, and Michael L. Maynard (1996), "Using Judgment Profiles to Compare Advertising Agencies' and Clients' Campaign Values," Journal of Advertising Research, 36 (2), 19-27.

Nelson, Ted, and Megan Kent (1999), "The Power of Conversations," Adweek Eastern Edition, 40 (28), 3a.

Newman, Jane (1998), "What Is the Client Relationship to Account Planning?" Essays on Account Planning, Account Planning Group-U.S. Web site, www.apgus.org (accessed February 10, 2000).

Peterson, Robert A. (1994), "A Meta-analysis of Cronbach's Coefficient Alpha," Journal of Consumer Research, 21 (September), 381-391.

Reid, Leonard N., Karen Whitehill King, and Denise DeLorme (1998), "Top-Level Agency Creatives Look at Advertising Creativity Then and Now," Journal of Advertising, 27 (2), 1-15.

Schofield, Albert (1990), "The Role of Account Planning in the Direct-Marketing Agency," Journal of Advertising Research, 30 (5), 61-66.

Spake, Deborah F., Giles D'Souza, Tammy Neal Crutchfield, and Robert M. Morgan (1999), "Advertising Agency Compensation: An Agency Theory Explanation," Journal of Advertising, 28 (3), 53-72.

Standard Directory of Advertising Agencies (January 2000), no. 242: New Providence, NJ: National Register Publishing.

Steel, Jon (1998), Truth, Lies and Advertising: The Art of Account Planning, New York: Wiley.

----- (1999), "Tests Ahead for Account Planning," Advertising Age (September 20), 36.

Taggart, Lynne (coordinator, APG-U.S.) (1999), telephone interview.

Wachman, Dan, Carin Salmon, and Charles Salmon (1986), "Developing an Advertising Agency-Client Relationship," Journal of Advertising Research, 226 (6), 21-28.

Wentz, Laurel (2001), "Ghosts of Cannes," Advertising Age (June 11), 1.

White, Rob (1995), "Planning and Its Relationship with Account Management," in What Every Account Executive Should Know About Account Planning, New York: American Association of Advertising Agencies.

This study was funded, in part, by a grant from the American Academy of Advertising. The authors thank the editor, three anonymous reviewers for their helpful comments, and Lynne Taggart of the Account Planning Group-U.S. (APG-U.S.).

Margaret A. Morrison (Ph.D., University of Georgia) is an associate professor in the Department of Advertising at the University of Tennessee.

Eric Haley (Ph.D., University of Georgia) is an associate professor in the Department of Advertising at the University of Tennessee.