A SAS MACRO for calculating intercoder agreement in content analysis.

Introduction

There has been an increased interest in the application of content analysis in the marketing and advertising literatures. Content analysis allows researchers to uncover and categorize underlying themes, patterns, values and perspectives which are contained in the objects of the analysis (e.g., television commercials or magazine ads). Content analysis is versatile in that it lends itself to both cross-sectional and longitudinal analysis. For example, cross-sectional studies have included investigations of the use of animation in television advertising (Bush, Hair and Bush 1983), the portrayal of hispanics and blacks in television commercials (Wilkes and Valencia 1989) and the analysis of information content in cross-national advertising (Meuller 1991). Longitudinal studies have included social responsibility themes in magazine advertising (Lill, Gross and Peterson 1986), the use of the elderly in magazine advertising (Ursic, Ursic and Ursic 1986) and the nature of time orientation (Gross and Sheth 1989).

A major methodological issue in content analysis is the consistency of the interpretation and categorization of the objects of interest, e.g., advertisements. The concern here is that the categorization is "reliable" in that coders place a given object consistently into the same category. The techniques used to assess the consistency of such coding are subsumed under the title of intercoder agreement.

Two major articles have recently addressed this issue (Perreault and Leigh 1989; Hughes and Garrett 1990). Hughes and Garrett (1990) review the approaches to assessing reliability in content analysis that published marketing research used during the period 1984-87. Of the 86 articles examined, only eleven (13%) used acceptable measures of intercoder agreement. The vast majority (65%) used percentage agreement, a measure criticized for many years by leading researchers in the area (Cohen 1960; Krippendorff 1980). The weaknesses of this very popular approach are cogently presented by Krippendorff (1980, p. 133-134). The major criticism is that percentage agreement does not correct for chance agreement. In their conclusions, Hughes and Garrett (1990) recommend a totally new approach to such calculations, based on generalizability theory.

In the second article Perreault and Leigh (1989) review the literature on the subject of intercoder agreement as it pertains to categorization schemes of nominal data only. They propose an index of agreement which we refer to as Perreault's Index (PI). This index is similar to those proposed by Bennett et al. (1954), Scott (1955), Cohen (1960), and Krippendorff (1980). In fact, it is the square root of Bennett's (1954) index.