A comparison of conjoint analysis response formats: reply. (Comments and Replies).

A recent article by Boyle, Holmes, Teisl, and Roe (BHTR) evaluated the convergent validity of three response formats that are commonly used to implement attribute-based, stated-preference experiments (or "conjoint analysis"). The primary conclusion of BHTR was that "... convergent validity of

ratings, ranks and choose one is not established" (p. 452). This finding suggests that economists need to develop a better understanding of the decision rules that people use when answering these types of stated-preference response formats if statistical results are to effectively inform policy. McFadden (2001) argues that "the existence of underlying preferences is a vital scientific question for economists," and "the evidence on decision making from cognitive psychology implies only that economists must look through the smoke screen of rules to discern the deeper preferences that are needed to value economic policies" (p. 363). It is to this larger research agenda that the BHTR study aspired to contribute.

Specifically, BHTR investigated convergent validity through a split-sample design where independent subsamples of survey participants were randomly assigned to answer either a rating, rank, or choice question. Responses to the rating and ranking questions were re-coded to choices. Models were then estimated with the re-coded rating and rank data so that preference parameters and WTP estimates could be directly compared with estimates based solely on the choice data. That is, for the rating, rank, and choice questions survey respondents were given four alternatives to consider. For each type of response data four observations were created per respondent. The alternative with the highest rating was re-coded as the choice (=1) and the other three alternatives were coded as not chosen (=0). Similarly, the alternative with the highest rank was re-coded as the choice (=1) and the other three alternatives were coded as not chosen (=0). For the choice data, the chosen alternative was coded as "1" and the other three alternatives were coded as not chosen (=0). These data were then analyzed using simple probit models.

The Comment

Lusk states "... that the estimates from the choose-one format (recoded ratings and ranks, and actual choices) may be inaccurate because of a modeling effort that did not account for all choices in the respondents' choice set." (p. 1170). Lusk argues that the data should have been analyzed with a multinomial logit model where respondents chose one of four alternatives. He proceeds to argue his point by simulating "true" utility functions to generate data that he then analyzes using multinomial logit (MNL), logit, and probit models.

The Reply

We agree with Lusk that an MNL model is generally the appropriate approach for analyzing the BHTR choice data, but there are conditions, which Lusk overlooks, where a binary logit (or probit) model is appropriate. We contend that the choice-modeling results reported in BHTR are accurate and that our conclusion of failure to find convergent validity of ratings, ranks, and choices is valid. Our arguments in support of this assertion are explained below.

First, Lusk's simulated data assume that people act like rational economic agents, which assumes away the foibles of actual data that lead to differences in how people answer preference questions when responses are elicited as ratings, ranks, or choices. Thus, Lusk does not address how his critique would affect the primary finding of BHTR--inability to establish convergent validity of rating, rank, and choice data.

Second, the BHTR analysis did account for all choice alternatives with their use of a binomial probit model. BHTR's use of a binomial probit model is justified by the independence of irrelevant alternatives (IIA) assumption. The IIA axiom states that the ratio of the probabilities of choosing any two alternatives is independent of the availability of other alternatives (e.g., Ben-Akiva and Lerman). Letting [P.sub.C](i) represent the probability of choosing alternative i from set C and [P.sub.C](j) represent the probability of choosing alternative j from set C, the IIA condition implies that:

(1) [P.sub.C](i)/[P.sub.C](j) = [P.sub.A](i)/[P.sub.A](j)

for (i, j) [member of] A ** C, i [not equal to] j (McFadden 2001). Thus, the odds of alternative i being chosen over alternative j in a polychotomous choice from a set C (with three or more alternatives) are the same odds of a binary choice of i over j from A, which is a two-alternative subset of C.

If data are consistent with IIA, then estimates of choice probabilities in expanded or reduced choice sets can be computed from the MNL model without re-estimation (Louviere, Hensher, and SwaiZt, p. 45). As noted by McFadden (1984, p. 1416), the IIA property implies that "... the MNL model can be estimated from data on alternatives sampled from the full choice set. In particular, the MNL model can be estimated from data on binary conditional choices."

The IIA axiom provides the basis for the Hausman-McFadden MNL specification test. In this case, some alternatives are "deleted" from the full choice set to perform the test. The test for the MNL model is based on the parameter estimates ([b.sub.C]), based on the full choice set C and a reduced choice set A, ([b.sub.A]). The test statistic is chi-square distributed:

(2) [chi square] = ([b.sub.A] - [b.sub.C])[[[V.sub.A] - [V.sub.C]].sup.-1] ([b.sub.A] - [b.sub.C])

where [V.sub.A] and [V.sub.C] are the respective estimates of the asymptotic covariance matrices. Rejection of the null hypothesis ([H.sub.0]: [b.sub.A] - [b.sub.C] = 0) indicates that the IIA property does not hold, or the model is misspecified, or both. If the IIA property holds, the vector of parameters estimated from a choice set with two alternatives is not statistically different from parameters estimated from the full choice set.

In the study conducted by BHTR, the full choice set was C = {1, 2, 3, 4}. Attribute levels were randomly assigned to each choice alternative in each choice set. Thus, the attribute levels associated with each alternative varied across alternatives for each individual in the sample and between individuals for each of the four alternatives. Considering subset [A.sup.1] = {1, 2} of the actual choice data, we are unable to reject the null hypothesis ([H.sub.0]: [b.sub.A1] - [b.sub.C] = 0) as having no difference between parameters estimated using the subset and the full set of alternatives ([chi square] = 14.25 < [[chi square].sub.0.05] = 23.68, df = 14). We are also unable to reject the null hypothesis of parameter equality between the binary choice model [A.sup.2] - {3, 4} and the full MNL model ([chi square] = 6.91 < [[chi square].sub.0.05] = 23.68, df = 14). Finally, the null hypothesis that parameters were no different based on responses contained in [A.sup.1] and [A.sup.2] could not be rejected ([chi square] = 4.38 < [[chi square].sub.0.05] = 23.68, df = 14). Thus, based on the results of these tests, we conclude that the parameters estimated by BHTR are not statistically different than MNL parameters estimated on the full choice set. While parameters estimated on subsets [A.sup.1] or [A.sup.2] individually will be less efficient than parameters estimated on the full choice set (C), it is not clear how efficiency will be affected by pooling subsets [A.sup.1] and [A.sup.2], which increases the number of observations in the binary model relative to MNL model.

Table 1 shows parameters estimated using a MNL model where respondents pick one of four alternatives, a binary logit model with four observations per respondent, and the binary probit model reported in BHTR. These models are based on BHTR's actual choice data. As anticipated, the parameter estimates and standard errors from the MNL and the binary logit models are very similar, particularly for attributes that are significant in both models. A comparison of the two harvesting plans described in BHTR (table 1) showed that estimates of compensating variation computed from the MNL and binary logit models only differ by 1.2%.

Finally, Holmes and Boyle analyze the BHTR rank and choice data with an MNL model and show that even after adjusting for differences in scale parameters, the hypothesis that taste parameters estimated from ranking and choice data are the same is rejected ([chi square] = 25.66 < [[chi square].sub.0.05] = 23.36, df = 13). Thus, the result of the primary objective of the BHTR article holds even when an MNL model is used.

Conclusion

In sum, we maintain the validity of the conclusions presented in BHTR; convergent validity of rating, rank, and choice data does not hold. The comment by Lusk is not capable of addressing the central theme of the BHTR article as simulation data, by definition, are free of messy, real-world complications such as the effects, which different response formats have on the information respondents, reveal about their preferences. Furthermore, results from the Hausman-McFadden specification test demonstrate that the BHTR data are consistent with the IIA property, and the parameter estimates using a binary logit model are indistinguishable from those of an MNL model. Thus, we contend that Lusk is incorrect in his statement "I contend that the estimates from the choose-one format may be inaccurate because of a modeling effort that did not account for all choices in the respondents' choice sets." (p. 1170). Even when an MNL model is used to compare the rank data, re-coded to choices, with the actual choice data, conver gent validity is not established. The more interesting and substantial issue remains, how and why do people behave differently when answering stated-preference questions that employ different response contexts?

Table 1

A Comparison of Multinomial Logit, Logit and Probit Coefficient
Estimates

Variables                 Multinomial Logit  Binary Logit  Binary Probit

Constant                         NA            -1.698 *       -1.015 *
                                               (0.273)        (0.156)
ROADS                           0.070           0.062          0.029
                               (0.153)         (0.148)        (0.087)
DEAD5                           0.576 *         0.512 *        0.310 *
                               (0.191)         (0.185)        (0.107)
DEAD10                          0.784 *         0.739 *        0.442 *
                               (0.187)         (0.184)        (0.107)
LIVE153                         0.813 *         0.753 *        0.432 *
                               (0.192)         (0.186)        (0.108)
LIVE459                         0.680 *         0.644 *        0.368 *
                               (0.192)         (0.183)        (0.105)
HOPEN35                        -0.014           0.027          0.089
                               (0.184)         (0.179)        (0.105)
HOPEN125                        0.024           0.034          0.026
                               (0.187)         (0.180)        (0.105)
H2OZONE                         0.034           0.014         -0.075
                               (0.149)         (0.147)        (0.086)
PERH50                          0.461 *         0.417 *        0.237 *
                               (0.179)         (0.175)        (0.104)
PERH80                         -0.150          -0.179         -0.091
                               (0.192)         (0.185)        (0.107)
REMSLASH                       -0.381 *        -0.343 *        -.201 *
                               (0.187)         (0.182)        (0.107)
DSTSLASH                        -156           -0.110         -0.072
                               (0.180)         (0.176)        (0.104)
TAX                            -0.0015 *       -0.0015 *      -0.0008
                              (0.0027)         (0.0027)       (0.0001)
N                                278             1112          1112
[chi square]([beta] = 0)         104             103            101

Note: Standard errors are reported in parenthesis.

* Significance at the 5% level.

[Received February 2002; final revision received February 2002.]

References

Ben-Akiva, M., and S.R. Lerman. Discrete Choice Analysis--Theory and Application to Travel Demand. Cambridge, MA: MIT Press, 1985.

Boyle, K.J., T.P. Holmes, M.F. Teisl, and B. Roe. "A Comparison of Conjoint Analysis Response Formats." Amer. J. Agr. Econ. 83(May 2001):441-54.

Hausman, J., and D. McFadden. "Specification Test for the Multinomial Logit Model." Econometrica. 52(1984):1219-40.

Holmes, T.P., and K.J. Boyle. 2003. "Using Stated Preference Methods to Estimate the Value of Forest Attributes." Forests in a Market Economy, E.O. Sills and K.L. Abt, eds., Dordrecht: Kluwer Academic Publisher.

Louviere, J.J., D.A. Hensher, and J.D. Swait. Stated Choice Methods-Analysis and Application. Cambridge: Cambridge University Press, 2000.

Lusk, J.L. "A Comparison of Conjoint Analysis Response Formats: Comment." Amer. J. Agr. Econ. 84 (2002):1165-71.

Maddala, G.S. Limited-Dependent and Qualitative Variables in Econometrics. Cambridge: Cambridge University Press, 1983

McFadden, D. "Econometric Analysis of Qualitative Response Models." Handbook of Econometrics, Z. Griliches and M.D. Intriligator, eds., New York: Elsevier Science, 1984

_____. "Economic Choices." Amer. Econ. Rev. 91 (December 2001):351-78.

Swait, J., and J. Louviere. "The Role of the Scale Parameter in the Estimation and Comparison of Multinomial Logit Models. J. Marketing Res. 30(1993):305-14.

Thomas P. Holmes is an economist with the U.S. Forest Service, Southeastern Forest Experiment Station; Kevin J. Boyle is the Libra Professor of Environmental Economics at the University of Maine; Mario F. Teisl is an associate professor at the University of Maine; and Brian Roe is an assistant professor at Ohio State University.

This research was supported by the U.S. Forest Service, South. eastern Forest Experiment Station; and the University of Maine Agricultural and Forest Experiment Station. Maine Agricultural and Forest Experiment Station Publication No. 2580.

*[Text unreadable in original source]