Evaluation of USDA interval forecasts of corn and soybean prices.

Agricultural prices are inherently unstable, primarily due to a combination of inelastic demand for food and production technology that is subject to the natural vagaries of weather, disease, and pests. Volatility of agricultural prices causes many individuals to rely on forecasts in their decision

making. Numerous studies demonstrate that the value of agricultural forecasts is substantial (e.g., Adam, Garcia, and Hauser; Byerlee and Anderson). The need for agricultural forecasts has long been addressed by the U.S. Department of Agriculture (USDA), which has provided both quantity and price forecasts of agricultural commodities since the 1920s (Kunze). It is a commonly held belief of market participants and analysts that USDA forecasts function as the "benchmark" to which other private and public estimates are compared (e.g., Irwin, Gerlow, and Liu; Kastens, Schroeder, and Plain).

Because of their significance, USDA forecasts have been the subject of analytical scrutiny since the 1950s (Allen). Examples of the latest studies include investigations of the market impact of USDA forecasts (e.g., Sumner and Mueller, McNew and Espinosa), informational content (e.g., Carter and Galopin), and accuracy (e.g., Bailey and Brorsen, Sanders and Manfredo 2002). Previous studies generally analyze USDA price forecasts as point estimates, but this is not always the form in which these forecasts are published.

A prominent example of USDA forecasting efforts is the WASDE (World Agricultural Supply and Demand Estimates) program, which provides monthly forecasts for major crops, both for the United States and the world. WASDE price forecasts (unlike all other WASDE forecasts) are published as an interval. Interval forecasts, different from point estimates, represent a range of values in which the realized value of the series is expected to fall with some (prespecified) probability (Diebold, p. 41). WASDE price forecasts are generated using a balance sheet approach, with published intervals reflecting uncertainty associated with prices in the future (Vogel and Bange). For example, the September 2002 WASDE forecast for the 2002-03 marketing-year-average farm price of corn was $2.35-2.75/bushel, and of soybeans, $5.15-6.05/bushel. Vogel and Bange note that, "The process of forecasting price and balance sheet items is a complex one involving the interaction of expert judgment, commodity models, and in-depth research by Department analysts on key domestic and international issues" (p. 10). This makes it clear that WASDE price forecasts are based on a variety of methods and information sources, but it is the expert judgment that combines and summarizes the analysis into published forecasts. Therefore, it appears appropriate to classify these forecasts as judgmental.

As noted above, WASDE price interval forecasts generally have been reduced to a midpoint in previous empirical analyses. This conversion causes a substantial loss of information regarding the uncertainty of forecasts. The more uncertain is the forecast, the wider is the interval relative to the average price. Thus, in the above example, the forecast range for corn is $0.40/bushel, while the forecast range for soybeans is $0.90/bushel. However, both ranges represent about 16% of the average crop price ($2.55/bushel for corn and $5.60/bushel for soybeans), and thus carry similar information about relative forecast uncertainty. The value of information about forecast uncertainty is particularly important for decision makers with different risk preferences. Rather than one possible outcome, as in the point forecast, interval forecasts give a range of possible outcomes, thereby, allowing for thorough contingency planning (Cristoffersen).

The need for probability and interval forecasting has been repeatedly expressed in the agricultural economics literature (e.g., Teigen and Bell, Timm, Bessler and Kling, Bessler). However, application and analysis of interval and probability forecasts have received very little attention. A few studies have been devoted to developing methods of constructing confidence intervals around point estimates (e.g., Prescott and Stengos, Bessler and Kling). Attempts to test such forecasts have been even less numerous and focused on model-based or market-based interval forecasts (e.g., Bessler, Fackler and King). To the best of our knowledge, only one study has investigated judgmental interval forecasts in an agricultural context. Sanders and Manfredo (2003) examined one-quarter-ahead forecasts for cattle, hogs, and broilers. They find that actual market prices fall in the forecasted ranges, a relatively small proportion of the time, between 35 and 48% of the quarters examined.

Interval and probability forecasts are more common in areas outside agricultural economics. Prominent examples are weather forecasts and sports picks. These types of forecasts have been studied to a large extent in the disciplines of behavioral economics and psychology. Recently, there has been rising interest in interval forecasting among financial economists (e.g., Taylor, Wallis, Berkowitz and O'Brien). Application of this knowledge to agricultural forecasts, and WASDE interval price forecasts in particular, may provide a better assessment of the quality of such forecasts, aid in their interpretation by forecast recipients, and assist analysts who provide these forecasts.

The purpose of this article is to evaluate the accuracy of WASDE interval price forecasts using methods suitable for testing judgmental interval forecasts. WASDE interval forecasts of corn and soybean prices for the 1980-81 through 2001-02 marketing years are examined. Corn and soybean price forecasts are of particular interest because these two crops account for about 80% of total U.S. grain and oilseed production. Empirical analysis of WASDE interval price forecasts proceeds in three steps. First, descriptive hit-and-miss statistics are presented and discussed. Second, unconditional and conditional tests of interval forecast accuracy developed by Christoffersen are introduced and applied to WASDE forecasts. Forecast accuracy is evaluated at two confidence level benchmarks, a 95% confidence level and an implied confidence level elicited from a small-scale survey of USDA analysts. Third, a behavioral framework developed by Yaniv and Foster (1995) that assumes forecast receivers' utility functions include not only accuracy, but also informativeness (specificity), is discussed and applied to the WASDE interval forecasts of corn and soybean prices.

The results of this study are expected to provide a fuller, more complete analysis of WASDE interval price forecasts by including analysis of uncertainty associated with these forecasts. Analysis of these forecasts based on more complete assumptions about forecast receiver's utility functions provides a new perspective compared to more traditional interval forecast analysis. These insights have not been possible from the analysis of these forecasts performed in previous studies.

Data

The subjects of this investigation are corn and soybean interval price forecasts from USDA WASDE reports over the 1980-81 through 2001-02 marketing years. These forecasts are part of reports released monthly by the USDA, usually between the 9th and 12th of the month. The first price forecast for a marketing year is usually available in May preceding the U.S. marketing year (September through August). Estimates are typically finalized by November of the following marketing year (figure 1). Thus, nineteen forecast updates of commodity prices are generated in the WASDE forecasting cycle each marketing year. (1) However, following April after harvest many of these forecasts converge to point estimates. Since the purpose of this paper is to analyze interval forecasts, only forecasts published from May prior to harvest though April after harvest are included. Note that forecasts in April after harvest converged to a point estimate once, in 1983. This observation was excluded from the analysis. Thus, 22 annual observations are available for each forecast month, except for April after harvest, where 21 annual observations are available.

[FIGURE 1 OMITTED]

Table 1 presents various descriptive statistics on WASDE interval price forecasts for corn and soybeans over 1980-81 to 2001-02. During the study period, average monthly price intervals were as wide as $0.39/bushel for corn and $1.33/bushel for soybeans in May prior to harvest. These average intervals narrowed to $0.15/bushel for corn and to $0.26/bushel for soybeans in April after harvest. The maximum price range was $0.60/bushel for corn and $2.50/bushel for soybeans. The magnitude of forecast intervals relative to the average forecast price prior to harvest was greater in soybeans than in corn, averaging about 20 and 16% of the average forecast price, respectively. (2) After harvest, the magnitude of relative forecast intervals between corn and soybean forecasts was comparable, averaging about 11 and 10% of the average forecast price, respectively. No trends in the magnitude of forecast intervals over time were detected. Thus, intervals in the same months did not become smaller (or larger) from the beginning to the end of the study period.

Unfortunately, WASDE forecasts are not accompanied by any information about the confidence levels associated with these forecasts. Information about confidence levels is necessary for interpretation and analysis of interval forecasts. Because of the absence of this information, assumptions had to be made for the purposes of this study. First, in the cases when the confidence level is not available, it is commonly assumed to be fairly high, about 90-95%, meaning that only 5-10% of the time-observed values are expected to fall outside the interval forecast. Second, more specific information about the confidence level associated with WASDE forecasts was generated from a survey of USDA analysts involved in compiling these forecasts. The following section describes how this survey was conducted and what data were generated.

Survey of Forecast Providers

An informal survey of USDA experts involved in compiling WASDE forecasts was conducted in August 2000. The survey was conducted via e-mail sent by an Economic Research Service (ERS) representative. The survey was sent to all ERS analysts and World Agricultural Outlook Board (WAOB) analysts involved in the WASDE corn and soybean forecasting process. The e-mail described the purpose of the survey and contained one question: "Each month, beginning in May prior to harvest, the WAOB presents a forecast of the marketing-year-weighted average price of corn and soybeans received by farmers. For each month, would you indicate the confidence, on average, that you have in the price forecast by indicating the percentage of time you think that the final price estimate for the marketing year will be within the forecast range presented that month." The response rate to this e-mail questionnaire was about 30%. The respondents included the Chair of the Feed Grains committee at the WAOB, the Director of the Feed Grains and Oilseeds Division at the Farm Service Agency (FSA), the senior feed grains analyst at the Foreign Agricultural Service (FAS), the senior soybean analyst at ERS, and the senior feed grains analyst at ERS. Each of the respondents has been involved in forecasting work at the USDA for an extended period of time, with an average of about 20 years. Thus, the survey respondents were knowledgeable about the WASDE forecasting process during the study period.

One respondent provided only a very general response, arguing that because of all the crop and yield uncertainty involved in the forecasting process in May prior to harvest this interval would have to be huge to correspond to even a 66% level. Unfortunately, this respondent did not provide any information about the other months of the forecasting cycle; therefore, this response was not included in the analysis. Considering that this person's confidence level for the May prior to harvest forecast was considerably lower than those of other respondents, it should be noted that exclusion of this respondent's information may bias reported confidence levels upward. In total, three complete responses for corn and four responses for soybean interval forecasts were available via the survey.

According to the survey results summarized in table 2, WASDE forecasters associate corn forecasts with an average of 77-94% levels of confidence and soybean forecasts with an average of 73-93% level of confidence for the months under investigation. Only one respondent indicated a 95% level of confidence associated with price forecasts for each month they are published. Analyst responses differed by as much as 30% in the beginning of the season (65 vs. 95% confidence level) and by as little as 5% late in the forecasting cycle (90 vs. 95% confidence level). Average confidence levels prior to harvest were below 86% for both crops. Confidence levels increased after harvest to 88-94% for corn and 85-93% for soybeans. The forecast uncertainty after harvest is caused by the continued variability in the parameters involved in the forecasting process and the revisions of monthly prices published by NASS (National Agricultural Statistical Service), associated with changes in monthly marketing weights. This information indicates that the confidence levels associated with WASDE interval price forecasts were, on average, lower than 95%. The average confidence levels generated by the survey are used in the following analysis as the confidence levels implied by the forecast providers.

Interval Forecast Accuracy

Many of the interval forecasts evaluated in the forecasting and finance literature are based on statistical models (see Chatfield for an overview). Consequently, among a number of approaches to testing interval forecast accuracy, most are based on the underlying statistical models. Accuracy tests based on statistical models are not directly applicable to judgmental interval forecasts. This section discusses both traditional tests and the most recent "model-free" tests of interval forecast accuracy proposed by Christoffersen and Christoffersen and Diebold.

Because WASDE provides fixed-event forecasts, a notation needs to be introduced that is tailored to a fixed-event framework. Fixed-event forecasts are a series of forecasts of the same terminal event ([y.sub.t]), such as monthly forecasts of the marketing-year-average price of corn. Within this framework, the annual nature of terminal events is indexed by t, the number of forecasting cycles within the study period, and the monthly nature of forecasts by k, the index of forecast dates within each forecasting cycle. Then, [I.sup.k.sub.t] is an indicator variable for a given interval forecast,

(1) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.]

where [[l.sub.t/k]([alpha]), [u.sub.t/k] ([alpha])] are the lower and upper limits of the interval forecast for terminal event [y.sub.t] made at time k with coverage probability cc This study examines WASDE forecasts during the 1980-81 to 2001-02 marketing years and thus t = 1, ... , 22. As mentioned before, 19 forecast updates are available for each marketing year and thus k = 1, ... , 19 (figure 1). Because forecasts near the end of the cycle were often reduced to a single value, this study focuses on interval forecasts for k = 1, ... , 12 (May before harvest through April after harvest). The last forecast of each cycle (November of the following marketing year, k = 19) is considered the best estimate of the terminal event ([y.sub.t]). Because WASDE forecasts are not accompanied by any information about the coverage probability, [alpha], two benchmarks were used, the 95% benchmark and the implied confidence level benchmark elicited from the survey of forecast providers.

[FIGURE 1 OMITTED]

Traditional measures of interval forecast accuracy are hit rates and forecast coverage. Hit rates describe the proportion of times the forecast intervals contain the final or "true" value ([y.sub.t]) and may be defined as E([I.sup.k.sub.t]). Forecast coverage examines whether the proportion of times the forecast interval includes the true value corresponds to a stated confidence level, or, in other words, if the interval hit rate is equal to the coverage probability. Thus, forecast coverage may be examined by testing the hypothesis [H.sub.0]: E([I.sup.k.sub.t]) = [alpha] against [H.sub.1]: E([I.sup.k.sub.t]) [not equal to] [alpha]. If [H.sub.0] is not rejected and the interval hit rate is equal to the coverage probability, forecasts are said to be calibrated. The likelihood function for the indicator variable [I.sup.k.sub.t], which has a binomial distribution, is (Christoffersen)

(2) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.]

under the null hypothesis and

(3) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.]

under the alternative hypothesis, where [n.sub.1] and no are the number of times an interval was "hit" (1) or "missed" (0) in the indicator sequence [I.sup.k.sub.t], and L is a likelihood function. Then, forecast coverage may be tested via the likelihood ratio test,

(4) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCCI.]

(unconditional coverage test)

where p = [n.sub.1]/([n.sub.0] + [n.sub.1]) is the maximum likelihood estimator of p. Because this test does not imply anything about the underlying information set, Christoffersen termed it an unconditional coverage test. He argued, however, that in addition to coverage, interval forecasts should be dynamic, in the sense of being "narrow in tranquil times and wide in volatile times, so that the occurrences of observations outside the interval forecast would be spread out over the sample and not come in clusters" (p. 842). Unconditional coverage does not provide any information about clustering of outliers when the forecasts fail to account for higher-order dynamics. Therefore, Christoffersen proposed additional tests that examine interval forecast independence and forecast coverage conditional on independence.

Christoffersen proposed testing independence of the indicator sequence [I.sup.k.sub.t] against an explicit first-order Markov alternative. First, define the transition probability of the first-order Markov chain for a given forecast date [kappa] as [[pi].sub.ij] = Pr([I.sup.k.sub.t] = J/[I.sup.k.sub.t-1] = i), where j = 1, 0 and i = 1, 0.3 Then, the likelihood ratio test of independence is given by

(5) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.]

(independence test)

where [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.], where [n.sub.ij] is the number of observations with value i followed by j, [[pi].sub.01] = [n.sub.01]/ ([n.sub.00] + [n.sub.01]), and [[pi].sub.11] = [n.sub.11]/([n.sub.10] + [n.sub.11]).

The conditional coverage test combines an unconditional coverage test (equation (4)) with a test of forecast independence (equation (5)) to account for higher order dynamics of time-series forecasts. The unconditional coverage and independence tests are additive within the likelihood ratio framework:

(6) [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.]

(conditional coverage test)

Thus, the conditional coverage test combines unconditional coverage and independence while retaining the individual hypotheses as subcomponents. According to Christoffersen, this test allows determination of whether "a given interval forecast deserves the label "good" (p. 842).

Note that, if any of the [n.sub.ij] elements in equation (5) are equal to zero, the independence test, and, therefore, the conditional coverage test is undefined. An alternative indication of interval forecasts independence proposed by Christoffersen and Diebold is:

(7) Corr([I.sup.k.sub.t], [I.sup.k.sub.t-1]) = [[pi].sub.11] - [[pi].sub.01].

This statistic provides an estimate of the first-order serial correlation of the indicator sequence [I.sup.k.sub.t]. (4) Because of the complex properties of the indicator sequence if, it is difficult to assess the underlying sampling distribution and, therefore, to test the statistical significance of such correlation coefficients. However, these correlation coefficients provide qualitative evidence about independence in cases when the L[R.sub.i] and L[R.sub.cc] cannot be computed.

The tests discussed in this section were applied to the WASDE interval forecasts of corn and soybean prices during 1980-81 to 2001-02 marketing years. The following section discusses the results of the empirical analysis.

Empirical Results

The first step of the empirical analysis of WASDE forecast accuracy is to examine descriptive hit/miss statistics. Table 3 shows hit rates for individual months ranging from 36 to 82% for corn and from 59 to 86% for soybeans. These hit rates were consistently lower than both the 95% confidence level and the level implied by the forecast providers (see table 2), with only one exception (August soybeans). Prior to harvest, hit rates were much higher for soybeans than for corn, averaging 73 and 50%, respectively. This implies that, on average, corn price interval forecasts prior to harvest contained the final price estimate only 50% of the time. After harvest, the hit rates for both commodities significantly improve, averaging about 77% for corn and 80% for soybean price interval forecasts. However, there were several exceptions to the improvement of the hit rates during the season. Hit rates decrease in March and April for corn and in October and November for soybeans. The decrease in the hit rates for corn at the end of the forecasting cycle may be explained by premature tightening of the forecast intervals. The decrease in the hit rates for soybeans is due to a sharp increase in soybean price volatility in the middle of the forecasting cycle. It appears that soybean price forecasts did not account for some additional information shocks in October and November (in order to maintain accuracy intervals would have had to be widened in these months). Overall, based on hit rates, soybean interval price forecasts appear more accurate than corn price forecasts prior to harvest. After harvest, the accuracy of interval price forecasts for both commodities is comparable.

One of the issues associated with interval forecast coverage is the symmetry of the forecast intervals and the underlying price distributions. Statistics on the proportion of misses above and below the forecast interval reported in table 3 provide some insight on this issue. The statistics indicate that the proportion of misses above tends to dominate the proportion of misses below the forecast intervals in both commodities for the first three months prior to harvest and the last two to four months after harvest. Furthermore, the magnitude of misses tends to be skewed toward misses above in seven out of twelve months in corn and in all cases in soybeans. This evidence may indicate that forecast intervals published by USDA are constructed symmetrically, while the distribution of forecasted prices is asymmetric. According to theory, spot prices of storable commodities tend to have highly skewed distributions with a long tail toward high prices (Williams and Wright, p. 105). It appears that WASDE interval forecasts do not reflect this asymmetry. (5)

Results of the formal accuracy tests of WASDE interval forecasts of corn and soybean prices are presented in table 4. As mentioned above, in these tests, the accuracy of WASDE forecasts is compared to two benchmarks: the 95% confidence level and the implied confidence level elicited from a survey of forecast providers. According to the unconditional coverage tests (columns 1 and 3), the calibration of WASDE corn price forecasts is always rejected at the 95% confidence level and generally rejected (three exceptions) at the implied confidence level. (6) In soybeans, forecast interval calibration is generally rejected (with two exceptions) at the 95% confidence level and not rejected at the implied confidence level. Thus, this initial test generally rejected forecast accuracy for both commodities at the 95% confidence level and for corn at the implied confidence level. Only soybean price interval forecasts were generally calibrated at the implied confidence level.

Interval forecast independence is typically not rejected in both commodities, with three exceptions for corn and two exceptions for soybeans. This result implies that WASDE forecasters dynamically adjusted their forecasts over time by publishing narrower forecasts in tranquil times and wider forecasts in volatile times. (7) However, because of the presence of zero observations for consecutive misses, the independence test was undefined in three cases for corn and in seven cases for soybeans. Serial correlation of each indicator series was computed as an alternative measure of forecast independence. Forecast independence is generally rejected in cases where serial correlation exceeded [+ or -]0.30. In most of the undefined cases (with one exception) serial correlation was below this level; therefore, it is not likely that forecast independence would have been rejected in those cases.

Finally, conditional coverage tests combine the results of unconditional coverage and independence tests and increase the degrees of freedom from one to two in the [chi square] statistic. For cases where the conditional test is defined, the results (columns 2 and 4) indicate that calibration of corn price forecasts was rejected at the 95% confidence level and at the implied confidence level (one exception). Calibration of soybean price forecasts was rejected at the 95% level and not rejected at the implied confidence level. The conditional coverage test results are ambiguous in the cases with undefined independence tests. Some insight on these test results may be attained by reviewing the individual components of this test (unconditional coverage and independence) and comparing them to a critical value of a [chi square] distribution in order to determine if they would have yielded statistically significant results. A critical value of the [chi square] distribution with two degrees of freedom at [alpha] = 0.95 is 5.99. The lowest possible value of the independence test is zero. Thus, if the value of an unconditional coverage test is greater than 5.99, it would also result in a significant conditional coverage test regardless of the result of the independence test. According to this logic, April corn forecasts would yield significant conditional coverage results at both confidence levels and October and January soybean forecast tests would be significant at the 95% confidence level. The combination of unconditional coverage and serial correlation as an indication of independence is suggestive of significant conditional coverage test results in January and February for corn and September, December, February, and March for soybeans at the 95% confidence level. Other undefined cases remain ambiguous.

The above findings were not surprising and concurred with the results of many previous studies. Previous analyses revealed low hit rates associated with 90-98% confidence intervals in a wide variety of applications: 24-62% (Trip, Huirne, and Renkema), 53-81% (Alpert and Raiffa), 60% (Lichtenstein and Fischhoff), 20-58% (Russo and Schoemaker), 43-55% (Yaniv and Foster 1997). Hit rates have been low even among the experts: Sanders and Manfredo (2003) report 35-48% hit rates for USDA forecasts of livestock prices and Russo and Schoemaker report 42-61% hit rates for a job-relevant quiz of business managers. Such findings are commonly interpreted as a result of overconfidence on the part of forecast providers (e.g. Lichtenstein, Fischhoff, and Phillips). However, alternative explanations are possible depending on the assumptions about forecast receivers' utility function. All of the tests described and applied above examine one aspect of interval forecast accuracy, namely, forecast coverage. While obviously important, interval forecast coverage may not provide a complete picture of forecast value. Focusing solely on interval forecast coverage implies that forecast receivers' utility increases in a monotonic fashion as forecast intervals become wider. However, little use can be made from forecasts that are too wide, even though they may have perfect coverage. For example, a forecast of the price of soybeans next year to fall between $1.00/bu and $10.00/bu is very likely to include the "true" value at the 100% confidence level. It would "pass" each accuracy test applied so far in this study (as long as it is independent from previous forecasts). The problem with this forecast is that it is not very specific. Grice argued that "conversational norms suggest that forecasts should be appropriately informative as well as accurate" (p. 42). This implies that excessively wide intervals (that are more likely to be accurate) may not be preferred by forecast receivers. Thus, an assumption that the loss function of forecast receivers is based solely on coverage may need to be extended to include other aspects, such as informativeness (specificity). A behavioral theory that includes both of these aspects is discussed and applied in the following section.

Accuracy-Informativeness Trade-Off Model

Yaniv and Foster's (1995) model is derived from their observations in the area of experimental psychology Rather than taking a normative approach to the issue of poorly calibrated judgmental intervals and exploring corrective procedures for it, Yaniv and Foster attempted to interpret judgment under uncertainty as a part of the communication process between forecast "senders" and "receivers." They argued that the evaluation of uncertain judgments involves a trade-off between two competing objectives: accuracy and informativeness. Experiments presented in their work demonstrate that individuals sometimes are willing to accept errors in the interest of securing more informative, or specific, judgments. Assuming that forecast providers respond to recipients' expectations, their objectives may be viewed as a continuous function in two dimensions: accuracy, expressed as a measure of distance of an interval from the truth, and informativeness, measured as the specificity of the estimate. Thus, the loss function that forecast providers try to minimize may be presented in a formal model of the form,

(8) L = f [[absolute value of y - m/g], ln(g)]

where the first element represents accuracy, with y as the true outcome and m as the midpoint of an interval forecast, and the second element reflects informativeness, with g being the width of the interval. Yaniv and Foster assume that f is a monotonically increasing function of its arguments and propose an additive form,

(9) L = [f.sub.1][absolute value of y - m/g] + [f.sub.2][ln(g)].

The trade-off occurs because as interval width (g) increases, the first element [[absolute value of y - m/g]] describing accuracy decreases, while the second element describing informativeness (ln(g)) increases. Because accuracy is measured by the forecast error relative to the interval width, a decrease in this element would indicate a smaller relative error and, therefore, a more accurate forecast. On the other hand, an increase in the informativeness (ln(g)) element would indicate a wider (less informative) interval. Thus, an improvement in either direction would be indicated by a lower value of the element; and the lowest L score would indicate a forecast interval that is likely to be preferred by receivers. For simplicity, Yaniv and Foster substituted individual functions [f.sub.1] and [f.sub.2] with the identity function and the coefficient [gamma], respectively, resulting in a specific model of the form,

(10) L = [absolute value y - m]/g + [gamma] ln(g)

where [gamma] is a trade-off parameter that reflects the weights placed on accuracy and informativeness of the intervals by forecast receivers ([gamma] [greater than or equal to] 0). Yaniv and Foster's (1995) experimental data suggested that the value of [gamma] was close to 1 (from 0.6 to 1.2). The authors also demonstrated that the fit of their trade-off model to experimental results was superior to that of alternative models. (8) The correlation of the model rankings to experimental rankings was 84%.

The accuracy-informativeness trade-off model (10) was applied to three different scenarios for WASDE forecasts: (a) actual WASDE price interval forecasts for corn and soybeans published from 1980/81-2001/02, (b) WASDE interval forecasts with ranges increased to correspond to the implied confidence levels, and (c) WASDE interval forecasts with ranges increased to correspond to the 95% confidence levels. (9) This analysis assumed that [gamma] = 1, following Yaniv and Foster's (1995) experimental observations. (10) L scores were computed for each observation and then the average L score for each month was used for the final comparison.

The results of this analysis are reported in table 5. The L scores based on actual forecast intervals (first scenario) were consistently lower for both crops before and after harvest. L scores were equivalent between the published intervals and the implied confidence level intervals (second scenario) only once, when these intervals were identical (August soybeans). The scores for the third scenario, based on the 95% confidence level forecast ranges, were consistently higher (worse) than the scores for the other scenarios, except for cases where the implied confidence level and the 95% confidence level were identical. These results suggest that the information users obtain from published forecast intervals may present more value than wider (more accurate) intervals (such as forecasts calibrated at the implied and the 95% confidence levels). Thus, forecast users may prefer narrower, more informative forecast intervals, even though they may be associated with lower confidence levels.

Summary and Conclusions

This article uses methods suitable for testing judgmental interval forecasts to evaluate WASDE interval forecasts of corn and soybean prices for the 1980/81 through 2001/02 marketing years. The first step of the analysis concentrates on descriptive hit/miss statistics, which show whether the actual price was contained in the forecast interval. Hit rates ranged from 36 to 82% for corn and from 59 to 86% for soybeans. Prior to harvest, hit rates were much higher for soybeans than for corn, averaging 73 and 50%, respectively. After harvest, the hit rates for both commodities become comparable, averaging about 77% for corn and 80% for soybean price interval forecasts. Actual prices are much more likely to be above the forecast intervals than to be below. Thus, forecast intervals published by the USDA are apparently constructed symmetrically, while the distribution of forecast prices is asymmetric.

Because WASDE does not state the confidence levels of their interval forecasts, forecast coverage was tested using two benchmarks: a 95% confidence level and an implied confidence level elicited from a small-scale survey of forecast providers. A traditional, unconditional coverage test revealed that WASDE interval forecasts were generally not calibrated at the 95% confidence level in both commodities. Calibration of these forecasts at the implied confidence level was generally rejected in corn, but not rejected in soybeans. Christoffersen extended this traditional test to include dynamics of the time-series forecasts via a test of independence. Application of this test to WASDE price interval forecasts suggested that these forecasts were generally independent across marketing years. A joint test of unconditional coverage and independence (conditional coverage test) rejected forecast calibration at the 95 % level for both commodities and for corn at the implied confidence level. The joint test did not reject calibration for soybeans at the implied confidence level.

WASDE interval forecasts were also examined within the framework proposed by Yaniv and Foster (1995), which suggests that evaluation of judgmental forecasts includes a trade-off between accuracy and informativeness. Within this framework WASDE interval forecasts were ranked higher than alternative forecasts calibrated at the implied and the 95% confidence levels. These results suggest that forecast users may prefer narrower, more informative forecast intervals, even though they may be associated with lower confidence levels.

Based on the results of this study, it is possible to offer some suggestions that may improve WASDE interval forecasts of corn and soybean prices. First, forecast intervals published by USDA are constructed symmetrically, while the distribution of forecast prices is asymmetric. WASDE forecast intervals could be adjusted to reflect this asymmetry. Second, WASDE interval forecasts are incomplete since no probability is assigned to them. Because of the absence of the exact confidence level information, it is likely that forecast users make their own assumptions about WASDE forecast coverage, which may not be correct. Therefore, it would be helpful both from the standpoint of forecast interpretation by users and forecast evaluation by analysts if interval forecasts published by USDA were "completed" by providing underlying confidence levels.

Finally, based on the accuracy-informativeness model results, even though the hit rates associated with WASDE forecasts have not been particularly high, WASDE interval forecasts should not necessarily be widened in order to achieve some high level of confidence. Our findings suggest that the information users obtain from published forecast intervals may present more value than wider (more accurate) intervals (such as forecasts calibrated at the implied and the 95% confidence levels). This conclusion, however, is based on the assumption that accuracy-informativeness model accurately represents the preferences of forecast users. Yaniv and Foster's model was calibrated to the responses of the university students. Their preferences may not necessarily correspond to the preferences of WASDE forecast receivers. Therefore, further research on preferences of WASDE forecast receivers and calibration of the applied model is needed.

[Received February 2003; accepted January 2004.]

Table 1. Descriptive Statistics for WASDE Interval Forecasts of Corn
and Soybean Prices, 1980-81 to 2001-02 Marketing Years

                    Average Forecast    Average     Minimum
Crop/Month               Price          Interval    Interval

Corn
Prior to harvest
  May                     2.31            0.39        0.20
  June                    2.33            0.39        0.20
  July                    2.39            0.38        0.20
  August                  2.42            0.38        0.20
  September               2.42            0.36        0.20
  October                 2.42            0.36        0.20
After harvest
  November                2.41            0.36        0.20
  December                2.39            0.33        0.20
  January                 2.39            0.29        0.15
  February                2.39            0.25        0.15
  March                   2.38            0.19        0.10
  April                   2.37            0.15        0.10
Soybeans
Prior to harvest
  May                     5.82            1.33        0.40
  June                    5.82            1.28        0.40
  July                    5.87            1.24        0.30
  August                  5.95            1.24        0.30
  September               6.04            1.10        0.30
  October                 6.01            0.99        0.30
After harvest
  November                5.97            0.90        0.30
  December                5.97            0.80        0.30
  January                 5.93            0.68        0.20
  February                5.91            0.59        0.15
  March                   5.87            0.45        0.15
  April                   5.88            0.26        0.10

                    Maximum
Crop/Month          Interval

Corn
Prior to harvest
  May                 0.60
  June                0.60
  July                0.50
  August              0.50
  September           0.50
  October             0.40
After harvest
  November            0.40
  December            0.40
  January             0.40
  February            0.40
  March               0.30
  April               0.30
Soybeans
Prior to harvest
  May                 2.50
  June                2.50
  July                2.50
  August              2.50
  September           2.50
  October             2.50
After harvest
  November            2.50
  December            2.50
  January             1.25
  February            1.25
  March               1.00
  April               0.50

Note: The averages are based on 22 annual observations, except April,
which is based on 21 observations. Average forecast price is
calculated by averaging the midpoints of forecast intervals. All values
are in $ per bushel.

Table 2. Confidence Levels for WASDE Corn and Soybean Price Interval
Forecasts Based on the Survey of USDA Analysts

                         Individual Respondent

Crop/Month          1     2     3     4     Average

Corn
Prior to harvest
  May               65    70          95      77
  June              65    70          95      77
  July              70    75          95      80
  August            80    80          95      85
  September         80    80          95      85
  October           80    80          95      85
After harvest
  November          80    90          95      88
  December          80    90          95      88
  January           85    90          95      90
  February          85    95          95      92
  March             85    95          95      92
  April             90    97          95      94
Soybeans
Prior to harvest
  May               65    65    65    95      73
  June              65    65    65    95      73
  July              70    70    70    95      76
  August            80    75    75    95      81
  September         80    80    75    95      83
  October           80    80    80    95      84
After harvest
  November          80    85    80    95      85
  December          80    85    80    95      85
  January           85    85    85    95      88
  February          85    90    85    95      89
  March             85    90    85    95      89
  April             90    95    90    95      93

Note: Responses for corn and soybeans do not necessarily belong to
the same individuals. All values are in percentage.

Table 3. Descriptive Accuracy Statistics for WASDE Interval Forecasts
of Corn and Soybean Prices, 1980-81 to 2001-02 Marketing Years

                      Hit        Misses       Misses
Crop/Month          Rate (%)    Below (%)    Above (%)

Corn
Prior to harvest
  May                  45          18           36
  June                 36          27           36
  July                 50          23           27
  August               55          32           14
  September            55          27           18
  October              55          27           18
After harvest
  November             73          14           14
  December             82           9            9
  January              82           9            9
  February             82           9            9
  March                68          14           18
  April                76           9           14
Soybeans
Prior to harvest
  May                  59          18           23
  June                 59          18           23
  July                 73           9           18
  August               86           5            9
  September            82           9            9
  October              77          14            9
After harvest
  November             73          14           14
  December             82           9            9
  January              77           5           18
  February             82           0           18
  March                82           0           18
  April                86           0           14

                     Avg. Miss       Avg. Miss
Crop/Month          Below ($/bu)    Above ($/bu)

Corn
Prior to harvest
  May                   0.21            0.28
  June                  0.19            0.26
  July                  0.26            0.19
  August                0.15            0.20
  September             0.16            0.16
  October               0.11            0.11
After harvest
  November              0.18            0.13
  December              0.15            0.20
  January               0.10            0.20
  February              0.10            0.15
  March                 0.05            0.06
  April                 0.05            0.05
Soybeans
Prior to harvest
  May                   0.36            0.57
  June                  0.36            0.57
  July                  0.29            0.51
  August                0.05            0.71
  September             0.40            0.71
  October               0.30            0.53
After harvest
  November              0.35            0.51
  December              0.15            0.42
  January               0.10            0.18
  February               NA             0.16
  March                  NA             0.15
  April                  NA             0.15

Notes: Results are based on 22 annual observations for each month,
except April, which contains 21 observations. NA implies it is not
possible to compute a statistic because of zero observations. The
total of misses below (%) and misses above (%) may not equal 100 minus
the hit rate (%) due to rounding.

Table 4. Accuracy Tests of WASDE Interval Forecasts of Corn and Soybean
Prices, 1980-81 to 2001-02 Marketing Years

                      Implied Conf. Level

                    Uncond.     Conditional
Crop/Month          Coverage     Coverage

Corn
Prior to harvest
  May               10.18 **     12.74 **
  June              16.49 **     17.84 **
  July               9.82 **     11.30 **
  August            11.53 **     13.96 **
  September         11.53 **     12.82 **
  October           11.53 **     15.49 **
After harvest
  November           3.75 *       9.44 **
  December           0.70         4.75
  January            1.35          NA
  February           2.35          NA
  March             10.34 **     13.87 **
  April              7.06 **       NA
Soybeans
Prior to harvest
  May                1.98         5.89
  June               1.98         5.89
  July               0.13         3.10
  August             0.45          NA
  September          0.02          NA
  October            0.67         1.26
After harvest
  November           2.18         3.46
  December           0.17          NA
  January            1.97          NA
  February           0.99          NA
  March              0.99          NA
  April              1.34          NA

                        95% Cent. Level

                    Uncond.     Conditional
Crop/Month          Coverage     Coverage

Corn
Prior to harvest
  May               42.61 **     45.17 **
  June              55.86 **     57.20 **
  July              36.54 **     38.01 **
  August            30.83 **     33.26 **
  September         30.83 **     32.12 **
  October           30.83 **     34.79 **
After harvest
  November          11.81 **     17.49 **
  December           4.95 *       9.00 **
  January            4.95 *         NA
  February           4.95 *         NA
  March             15.96 **     19.49 **
  April              8.55 **        NA
Soybeans
Prior to harvest
  May               25.49 **     29.40 **
  June              25.49 **     29.40 **
  July              11.81 **     14.79 **
  August             2.40           NA
  September          4.95 *         NA
  October            8.12 **      8.70 **
After harvest
  November          11.81 **     13.09 **
  December           4.95 *         NA
  January            8.12 **        NA
  February           4.95 *         NA
  March              4.95 *         NA
  April              2.60           NA

                                      Serial
Crop/Month          Independence    Correlation

Corn
Prior to harvest
  May                  2.56           -0.25
  June                 1.34           -0.14
  July                 1.48           -0.05
  August               2.44            0.24
  September            1.30            0.05
  October              3.97 *          0.33
After harvest
  November             5.68 *          0.37
  December             4.05 *          0.34
  January                NA           -0.18
  February               NA           -0.18
  March                3.53           -0.21
  April                  NA           -0.27
Soybeans
Prior to harvest
  May                  3.91 *          0.31
  June                 3.91 *          0.31
  July                 2.98           -0.10
  August                 NA           -0.17
  September              NA           -0.22
  October              0.58           -0.05
After harvest
  November             1.28           -0.17
  December               NA           -0.24
  January                NA           -0.31
  February               NA           -0.24
  March                  NA           -0.24
  April                  NA           -0.19

Notes: Results are based on 22 annual observations for each month,
except April, which contains 21 observations. NA stands for undefined
test values. Unconditional coverage and independence tests are
distributed asymptotically as [chi](1), while the conditional coverage
test is distributed as [chi](2). * and ** indicate statistical
significance at the 5%. and 1% levels, respectively.

Table 5. Accuracy-Informativeness Trade-Off Scores for WASDE Interval
Forecasts of Corn and Soybean Prices, 1980-81 to 2001-02 Marketing
Years

                                  Implied         95%
                    Published    Confidence    Confidence
Crop/Month          Intervals    Intervals     Intervals

Corn
Prior to harvest
  May                -0.18#         0.28          0.58
  June               -0.19#         0.28          0.40
  July               -0.30#         0.16          0.24
  August             -0.39#        -0.07          0.33
  September          -0.42#         0.00          0.17
  October            -0.53#        -0.25         -0.05
After harvest
  November           -0.61#        -0.09          0.01
  December           -0.75#        -0.58         -0.05
  January            -0.92#        -0.56         -0.38
  February           -1.09#        -0.46         -0.46
  March              -1.29#        -0.81         -0.81
  April              -1.58#        -0.97         -0.97
Soybeans
Prior to harvest
  May                 0.71#         0.83          1.32
  June                0.67#         0.79          1.30
  July                0.66#         0.68          1.02
  August              0.56#         0.56          0.78
  September           0.45#         0.49          1.09
  October             0.31#         0.43          1.05
After harvest
  November            0.23#         0.49          1.01
  December            0.02#         0.28          0.58
  January            -0.13#         0.07          0.36
  February           -0.29#        -0.17          0.00
  March              -0.62#        -0.45         -0.36
  April              -1.13#        -0.91         -0.91

Notes: Results are based on 22 annual observations for each month,
except April, which contains 21 observations. Scores are monthly
averages based on accuracy-informativeness trade-off from Yaniv and
Fosters model. Lowest (best) scores are highlighted in bold.

Note: Lowest (best) scores are indicated with #.

The funding support of the Economic Research Service of the U.S. Department of Agriculture under Cooperative Agreement No. 43-3AEK-8-80106 is gratefully acknowledged. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of the U.S. Department of Agriculture. The assistance of Joy Harwood with the survey of forecast providers is greatly appreciated.

(1) WASDE reports became available in 1973/74, but price interval forecasts were not part of these early publications. Price interval forecasts were first published in 1976/77, but these forecasts were published sparingly. Because of many missing observations, price interval forecasts published from 1976/77 to 1979/80 are not included in the sample. The current calendar of forecast releases described in the text was adopted in 1985/86. Prior to that, two forecasts of crop prices were published in some months. Our analysis uses forecasts published at the beginning of the month, similar to the current calendar of releases. The results were consistent when the forecasts published at the end of the month were used (not presented here).

(2) Average forecast price is computed by taking an average of midpoint forecast prices for each month.

(3) Within a fixed-event framework forecast independence should be tested across events ([I.sup.k.sub.t], [I.sup.k.sub.t-1]), which in this case means across marketing years. For example, the May 2003 forecast should be compared with the May 2002 forecast. This is different from testing independence among forecasts of the same event, such as April vs. May 2003 forecasts. Because these are forecasts of the same event, data horizons are overlapping and independence is not possible.

(4) Christoffersen and Diebold demonstrate that equation (6) is the first-order serial correlation coefficient of the hit sequence based on the following results:

E[[I.sup.k.sub.t]] = [alpha] = [alpha][[pi].sub.11] + [(1 - [alpha])[pi].sub.01] = [[pi].sub.01]/1 + [[pi].sub.01] - [[pi].sub.11]

Var[[I.sup.k.sub.t]] = [alpha](1 - [alpha]) = [[pi].sub.01] (1 - [[pi].sub/11])/1 + [[pi].sub.01] - [[pi].sub.11]

Cov ([I.sup.k.sub.t], [I.sup.k.sub.t-1]) = E[[I.sup.k.sub.t] [I.sup.k.sub.t-1]] - [E.sup.2][[I.sup.k.sub.t]] = [[alpha].sup.2] = [alpha]([[pi].sub.11] - [alpha]).

Thus, the correlation coefficient is

Corr([I.sup.k.sub.t] [I.sup.k.sub.t-1]) = [[pi].sub.11] - [alpha]/1 - [alpha] = [[pi].sub.11](1 + [[pi].sub.01] - [[pi].sub.11] - [[pi].sub.01]/1 - [[pi].sub.11] = [[pi].sub.11] - [[pi].sub.01].

(5) This assertion is consistent with a survey response, which indicated that "each month a midpoint is forecast using the U.S. and global supply and use and then a range is put on each side of the midpoint."

(6) Statistical differences from the implied confidence level should be interpreted with care. because the implied confidence levels obtained from the survey may be biased upwards due to the excluded observation.

(7) Forecast intervals were significantly positively correlated with squared forecast errors (final estimate minus midpoint forecast price), with correlation coefficients equaling 0.32 and 0.18 for soybeans and corn, respectively. This finding indicates that forecast intervals were larger in marketing years with greater uncertainty.

(8) Results were compared to absolute error plus half-width model, nearest boundary model, lexicographic semiorder model, absolute error model, normalized error model, interval-width model, and inclusion model (Yaniv and Foster, 1995, p. 429).

(9) In order to correspond to a 95% confidence level, twenty-one out of twenty-two intervals should have included the final estimate. Therefore, the second largest distance of the final estimate from the interval was added to both sides of each interval. Similar calculations were performed for the implied confidence levels with different distances added to reflect respective confidence levels. This analysis assumes that forecast intervals were symmetric. As noted in footnote (5), this assumption is consistent with the observations of one survey respondent. Additionally, forecast intervals were tested for bias. A standard t-test revealed that average forecast errors were not statistically different from zero and thus interval forecasts were unbiased during the period of study.

(10) The results of this analysis were not sensitive to [gamma] = 0.6 and [gamma] = 1.2, the extremes found in Yaniv and Foster's experiments.

References

Adam, B.D., P Garcia, and R. J. Hauser. "The Value of Information to Hedgers in the Presence of Futures and Options." Review of Agricultural Economics 18(1996):437-47.

Allen, G.P. "Economic Forecasting in Agriculture." International Journal of Forecasting 10(1994):81-135.

Alpert, M., and H. Raiffa. "A Progress Report on the Training of Probability Assessors." In D. Kahneman, P. Slovic, and A. Tversky, eds. Judgment under Uncertainty: Heuristics and Biases. New York: Cambridge University Press, 1982, pp. 294-305.

Bailey, D.V., and B.W. Brorsen. "Trends in the Accuracy of USDA Production Forecasts for Beef and Pork." Journal of Agricultural and Resource Economics 23(1998):515-26.

Berkowitz, J., and J.O' Brien. "How Accurate Are Value-at-Risk Models at Commercial Banks?" Journal of Finance 57(2002):1093-1111.

Bessler, D.A. "Subjective Probability." In P.J. Barry, ed. Risk Management in Agriculture. Ames: Iowa State University Press, 1984, pp. 43-52.

Bessler, D.A., and J.L. Kling. "The Forecast and Policy Analysis." American Journal of Agricultural Economics 71(1989):503-6.

Byerlee, D., and J.R. Anderson. "Risk, Utility, and the Value of Information in Farmer Decision Making." Review of Marketing and Agricultural Economics 50(1982):231-46.

Carter, C.A., and C.A. Galopin. "Informational Content of Government Hogs and Pigs Reports." American Journal of Agricultural Economics 75(1993):711-18.

Chatfield, C. "Calculating Interval Forecasts." Journal of Business and Economics Statistics 11(1993):121-35.

Christoffersen, P.F. "Evaluating Interval Forecasts." International Economic Review 39(1998):841-62.

Christoffersen, P.F., and F.X. Diebold. "How Relevant Is Volatility Forecasting for Financial Risk Management?" Review of Economics and Statistics 82 (2000):12-22.

Diebold, F.X. Elements of Forecasting. Cincinnati: South-Western College Publishing, 1998.

Fackler, P.L., and R.P. King. "Calibration of Option-Based Probability Assessments in Agricultural Commodity Markets." American Journal of Agricultural Economics 72(1990):73-83.

Grice, H.P. "Logic and Conversation." In E Cole and J.L. Morgan, eds. Syntax and Semantics 3: Speech Acts. New York: Academic Press, 1975, pp. 41-58.

Irwin, S.H., M.E. Gerlow, and T.R. Liu. "The Forecasting Performance of Livestock Futures Prices: A Comparison to USDA Expert Predictions." Journal of Futures Markets 14(1994):861-75.

Kastens, T.L., T.C. Schroeder, and R. Plain. "Evaluation of Extension and USDA Price and Production Forecasts." Journal of Agricultural and Resource Economics 23(1998):244-61.

Kunze, J. "The Bureau of Agricultural Economics' Outlook Program in the 1920s as Pedagogical Device." Agricultural History 64(1990):252-61.

Lichtenstein, S., and B. Fischhoff. "Training for Calibration." Organizational Behavior and Human Performance 26(1980):149-71.

Lichtenstein, S., B. Fischhoff, and L.D. Phillips. "Calibration of Probabilities: The State of the Art to 1980." In D. Kahneman, P. Slovic, and A. Tversky, eds. Judgment under Uncertainty: Heuristics and Biases. Cambridge: Cambridge University Press, 1982, pp. 306-34.

McNew, K., and J. Espinosa. "The Informational Content of USDA Crop Reports: Impacts on Uncertainty and Expectations in Grain Futures Markets." Journal of Futures Markets 14(1994):475-92.

Prescott, D.M., and T. Stengos. "Bootstrapping Confidence Intervals: An Application to Forecasting the Supply of Pork." American Journal of Agricultural Economics 9(1987):266-73.

Russo, J.E., and P.J. Schoemaker. "Managing Overconfidence." Sloan Management Review 33(1992):7-17.

Sanders, D.R., and M.R. Manfredo. "USDA Production Forecasts for Pork, Beef, and Broilers: An Evaluation." Journal of Agricultural and Resource Economics 27(2002):114-28.

--. "Keep Up the Good Work? An Evaluation of the USDA's Livestock Price Forecasts." Proceedings of the NCR Conference on Applied Commodity Price Analysis, Forecasting, and Market Risk Management. St. Louis, Missouri, April 2003, Available at http://agebb.missouri.edu/ncrext/ncr134/.

Sumner, D.A., and R.A.E. Mueller. "Are Harvest Forecasts News? USDA Announcements and Futures Market Reactions." American Journal of Agricultural Economics 71(1989):127-34.

Taylor, J.W. "Evaluating Volatility and Interval Forecasts." Journal of Forecasting 18(1999):111-28.

Teigen, L.D., and T.M. Bell. "Confidence Intervals for Corn Price and Utilization Forecasts." Agricultural Economic Research 30(1978):23-29.

Timm, T.R. "Proposals for Improvement of the Agricultural Outlook Program in the United States." Journal of Farm Economics 48(1966):1179-84.

Trip, G., R.B.M. Huirne, and J.A. Renkema. "Price-Predicting Ability of Farm Managers: Empirical Findings with Flower Producers in the Netherlands." Review of Agricultural Economics 22(2000):464-76.

United States Department of Agriculture (USDA), Office of the Chief Economist, Agricultural Marketing Service, Economic Research Service, Farm Service Agency, and Foreign Agricultural Service, World Agricultural Supply and Demand Estimates, various issues.

Wallis, K.F. "Chi-Squared Tests of Interval and Density Forecasts, and the Bank of England's Fan Charts." International Journal of Forecasting 19(2003):165-75.

Williams, J.C., and B.D. Wright. Storage and Commodity Markets. Cambridge: Cambridge University Press, 1991.

Vogel, F.A., and G.A. Bange. Understanding USDA Crop Forecasts. Washington DC: US Department of Agriculture, National Agricultural Statistics Service and Office of the Chief Economist, World Agricultural Outlook Board, Miscellaneous Publication No. 1554, 1999.

Yaniv, I., and D.P. Foster. "Graininess of Judgment under Uncertainty: An Accuracy Informativeness Trade-Off." Journal of Experimental Psychology: General 124(1995):424-32.

--. "Precision and Accuracy of Judgmental Estimation." Journal of Behavioral Decision Making 10(1997):21-32.

Olga Isengildina is postdoctorate research associate. Scott H. Irwin is Laurence J. Norton professor of agricultural marketing, and Darrel L. Good is professor, Department of Agricultural and Consumer Economics, University of Illinois at Urbana-Champaign.