Reconsidering the Harvard Medical Practice Study Conclusions about the Validity of Medical...

Over fifteen years after first reporting to the State of New York, the Harvard Medical Practice Study (HMPS) continues to have a significant impact in medical malpractice policy debates. In those debates the HMPS has come to stand for four main propositions. First, "medical injury...accounts for more deaths than all other kinds of accidents combined" and "more than a quarter of those were caused by substandard care."1 second, the vast majority of people who are injured as result of substandard care do not file a claim.2 Third, "a substantial majority of malpractice claims filed are not based on provider carelessness or even iatrogenic injury."3 Fourth, "whether negligence or a medical injury had occurred. ..bore little relation to the outcome of the claims."4

Medical malpractice researchers have long known that the HMPS provides far stronger support for the first two of these propositions than for the last two; the HMPS was not designed or powered to reach strong conclusions about the validity of medical malpractice claims.5 Despite the fact that these latter conclusions were the weakest aspects of the HMPS, many policymakers and even some well-regarded researchers give them the same "gold standard" deference that appropriately is given to the findings regarding the rate of medical malpractice and the rate of medical malpractice claims.

For example, in a 2005 White Paper on medical liability reform, the Joint Commission on Accreditation of Healthcare Organizations cited the HMPS for the conclusion that "only 17 percent of claims appeared to involve negligent injury."6 Similarly, in a March 2005 report on medical liability, the Joint Economic Committee of the U.S. Congress cited the same finding and also referred to the well known HMPS analogy: "it is similar to a situation in which a traffic officer is giving tickets to large numbers of motorists who are not speeding."7 In a December 2004 fact sheet, the American Medical Association cited the HMPS follow-up closed claim study for the proposition that "medical liability payments correlate with disability not negligence."8 In addition, in the 2002 volume of the Journal of Health Economics, two respected economists cited the HMPS for the proposition that "five-sixths of the cases that receive compensation have no evidence of negligence."9 In these and similar circumstances, neither the acknowledged weaknesses of the HMPS findings nor the contrary evidence are disclosed or discussed.

In light of this continuing reliance on the HMPS and the follow up closed claim study, this article reviews the evidence regarding their findings about the validity of medical malpractice claims. Much of the analysis tracks that of earlier researchers. The value-added lies in collecting this analysis in one easily accessible place, in setting out for the first time a detailed critique of the follow-up study of the 47 closed claims that were matched to the HMPS hospital records, and in comparing the results of the follow up study with the results of other research on medical malpractice claims.

The results of this review are as follows: First, the finding that most eligible people do not bring medical malpractice claims is well supported and confirmed by other studies using both similar and very different research methods. second, the finding that most medical malpractice claims are not based on either iatrogenic injury or provider negligence stands on a small and precarious empirical base. Indeed, the HMPS data are as likely to support a very different finding, namely that most malpractice claims are reasonably related to medical management injuries and provider negligence. Finally, the finding from the follow-up closed claim study rests on an even weaker base and is contradicted by a large body of research on closed medical malpractice claims. In fact, the research reviewed in part IIIB of this article suggests that the legal system filters out most of the weaker claims.

I. The HMPS Finding Regarding the Low Rate of Claiming

The HMPS included an impressive effort to determine the extent of medical malpractice and compare it with the extent of medical malpractice claiming. The researchers put 31,000 randomly selected hospital records from 51 hospitals in New York through a twostage review process that identified the rate of negligent medical injuries among that sample. They applied that rate to the total number of hospitalizations in New York to estimate the total number of negligent medical injuries in New York during the research period and then compared that number to the total number of malpractice claims filed in New York during the same period.

The HMPS estimated that there were about 27,000 injuries that resulted from medical negligence in hospitals in New York during 1984 and that only about 3800 claims were filed under malpractice insurance policies covering the year 1984.10 This means that there were more than seven negligent injuries for every medical malpractice claim and, accordingly, that most patients injured as a result of negligent medical management do not make a claim. This research finding is consistent with the comparable findings reached in other large population-based studies using similar record-review research methods,11 as well as ethnographic research in which researchers closely observed hospital practice over an extended period.12

Taken together, these studies demonstrated that medical malpractice is real and that it is widespread. Over 75,000 people die every year from medical malpractice, more than the total number of deaths from automobile and workplace accidents combined.13 These results also demonstrated that most medical malpractice victims do not file claims and that the real medical malpractice litigation problem is "the malpractice system is too inaccessible, rather than too accessible, to the victims of negligent medical treatment."14

II. The HMPS Finding Regarding the Validity of Filed Claims

The HMPS also attempted to evaluate the validity of medical malpractice claims. The researchers collected all of the medical malpractice claims filed in New York during the study period and subsequent years and then matched those claims against the 31,000 hospitalizations that they studied. This match identified a surprisingly small number of claims - only 47 - that arose out of these hospitalizations.15 Out of these 47 claims, HMPS reviewers had identified a negligent medical management injury in only 8 of the matched hospital records.16 This 8 out of 47 number captured the imagination of advocates of tort reform, leading them to assert that most medical malpractice claims have no basis.17

In fact the HMPS does not support that assertion. The HMPS was well designed to produce a conservative, lower bound estimate of the rate negligent medical management injuries in a large population of hospitalizations, but it was not well designed to make judgments about the validity of a small number medical malpractice claims or to draw significant policy conclusions based on those claims. Understanding this point requires more careful attention to the HMPS research design than even most informed observers have given to this matter.

ILA. The HMPS Review Process19

The HMPS review was a two-stage process. In the first stage, medical record analysts reviewed the record of each hospitalization using a detailed screening form that guided them in deciding whether the hospitalization contained one or more clinical indications of a possible iatrogenic injury (i.e. an "injury caused by medical management"). If the record did not contain one of these indications, the hospitalization was classified as not involving a medical management injury, and the record related to the hospitalization was not reviewed further. The first stage review determined that there was an indication of a possible iatrogenic injury in thirty percent of the hospitalizations.19

If the record did contain an indication of a possible iatrogenic injury, the hospitalization advanced to the second stage of the review process. In the second stage, two physicians independently reviewed the record. Each physician first confirmed that the medical record analyst had appropriately interpreted the hospital record. If not, the hospitalization was classified as not involving a medical management injury, and the record was not reviewed further. If the medical record analyst had appropriately interpreted the record, each physician reviewer then assigned the hospitalization an "adverse event" score from 1 to 6 indicating the strength of the evidence of a medical management injury.20 If that score was 2 or more, the physician answered a series of questions regarding the nature of the injury and then completed a section of the review form captioned "Is there evidence for negligence?"

The first step in the negligence section directed the reviewer to answer "yes" or "no" to the following question: "Was this AE possibly due to a reasonably avoidable error, or carelessness by either an individual or medical care system, or both?" If the answer was "no," the review was over, and the hospitalization was classified as not involving negligence. Importantly, a "no" answer by either reviewer was sufficient to classify the hospitalization as one that did not involve negligence, even if the other reviewer was very confident that there was negligence.

If the answer to the first negligence question was "yes," the physician then answered a series of specific questions that ended by asking the reviewer to re-assess the opinion that the injury was possibly due to negligence.21 If the physician did change his or her opinion, the review was over, and the hospitalization was classified as not involving negligence. If the physician did not change his or her opinion, the physician then rated the degree of confidence in the evidence for negligence on a one to six scale that was comparable to the confidence scale for the medical management injury determination.22

A supervising physician then examined the forms filled out by the two physicians, verified that the forms related to the same hospitalization, and computed the averages of the reviewers' confidence scores. The hospitalization was classified as involving a medical management injury only if the average of the confidence scores for that determination was greater than 3.5, and it was classified as involving negligence only if the average of the confidence scores for that determination was greater than 3.5. (The practical effect of the 3.5 cutoff is that a decision by either physician reviewer that there was not an medical management injury is sufficient to rule out a medical management injury, even if the other reviewer was highly confident that a medical management injury had occurred, and likewise with the negligence classification.) The second stage determined that there was strong evidence of a medical management injury in just under four percent of hospitalizations and strong evidence that the injury resulted from negligence in one percent of hospitalizations.23

The process just described represents a very conservative approach to identifying negligent medical management injuries. This approach was well suited to the task of providing a solid, lower bound estimate of the rate of those injuries, an estimate that would be trusted by medical providers. Physicians designed and supervised the study. Physicians trained the medical record analysts and conducted the second stage review. The scoring process meant that, in effect, either physician reviewer could veto the medical management injury or negligence finding of the other. And the review process contained a variety of safeguards against a mistaken conclusion that there was a medical management injury or negligence, and essentially no safeguards against a mistaken conclusion that there was not.24 This means that, if anything, the HMPS estimates about the rate of medical management injuries and medical malpractice are low (as the HMPS researchers were aware).25 As a result, the HMPS, together with an earlier California study and a later Colorado/Utah study using similar research methods,26 provide persuasive evidence that the real medical malpractice problem is medical malpractice, not medical malpractice litigation.

ILB. HMPS and the Merits of Individual Medical Malpractice Claims

The HMPS was not well suited to the task of evaluating the merits of a small sample of medical malpractice claims. There are three main reasons for this: the lack of controls to avoid false negatives discussed in the preceding section, the problem of reviewer reliability, and the inherent limits of relying exclusively on hospital records.

II.B.I. The Reliability Problem

The reviewer reliability problem refers to the fact that repeated reviews of the same records showed that physician reviewers did not always agree about which records contained good evidence of a medical management injury, and they agreed less often about which records contained good evidence of negligence.-7 Because the team reviewed so many hospital records, the reliability of judgments in individual cases does not undercut their finding about the rate of errors in hospitalizations, but it does undercut their finding that most medical malpractice claims lack merit.

HMPS researchers were cognizant of the limits of making negligence judgments based on hospital record reviews. The researchers conducted a pilot study that measured the degree of agreement between independent reviewers examining the same records.28 The pilot study found a high level of agreement among reviewers regarding the presence or absence of a medical management injury, but a lower rate of agreement among reviewers regarding the existence of substandard care.29 This lower rate of agreement reflects the difficulty of determining whether substandard care was given, as well as differences of opinion regarding what constitutes substandard care. Like other researchers, the HMPS researchers decided that the lower rate of agreement with regard to substandard care was acceptable for drawing conclusions about the rate of substandard care in a population because later independent reviews consistently found the same rate of substandard care, even though the reviewers did not always agree about which specific cases fell into this category.30

The HMPS researchers also conducted a similar test of reviewer reliability at the beginning of the New York study.31 The New York test illustrates why the HMPS record review process is not a very reliable way to form strong conclusions about the validity of individual medical malpractice claims.

In the New York test, the HMPS research team took all the records that passed through the medical record analyst screen at two hospitals and put them through the physician review and classification process twice, using two different reviewers each time.32 (None of the four physicians that reviewed each record knew that the record was receiving any more attention than any other record.) The two teams examined a total of 318 records. Out of that group, the two sets of physician reviewers identified the same medical management injuries 35 times and different medical management injuries 34 times. In percentage terms, this means that the review teams agreed that there was not a medical management injury in 78 percent of the cases, but they disagreed as often as they agreed about whether there was a medical management injury in the remaining 22 percent of the cases. Moreover, the two sets of reviewers disagreed far more often than they agreed about which of the medical management injuries resulted from negligence. They identified the same negligent injuries only four times and different negligent injuries 21 times.33

In summary, the New York reliability test showed that, although the two teams largely agreed about which records did not contain evidence of a medical management injury, they agreed less often about which cases did have a medical management injury. And when it came to the question of which medical management injuries resulted from negligence, they disagreed five times more often than they agreed. This is an especially high rate of disagreement given that the reviewers only considered whether there was evidence of negligence or not if they first concluded that there was evidence of a medical management injury.

Considering the particular 47 cases the HMPS identified through the medical malpractice match increases concerns about the reliability of the classifications of those cases. As discussed below, many of those cases either were unusually hard for the medical record analysts to evaluate or involved disagreements among the physician reviewers. This means that there is good reason to believe that these particular 47 cases would produce an even higher rate of disagreement among any given set of reviewers than the "average" case put through the duplicate review in the New York test.

The unreliability of individual decisions would not be as troubling if the HMPS had studied a very large number of medical malpractice claims. But when the number is so small, changing the results of only a few cases could have a very substantial change in the rate of cases with findings of negligence, as the HMPS team explained in their report to the State of the New York. Indeed, they identified a few plausible differences in the classifications that might have tripled the percentage of the 47 cases with good evidence of negligence - from 17% to 50 %.34 As a result, it is clearly a mistake to draw strong conclusions about the validity of medical malpractice claims generally on the basis of the HMPS.

II.B.2. The Limits of Hospital Record-Based Research

The third reason that the HPMS was not well suited to the task of evaluating the merits of individual malpractice claims relates to the fact that the HPMS only examined hospital records, not the full range of evidence available in a medical malpractice case. Hospital records provide at best only a partial picture of what occurs in a hospitalization. Sometimes adverse events are deliberately omitted from hospital records; other times they are omitted inadvertently.35

For example, one important ethnographic study concluded that adverse events occur far more often than they are documented in hospital records.36 The ethnographic research team closely observed hospital care in three surgical units in a large tertiary care hospital affiliated to a university medical school and recorded all "adverse events" discussed by health care providers at any regularly scheduled daytime meeting.37 For purposes of this research, "adverse events" were defined as "situations in which an inappropriate decision was made when, at the time, an appropriate alternative could have been chosen." The researchers found that there was an adverse event in nearly half of the hospitalizations (not all of which resulted in injury) and that the adverse event was "serious" in eighteen percent of the hospitalizations. (A "serious" event produced at least a temporary physical disability). These results are consistent with other observation-based hospital research.38 The researchers noted that this is a much higher rate than that found in the HMPS, but they concluded that even this higher rate is an underestimate because they "only studied discussions of adverse events at regularly scheduled daytime meetings."39

Like the HMPS researchers, the ethnographic research team matched the studied hospitalizations with malpractice claims filed in the jurisdiction. With access to a more complete range of evidence, it is not surprising that this research team found that a much higher percentage of the matched malpractice claims arose out of hospitalizations in which there had been an "inappropriate decision" noted in their research records (85%). Like the HMPS, their sample of medical malpractice claims is very small (13). Nevertheless, the classifications of the ethnographic researchers gave to individual hospitalizations are more reliable than that of the HMPS reviewers for two reasons. First, the ethnographic researchers considered a broader range of evidence. second, the ethnographic researchers did not make independent evaluations of the presence or absence of an "inappropriate decision"; they only noted whether one of the doctors or hospital staff stated that there had been one. By contrast, the HMPS researchers made an independent evaluation of the hospital record, introducing the possibility of bias.

These are not criticism of the HMPS research methods. All research involves tradeoffs. The HMPS researchers chose a large sample and thin data in order to identify a robust, credible lower bound estimate of the rate of preventable errors in hospitals. Given that choice, however, it is a mistake to conclude that the HMPS could determine that the care was appropriate in a particular case. The most that the HMPS researchers could determine was whether there was evidence of a medical management injury and negligence in the hospital record.

In their early articles, the HMPS researchers acknowledged that they could not draw reliable conclusions about the validity of individual claims. Their NEJM article states:

Our study was not designed to evaluate the merits of individual claims. ...[W]ithout access to the full [liability] insurance records, we cannot assess the prospects of individual cases.

...Our reviewers sometimes disagreed about causation and negligence; when only one found negligence, the case did not qualify as an adverse event due to negligence (except in the rare case when there was only a single reviewer). In a lawsuit a single expert opinion might be sufficient to support a finding of negligence; under our protocol it would not....Thus our findings are not directly comparable to the results of civil litigation.40

Unfortunately, this note of caution was less prominent in the books published for general audiences,41 and it disappeared almost entirely in the report of the follow up study of the medical malpractice claims.42

II.B.3. An Australian Comparison

One final piece of evidence regarding the inappropriateness of using the HMPS to draw strong conclusions about the validity of medical malpractice claiming comes from research in Australia that was modeled on the HMPS.43 Australians were dismayed when the results of the Quality in Australian Health Care Study indicated that nearly 17 percent of hospital admissions involved an "adverse event" that many researchers initially determined to be equivalent to a finding of a medical management injury in the HMPS.44 After closely comparing their research methods with those used in a recent Utah/Colorado study modeled on the HMPS, however, the researchers found several previously unnoticed, small methodological differences. Applying the U.S. approach to the Australian data resulted in a reduction of hospitalizations involving medical management injuries to about 11 percent.43 Similarly, the researchers concluded that applying the small differences in the Australian approach to the US data would have resulted in an astounding 70 percent increase in the number of hospitalizations determined to have resulted in medical management injuries.41'

This does not mean that the Australians were "right" and the HMPS was "wrong." Instead, it simply indicates the sensitivity of hospital record reviews to small changes in the research method and, therefore, that it is inappropriate to use the HMPS to make strong conclusions about the absence of a medical management injury in a particular case. Once again, this is not a criticism of the HMPS research methods, simply a limit on what the HMPS results mean.

U.C. Reconsidering the HMPS Finding Regarding the Validity of Filed Claims

As Professor Michael Saks has explained, the HMPS research actually suggests that most of the claims did have some basis in fact.47 First, there was evidence of a possible medical management injury in all but twelve of the forty-seven hospitalizations. Moreover, the claims relating to five of the twelve hospitalizations that were screened out by the medical record analysts alleged a negligent failure to diagnose a condition in an earlier outpatient visit, making the claims unusually difficult for the analysts to evaluate.48 For the thirty-five hospitalizations that advanced to the second stage review, at least one physician concluded that there was a medical management injury in all but ten. For the twenty-one cases for which negligence determinations were made,49 the reviewing physicians voted unanimously that there was not negligence in only six, they agreed that there was negligence in eight, and they reached a split decisions in seven.

Putting all this together, the hospital records, alone, provided some factual support for three quarters of the claims (35 out of 47 claims). Further, at least one reviewer found evidence of substandard care in three quarters of the claims for which that determination was made (15 out of 21 claims), and both reviewers found evidence of substandard care in over one third of those claims (8 out of 21 claims).50

Indeed, given that claimants and their lawyers generally must file suit in order to obtain all the information that they need to determine whether the claimants' injuries resulted from negligence,51 the percentage of claims with strong evidence of negligence in the hospital record is higher than would be expected in world in which medical providers typically do not disclose that an injury is iatrogenic, let alone the result of negligence.52 For this reason, the analogy to a situation in which "police regularly gave out more tickets to drivers who go through green lights than to those who go through red lights"53 is flawed. A traffic officer has access to all the information about the color of the traffic light at the time the driver went through. The HMPS reviewers had access only to the hospital record - an important part of the information about patients' treatment, but not the whole. In addition, a traffic officer is for all practical purposes the controlling legal authority able to impose a legally binding judgment regarding a traffic offense. An injured patient is just a complaining witness, and a medical malpractice complaint only begins a legal process; it is not a legally binding judgment.

Another way of evaluating what the HPMS tells us about the validity of medical malpractice claims is to examine how the strength of the evidence of malpractice in the hospital record affected the likelihood that a medical malpractice claim would be filed. Evidence of a possible medical management injury sufficient to satisfy the first screening criterion raised the likelihood of a claim eight times. Evidence that was sufficient to support a unanimous finding of a medical management injury by the physician reviewers raised the likelihood of a claim an additional six times. Evidence that was sufficient to support a unanimous finding that the medical management injury resulted from negligence raised the likelihood of a claim an additional two and a half times. Put another way, a patient whose hospital record provided the strongest evidence of medical malpractice was twenty two times more likely to make a claim than the average patient, eight times more likely to make a claim than all other patients with sufficiently strong evidence of a medical management injury to reach the second stage of the HMPS review, and two and a half times more likely to make a claim than all other patients the HMPS physicians determined to have suffered a medical management injury.54 This is hardly a picture of malpractice claiming run amok.55

III. The Follow Up Study's Conclusion about the Validity of Paid Claims

In 1996, some HMPS researchers reported the results of a ten-year follow up study of the medical malpractice claims that arose out of the hospitalizations reviewed in the HMPS.56 The researchers concluded that "[t]he results of these follow up assessments of whether negligence or a medical injury had occurred 10 years earlier bore little relation to the outcome of the claims, just as in our earlier study they were found to bear little relation to the initial decision to file the claims."57

For the reasons just explored, this conclusion regarding the results of the earlier study is questionable. The earlier study did not find that the assessments bore "little relation to the initial decision to file the claims." In fact the strength of the evidence of negligent medical injury in the hospital record was related to the likelihood that a malpractice claim would be filed. The similar conclusion regarding the results of the follow up study is questionable as well.

The methods for conducting the follow up study were as follows.55 Starting in 1991 a member of the HMPS team contacted each of the medical malpractice insurers handling one or more of the claims every six months and inquired into the status of the claims. Once a claim was closed, the researchers obtained a claim summary from the insurer and used that summary to identify any "discrepant" cases - i.e. claims that were paid but that the HMPS had classified as not involving a negligent injury, or claims that were not paid but that the HMPS had classified as involving a negligent injury. At the time the follow up study was concluded, forty-six of the fifty-one claims that had been matched to the HMPS hospitalizations had been closed.59 A researcher who was unaware of how the HMPS had classified the hospitalizations in question reviewed the litigation file of twelve of the thirteen discrepant cases to determine whether there was a medical management injury and, if so, whether the injury resulted from negligence.

III. A. Problems with the Follow Up Study Research Methods and Analysis

The follow-up study shares the small sample problems of the early HMPS analysis of the 47 claims, plus some new problems. The first significant new problem is the "observer effect" introduced by the fact that the research team repeatedly contacted the medical malpractice insurers about these cases. By 1991 the HMPS study was well known in the medical malpractice insurance world. Executives in medical malpractice insurance companies can be expected to have had intense interest in whatever judgments the HMPS researchers would make about the results of those cases.60 That interest may well have affected how the insurance companies handled those cases. It is impossible to know exactly what this observer effect was, but it is not difficult to speculate about ways that it could have biased the handling so as to support the insurance industry's perceived interest in proving that the tort litigation process is erratic.61 This is a potentially serious flaw in the research design.

A second significant new problem with the follow up study concerns the analysis used to support the conclusion that the claims were paid without regard to negligence. The researchers used two statistical analyses. Both raise concerns.

In the first analysis, the researchers compared the mean (average) payment amounts in cases according to the presence or absence of a medical management injury and negligence. They reported that the mean payment in non-negligent medical management injury cases was significantly larger than the mean payment in negligent medical management injury cases ($98,192 as compared to $66,944), supporting their conclusion that negligence did not matter to the outcome of the claim.62

The problem with this analysis follows from the weakness of using means to analyze a very small number of cases. A single "outlier" can bias the results, which is exactly what happened here. Careful examination of the follow up study data shows that the difference in the means is attributable to one very large payment in one close case that the HMPS classified as a non-negligent medical management injury case.63 Reclassifying that case (and two other cases for which the researchers noted that there was reason to believe that the HMPS classification may have been incorrect), results in a dramatic reduction of the mean payment in non-negligent medical management injury cases (from $98,192 to $31,375) and a correspondingly dramatic increase in the mean payment in negligent medical management injury cases (from $66,944 to $162,750).64 With this reclassification, the means for the three categories of cases are as follows:

Significantly, a substantial percentage of the payments in each of the first two categories is attributable to a single large payment within that category, suggesting that even these adjusted means are biased upwards.65

More strikingly, even without this reclassification the median payment in the first two categories is $0, because most of the claims in those categories were closed without any payment.66 Among paid claims in the three categories, the median payments are as follows:

These comparisons suggest that - contrary to the conclusions stated in the published account of the follow up closed claim study - the medical malpractice claiming system is appropriately distinguishing among cases according to their merits.

In the second statistical analysis, the researchers used a logistic regression approach that compared how different factors predicted whether a claim would be paid.68 The factors included the severity of the injury, the HMPS classifications regarding medical management injury and negligence, and income, race and age. The researchers reported that the only factor that affected the payment of a claim significantly was the presence or absence of a disabling injury and, thus, concluded that negligence did not matter to the outcome of the claims. This is the finding that is repeated in the medical malpractice debates and the economic literature.69

There are two main problems with this logistic regression analysis and, accordingly, with this finding. The first problem follows from the fact that the HMPS team only assigned a disability score to hospitalizations determined to involve a medical management injury.70 As a result, their finding about the predictive effect of disability means something other than it first seems. What the follow-up study really found was that a plaintiff with a permanently disabling medical management injury was more likely to be paid than a plaintiff without a permanently disabling medical management injury. While that still leaves out the question of negligence, it is a far less troubling finding than the HMPS researchers suggested, especially in light of what the pilot study and the New York test tell us about the high level of disagreement between their reviewers about the presence or absence of negligence in medical management injury cases. (Recall that the two teams in the New York test disagreed five times more often than they agreed about which medical management injury cases contained good evidence of negligence.)

The second, more serious problem with the logistic regression analysis relates to the treatment of cases in which the only "payment" was a "write off" of the unpaid hospital bill (meaning that the plaintiff did not actually receive any money). On the one hand, the researchers regarded these cases as so insignificant that they did not need to review the litigation file to determine whether the HMPS classifications assigned to those cases were accurate.71 On the other hand, the logistic regression treated these cases the same as those in which the plaintiff received a large payment. This is because a logistic regression analysis evaluates how factors predict the outcome to a "yes/no" question. In the logistic regression here, the "yes/no" question was whether there was a settlement in favor of the plaintiff, and the regression treated a "write-off" as a settlement in favor of the plaintiff.

Write off cases almost certainly were among the least severe or the least negligent,72 so including these cases with the plaintiff wins reduces the estimated effect of the merits in the logistic regression. In other words, treating the write off cases as plaintiff wins made it much more likely that negligence would not be a significant factor in the prediction of the outcome.

Especially in light of their initial determination that they did not need to review the claim files in the write off cases, the researchers should have treated those cases as settlements in favor of the defendant (which is how a plaintiff and his or her lawyer would have understood a case in which they did receive any money) or at the very least the researchers should have run the logistic regression without those cases. Nearly half of the cases that the researchers classified as payments to undeserving claimants appear to have been "write off" cases (7 out of 16).73 In a sample of only 46 cases, the effect of changing the classification of 7 cases, or of eliminating 7 similarly classified cases, could have a very large effect on the logistic regression results. Thus, like the analysis based on the means, the logistic regression analysis falls apart when small, and very reasonable adjustments are made.

A third problem with the follow-up study is the fact that the researchers did not address either the criticisms Professor Michael Saks lodged in his review of the earlier HMPS analysis of the validity of medical practice claims or the statements in the HMPS report to the State of New York regarding the unreliability of the assessments among some of the matched cases. As Professor Saks had highlighted, many of the physician reviews of the hospitalizations that lead to the malpractice claims were not unanimous.74 In addition, the HMPS report disclosed that 5 of the claims related to alleged negligence in earlier outpatient visits and that those hospitalizations had not been reviewed by physicians (because they were screened out during the first stage of the reviewing process).75

As discussed earlier, there is substantial reason to doubt the HMPS classification of a large percentage of the hospitalizations that led to the malpractice claims. Yet the follow-up study treats those classifications as if they were certain. Moreover, the article publishing the results of the follow up study does not disclose which of the cases involved these more doubtful classifications or whether they were among the 12 cases for which the researchers reviewed the claims file. As just noted, changing the classification of only a few cases in such a small sample could have a very substantial effect on any and all of the follow up study results.

III. B. The Contrary Findings of a Large Body of Research

It would not be necessary to be as sharply and publicly critical of the HMPS follow-up closed claim study if its results were consistent with the results of other studies that examined larger numbers of medical malpractice claims under conditions in which there were no observer effects. But the results of the follow-up study are diametrically opposed to the results of other studies, which find that the medical malpractice claim handling and litigation system is appropriately filtering out most non-meritorious cases.

All of the other academic research on the outcomes of medical malpractice claims involved the analysis of malpractice claims that were already closed; most of that research analyzed a substantially larger number of claims. This is a superior research method for two main reasons. First, because the claims were already closed when the research began there could have been no observer effects. Second, analyzing a larger number of claims provides more confidence in the reliability of the results, especially because the sampled population (closed claims) is the appropriate body of cases for studying the claim closing process.

The paragraphs that follow briefly describe the other studies and their results. The studies are presented in chronological order.

Si. Paul Obstetrics Claims Study

Ogburn et al reviewed the claims files of the St. Paul Company and identified all birth injury claims filed from 1980 to 1982 that alleged malpractice on the part of an obstetrician.76 They found 600 such claims. Of those 600, 380 were closed with no indemnity payment and a defense payment of less than $1000. Although the study is not clear on this point, it is likely that these 380 "claims" involved situations in which a physician reported a potential claim but no suit was ever filed.77 From the 220 cases with either an indemnity payment or a defense payment greater than $1000, the researchers identified 153 claims that alleged serious permanent injury or death to the baby (the remaining 67 cases apparently involved injury to the mother or less serious injury to the baby). Five obstetricians reviewed those 153 claims in detail and determined that payments to the plaintiff were made in 90% of the cases in which the physician was negligent. They also reported that there was either an indemnity payment or a defense cost greater than $100O in 55% of the cases in which the physician was not negligent. Because the researchers did not distinguish between defense and indemnity expenditures in the non-negligent cases, the study does not permit any conclusions regarding indemnity payments in non-negligence cases, other than the general sense that indemnity payments are much more unusual in non-negligence cases.78 Nevertheless, the study does strongly support the conclusion that negligence strongly predicts an indemnity payment in medical malpractice cases.

Physicians Mutual Obstetrics Claims Study

Rosenblatt and Hurst reviewed the claims files of an unspecified physician-sponsored medical malpractice insurer and identified all obstetric claims closed during the 1982 to 1988 period.79 They found 54 such claims. In contrast to Ogburn et al, they reported defense and indemnity payments separately and, thus, provide a better basis for evaluating indemnity payments in nonnegligence cases. They found that 21 of the 54 claims were closed before suit with no payment. Rosenblatt reviewed the claims files for the 33 cases put into suit and determined that no indemnity payments were made in any of the non-meritorious cases and that "for those cases in which payments were made, there was general consensus among insurance company staff, medical experts, defense attorneys and the physician defendants that some lapse in standard of care contributed to the observed outcome."80 Because the number of claims reviewed was even smaller than in the Harvard study, the Rosenblatt and Hurst study does not provide strong evidence for a general pattern. Nevertheless, the results do support the conclusions reached by other researchers.

Anesthesiologist Closed Claim Review

Cheney et al reported interim results of the American Society of Anesthesiologist's closed claim review.91 The ASA reviewed the claims files of 17 insurers operating in 40 states and identified 1175 claims against anesthesiologists that were closed before January 1988. Anesthesiologists reviewed each claim and evaluated whether there was "inappropriate care" (i.e. below "the standard of care for a prudent anesthesiologist practicing anywhere in the United States at the time of the event"). In addition, the reviewers recorded information regarding whether a lawsuit was filed, the amount of any award or settlement, the nature of the injury, and various other factors that were relevant to the risk management goals of the study. The reviewers were able to make a judgment regarding the appropriateness of the care in 869 of the cases. In those 869 cases, payments were made in 82% of the instances in which the reviewer determined that the care was inappropriate and in 42% of the instances in which the reviewed determined that the care was appropriate.82 Among the cases in which payments were made, the amount of the payment was strongly correlated to the presence of negligence and to the type of injury. Among paid cases, the median payment in negligent, disabling injury cases was $463,000 while the median payment in non-negligent disabling injury cases was $93,000, and the median payment in negligent death cases was $200,000 while the median payment in non-negligent death cases was $115,000. Among all cases, the median "payment" in non-negligent cases was $0.83 This study supports the conclusion that (a) the presence or absence of negligence most strongly explains both the likelihood of payment and the amount of payment and (b) the severity of the injury affects the size of a payment given the presence or absence of negligence.

Economists'Single Hospital Study

Farber and White analyzed all malpractice claims relating to care provided in a single hospital initiated in 1977 or later and resolved by 1989 and that involved claims solely against the hospital or a member of its staff. There were a total of 252 such cases.84 For each of these cases the hospital obtained expert reports on the quality of care. Quality of care was "good" if the professional standard of care was met, "bad" if it was not, and "ambiguous" if the experts disagreed or could not make an evaluation. Farber and White found that there was a strong relationship between the outcome of the case and the quality of care. Claimants were paid in 89% of the "bad" care cases, 25% of the "good" care cases, and 69% of the "ambiguous" cases. Based on a regression analysis, Farber and White determined that the hospital's expected liability for damage was 25 times as high on average in a "bad" care case as in a "good" care case.

Florida Closed Claims I

Sloan and Hsieh used the Florida Department of Insurance database of all medical malpractice claims closed in Florida from October 1985 to March 1988.85 After eliminating the "claims" that never resulted in an insurer expense, they were left with a database of 6,612 cases. The database contained information about the defense and indemnity payments, the nature of the allegations, and the nature of the injury. They obtained physician liability reviews for a subsample of claims filed against obstetrician-gynecologists, general surgeons and orthopedic surgeons and added that information to the database.86 Their analysis of that subsample indicated that "cases involving a higher appearance of fault were more likely to be paid" and that the amount of the payment was positively correlated with the severity of the injury.87

Florida Closed Claims II

Sloan et al conducted a more intensive analysis of a randomly selected sample of Florida closed claims filed against obstetricians and emergency room physicians.88 Four panels of physicians reviewed each of 127 birth injury cases, using the Department of Insurance closed claim forms, information from interviews of the claimants, and an abstract of the hospital record.89 A majority of the reviewers determined that negligent care caused an injury in 22 of the cases and that the injury was not caused by negligence in 28 of the cases.90 Of the remaining 77 cases, the opinions of the reviewers were split in 42 cases and were uncertain in 35 cases due to the availability and quality of the medical records.91 Four panels of physicians also reviewed each of 60 emergency room cases using similar information. A majority of the reviewers determined that there was negligence in 32, no negligence in 18, and insufficient information in 10.92 When the researchers compared these liability assessments with the outcomes of the claims, they found a close correlation between the liability assessment and the outcome. "No negligence" claims were most likely to have been voluntarily dropped by the plaintiff with no payment, "negligence" claims were most likely to have been settled with a payment to the plaintiff, and claims with split or uncertain determinations were more mixed.93

New Jersey Closed Claim Study

Taragin et al analyzed 8231 claims closed between 1977 and 1992 by the largest medical malpractice insurer in New Jersey.94 Based on a review process conducted immediately after a claim was filed, the malpractice insurer had rated each of the claims as "defensible," "indefensible," or "unclear," according to the prevailing standard of care. The review process involved an interview with the defendant physician, an evaluation by a claim representative employed by the insurer, and peer review for any case that the claims representative did not conclude was clearly defensible. The researchers compared those evaluations with the outcomes in the closed claims and found that there was a strong correlation. Claimants received payments in 91% of the indefensible cases, 21% of the defensible cases, and 59% of the unclear cases.93 The severity of the injury did not affect the likelihood of payment (though it did affect the size of the payment in cases in which payments were made, as is appropriate). The researchers noted that because the defensible/indefensible determination was made very early in the claim process and because the peer review process was likely to be biased against determining that the physician was liable, the "defensible" cases that resulted in payments may have been incorrectly classified. In sum the researchers concluded, "Further efforts to clarify the frequency of unjustified payments are needed, but our data suggest that such payments are uncommon."96

Michigan Single Hospital Study

Spurr and Howze analyzed 165 closed claims from a single hospital in Michigan constituting all of the claims brought against the hospital or its staff and closed between 1987 and 1995.9" They reviewed the entire claim file for each of the claims and assigned a "fault" score on a continuous scale from O (beyond reproach) to 1 (conclusive evidence of negligence) based on "the opinion of the hospital's risk management staff of the quality of the defendant's medical care, as disclosed by internal memoranda, reports and correspondence."98 They used a Probit analysis to estimate the relative effect of fault and a variety of other factors on (a) whether the case was settled or dropped by the plaintiff and (b) size of a payment, if any." The other factors included the severity of the injury, age of the claimant, whether the claimant had private insurance or Medicaid, and the population density of the county of the court or arbitration in which the claim was filed. In their analysis of the settlement vs. drop decision, only the fault factor was significant, meaning "[t]here is no evidence of a tradeoff between potential damages and the plaintiff's ability to prove negligence, in the determination of the issue between drop and settlement."100 In their analysis of the settlement payment amount, both fault and severity were significant, but fault was more powerful (consistent with the results in anesthesiologists' closed claim study).

North Carolina Study

Peeples et al analyzed 87 closed claims files from malpractice lawsuits filed in North Carolina between 1991 and 1995 that were directed to mediation prior to trial and involved either a "major teaching hospital" or a physician insured by a "principal liability insurer of physicians" that had agreed to cooperate in the research.101 The researchers believe that this sampling method resulted in an unusually high percentage of "durable" cases and, thus, would not be reflective of cases dropped early in the litigation process. The researchers reviewed the claim file and coded the cases using several indications of fault: the initial and last assessment by the insurer of liability (probable, uncertain or likely), the consensus on liability of outside expert reviewers engaged by the insurer (probable, uncertain or likely), whether the insurer believed the standard of care had been breached (yes, no or uncertain), and whether the insurer believed that causation existed between the defendant's conduct and the alleged injuries (yes, no or uncertain). They also coded the cases for outcome variables: defense made offer (yes or no), settled (yes or no), trial (yes or no), who won (plaintiff or defendant), money paid to plaintiff (yes or no). This study contains a variety of interesting findings. For example, the insurer evaluated the liability as probable or uncertain in 86% of the cases handled by experienced plaintiffs' lawyers, but only 42% of the cases handled by less experienced lawyers. With regard to the effect of fault on the outcome, the results are consistent with the other research. The insurer made an offer in 96% of the cases in which it concluded that the standard of care was breached, and money was paid to the plaintiff in 93% of those cases.102 Money was paid to plaintiffs in only 15% of the cases in which the insurer concluded that the standard of care was not breached and in 37% of the cases in which the insurer was uncertain.103

Summary

The picture presented by these other closed claim studies is very different than that presented in the report of the HMPS follow up closed claim study. Negligence matters a great deal to the outcome of a medical malpractice claim, and the litigation process weeds out most of the weaker claims.

IV. Conclusion

In taking an epidemiological approach to the incidence of medical malpractice, the Harvard Medical Practice Study performed an enormous service. The HMPS team deserves much of the credit for the fact that medical malpractice has come to be understood as a public health problem.104 Unfortunately, the HMPS law reform legacy is decidedly more mixed. Policymakers have ignored the HMPS researchers' call for abroad nofault approach to compensation for medical injuries. Instead, policymakers have seized on the weakest aspect of the HMPS, the analysis of the validity of medical malpractice claims, and used that analysis to justify imposing caps on damages in medical malpractice cases and additional procedural hurdles for medical malpractice claimants. For that reason the practical impact of the HMPS may well have been to expand the gap between the large number of people who are injured by medical malpractice and the few people who are compensated and to increase the likelihood that the compensation that is received will be inadequate.

This does not have to be the legacy of the HMPS. Advocates have pushed the HMPS medical malpractice claim data well beyond their limits. A good part of what is reported here can be found in caveats and qualifications contained in the HMPS report to the State of New York and in publications by HMPS researchers, as well as by early critics of the malpractice claiming aspects of the HMPS. But those caveats and qualifications have not received adequate attention in the public debate. What the public heard was that medical malpractice claims are frivolous and that the wrong people get paid. The HMPS and the follow up study prove no such thing and the rest of the literature suggests exactly the opposite conclusion.

Acknowledgements

Thank you to Bernard Black, Michelle Mello, Peter Siegelman, Charles Silver and an anonymous reviewer for helpful comments on an earlier draft and to Yan Hong and Kathleen Sorentino for excellent research assistance.