A. Rickey on the Bloomberg Report

A. Rickey took the time to write a detailed and thoughtful response to my blog on the Bloomberg report, so I think it's worth reprinting on the main page for those readers who don't look at the comments:

Dear Fitz-Hume:

For Prof. Heller's general standards of truth when it comes to something that might hurt Bush, it might be useful to check here (observing Prof. Heller defending pretty much to the death the Rather memoranda). Whilst I'm sure he'll consider this an ad hominem, simply put, I've yet to see him post an anti-Bush statement he didn't like.

What Prof. Heller just hit you with is called an "argument from authority": who are you to question The Lancet or John's Hopkins? Of course, the fact that both institutions have rushed to publish a work just days before a presidential election might have given some of us pause in proclaiming the accuracy of the data--it did you--but doesn't give Prof. Heller pause at all.

What does give me pause is this:

Many of the Iraqis reportedly killed by US forces could have been combatants. 28 of 61 killings (46%) attributed to US forces involved men age 15–60 years, 28 (46%) were children younger than 15 years, four (7%) were women, and one was an elderly man. It is not clear if the greater number of male deaths was atributable to legitimate targeting of combatants who may have been disproportionately male, of if this was because men are more often in public and more likely to be exposed to danger.

Now, one thing it strikes me about this study, having read it closely, is that the number 100,000 and civilian are never used together. It reports 100,000 "excess" casualties, and then says that civilian casualties are high. But it doesn't use the numbers together. The paper purports to be tracking civilian casualties, but given the paragraph above, it's quite possible that it isn't tracking them well at all.

Also, there's a significant methodological question here that the data given doesn't allow one to evaluate. One would assume--it would be reasonable--that pre- and post-invasion deaths would be clustered differently. (One suspects, for instance, that Fallujah and Baghdad would have a higher post-war mortality, and any highly Kurdish areas would show a lesser increase, if not a decrease, in mortality. It's also worth noting that interviews were conducted in Arabic, which may have prejudiced non-Arabic speaking areas.)

Further, the methodology and the conclusion make some rather startling suggestions. Take this, for instance: "Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths." This is believable for the data collected, but let's examine what that means for their "extrapolated" number. Let's assume that 50% of all "excess" deaths were violent, and that 50% of those were caused by airstrikes. (That's the minimum which must be true for the numbers above to be true.) That's 25,000 dead in air strikes.

But air strikes are going to be highly, highly localized: they're going to be reported mostly in areas where fighting has been intense and of a particular type. Extrapolating from that--especially from a very small number of total deaths reported (142)--seems odd indeed.

Finally, the method of calculation seems a bit curious:

We estimated the death toll associated with the conflict by subtracting preinvasion mortality from post-invasion mortality, and multiplying that rate by the estimated population of Iraq (assumed 24·4 million at the onset of the conflict) and by 17·8 months, the average period between the invasion and the survey.

But given the relatively small sample-size, the possibility that they over or underselected... this seems risky. How would the numbers change, for instance, if instead of extrapolating from total death rates, they extrapolated from each Governorate, giving each its own mortality rate? They don't give us the numbers to tell, although divergent results would suggest the sample was insufficient. (There's a graph on pg. 3 that would allow you to do it, if they actually gave you numbers instead of bar charts.)

Now, perhaps there have been 25,000 deaths from air strikes. Notably, the methodology doesn't allow for how many of these deaths were civilian deaths. (For instance, if only five of the adult male non-Fallujah casualties were actually combatants, the extrapolated death rate shifts magnificently, assuming that the 100,000 number is talking about civilians.) But perhaps the methodology above is somehow more accurate than it seems. Certainly I'd expect the casualty rate to be higher after a war than before it, and I'd especially consider it to be higher in the immediate aftermath. And of course, I'm not a statistician or an epidemiologist: maybe my doubts here are wrong. Still, I rather suspect the small sample size means that the confidence level one can have in that number is fairly low.

Anyway, it's a piece that's come out right before an election, and been rushed to press by an author (and a publication) that seemingly have an axe to grind against the use of air strikes. There's not enough data in it to actually evaluate the methodology. But this satisfies Prof. Heller, Fitz-Hume, and is enough. The burden of proof, you see, rests on anyone who wishes to prove that 100,000 excess deaths haven't occurred, so long as the data would be bad for Bush.

For what it's worth, here's a section of an e-mail I sent to Fitz-Hume regarding a private exchange on the report:

... I appreciate the e-mail, and your recent comment is articulate and well-argued. I can't say for sure that the conclusions in the report are accurate; I'm neither a statistician nor an Iraq expert. I was simply pointing out that the report's conclusions, if correct, are disturbing. Do I believe the report is accurate? More or less. But that's because I tend to trust the source, the Bloomberg School, not because I have any particular knowledge of how it conducted the investigation. So I appreciate the input...

Best,

Kevin

I hope that addresses A. Rickey's belief that I'm simply relying on an "argument from authority." The authoritativeness of a source is certainly relevant to its believability -- especially in an area where most of us have neither training nor expertise -- but that's a presumption, nothing more. It can certainly be rebutted with contrary evidence.

Comments

[Kevin Jon Heller]

For the record, A. Rickey's comment that I've never posted an anti-Bush statement that I didn't like is inaccurate. When Bush commented that we could never completely win the war on terror, I criticized the hysterical responses from Democrats in no uncertain terms.

[A. Rickey]

As you note, "never" is an overstatement. Perhaps I should revise it to something like "Prof. Heller frequently posts anti-Bush statements from rabidly partisan sources, or fantastically anti-Bush figures that demand critical review, without giving them sufficient, or even cursory, evaluation. This hurts his credibility." I figured that was a bit wordy, but as you point out, never is too absolute.

As for the argument from authority that was referenced, let me repeat your argument to poor Fitz-Hume:

I see -- I'm supposed to take your conclusory statement that 100,000 people haven't been killed over the heavily documented conclusions of one of the pre-eminent schools of public health in the world. Thanks, but given the absurdity of your analogy (the Bloomberg report deals with an 18-month period, and the Tokyo firebombing took place in one day), as well as your obvious disinterest in reading the article or the report itself (which expressly excluded deaths in Fallujah, because of the abnormally intense violence there), I'll pass.

But please, feel free to submit your polling method, so I can compare it to the one the Bloomberg team used.

(emphasis mine)

Now, as I point out above, what's there isn't "heavily documented." (Or rather, it probably is, but not publicly documented.) Unless you've got access to more than the eight-page report I read, you have a methodological summary that should cast some immediate doubts in your mind, even if you're a non-statistician. You have a number of deaths from airstrikes (at least 25,000, or roughly 46 people per day, every day, over the study period) that should at least raise an eyebrow. And the lack of the word "civilian" near the number of 100,000 in the report? That is pretty telling, even if you do no math at all. What you don't have is enough numbers to do a reasonable evaluation.

The point is, you wanted to believe, so you printed it. In the past, you've said you're "resigned" to such things, but that's a resignation that I simply don't think is acceptable in someone who is trying to provide commentary and analysis, and wants to remain credible. This isn't a specialist/non-specialist issue: if you really don't feel you can evaluate something, even on a low level, don't comment on it. But none of the above requires more than some critical thinking and some high-school math.

In the meantime, you fell back on the argument from authority. Again, I have to wonder: a report that is highly useful in a partisan fashion is rushed to press days before an election. You didn't even wonder that the "highly respected" establishment you were reprinting might have had some less-than meritorious motives?

[Kevin Jon Heller]

That's an interesting semantic trick -- subtlely implying (again) that the Bloomberg report is biased against Bush, despite having no evidence of that fact other than (I guess) paranoia that any information that could be used to criticize Bush must be motivated by anti-Bush zealotry. In your first post, you note that the author of the report has an "axe to grind" against airstrikes and use that fact -- despite not having any evidence to back it up -- to imply that the report is anti-Bush. It's not, of course; there's no reason to believe Kerry would be any less willing to use airstrikes than Bush. And now you mention that the report could be "useful in a partisan fashion" and use that fact to imply that the report is anti-Bush. That's a logical fallacy -- the fact that something could be used by others to hurt Bush in no way proves that the author of the report intended it to be used that way or harbors anti-Bush sentiment himself. (The same is true of the fact that the Lancet published the report, which also tells us nothing about the author of the report's motives.)

All in all, an excellent job of trying to impeach a non-partisan report -- which may or may not be correct in its conclusions -- by making unsubstantiated assumptions about its motivations. That seems far less intellectually honest than making use of a rebuttable presumption that conclusions reached by a highly-respected, non-partisan academic institution are more likely to be true than, say, the conclusions of a Cato Institute or Center for American Progress.

The Bush administration, which makes a living attacking the messenger in order to divert attention from the message, would be proud.

[A. Rickey]

All in all, an excellent job of trying to impeach a non-partisan report -- which may or may not be correct in its conclusions -- by making unsubstantiated assumptions about its motivations.

Come, now, Prof. Heller. Let's argue about your term "unsubstantiated." First, let's look at the fact that this did not go through the normal peer-review process, but was "expedited." Secondly, let's cut the crap about "non-partisan" institutions. While an institution may not be partisan, it can be biased. Rushing a report out in front of an election does seem to be an attempt at influence.

And wow... two minutes of further research points at further hints that your "impartiality" may be missing. Let's see what the lead author of the report has to say for himself:

"I emailed it in on Sept. 30 under the condition that it came out before the election," Roberts told The Associated Press. "My motive in doing that was not to skew the election. My motive was that if this came out during the campaign, both candidates would be forced to pledge to protect civilian lives in Iraq.

Nope. No bias there. Completely standard process to want to rush peer-review to make sure something appears before an election, I'm sure. And the fact that it will be seized upon by anti-Bush partisans--such as you, Prof. Heller, or am I impugning non-partisan academics here?--should raise no suspicions whatsoever.

But again, we get the argument from authority. You're a law professor, Heller, so let's ask: why is the truth of a statement--that 100,000 "excess" civilian deaths--presumptively true, and its my responsibility to rebut them? Why is it not reasonable to say, "This deviates from a standard procedure, admittedly because of an election? That's an indication of bias?" The fact that the author admits that he was opposed to the war (same article as above) doesn't case suspicions of partisanship? But no. It's a "highly respected, non-partisan academic institution," so it must just be that crazy A. Rickey again.

Prof. Heller, I've put forward an argument for why the statistics don't hold water, to which you're perfectly welcome to reply. Furthermore, I've backed it both with research and quotations from the work itself, and from the press. You, on the other hand, have fallen back upon your normal tactics of accusing your opponents of smears, of maligning something sacrosant--last time it was your "academic integrity", if I recall correctly--and otherwise refusing to answer the question.

You've got the report in front of you. On the first page is the author's email address, to which you could email the author and ask for a full set of statistical data. (You, being a professor, are much more likely to get a response.) If you wish to say that 100,000 "excess" "civilian" casualties have occurred as a result of the war, check the data and see if it gives you the stomach to back it. But stop saying that the data is reliable because the institution has a certain reputation, or that there's some rebuttable presumption that exists for anyone who provides evidence that suits your own political prejudices.

I have never reprinted data on my site that I felt was "too good to be true" without looking into it, simply because I feel it hurts my credibility: inevitably it will be shown to be full of wholes. Stop saying that your failure to do so is somehow a failure of your critics to share your allocation of "rebuttable presumptions." Stop saying that so doing is "attacking the messenger." For god's sake, stop being "resigned."

[A. Rickey]

Ach. That's "full of holes" not "full of wholes."

[A. Rickey]

Oh yes, incidentally:

In your first post, you note that the author of the report has an "axe to grind" against airstrikes and use that fact -- despite not having any evidence to back it up -- to imply that the report is anti-Bush.

The mind boggles. My "evidence to back it up" comes from the Lancet report itself, the very last sentence:

In the interim, civility and enlightened self-interest demand a re-evaluation of the consequences of weaponry now used by coalition forces in populated areas.

Or how about from the first page?

Our results need further verification and should lead to changes to reduce noncombatant deaths from air strikes.

Given that he attributes (at least) 25% of the deaths to airstrikes, and that he doesn't mention much else in the way of weaponry now used in populated areas, I think it's safe to say he's arguing for a reduced role for air strikes. That's pretty clear upon reading the report, so I didn't feel the need to cite one of the reports major conclusions directly. After all, you'd read it as well.

[Fed.No.84]

I'm trying to find out what the argument is.

Is the argument that Kevin Heller's biases interfere with his analysis of political issues. Thus, others are presenting data contradicting the Bloomberg report to prove this proposition?

Or is that argument that although 100,000 dead civilians is bad, if Bloomberg is really wrong, only 25,000 (or some other smaller number) civilians died. While 100,000 dead is bad, 25,000 is not that big of a deal.

I think that Kevin's initial point was, "Look at all these dead people. What do you think of the Iraq war now." I'm not sure that we can make a principled arguement why 25K is acceptable, but 100K is not. IOW, even if Bloomberg is really wrong (re: their number is off by 75%), Kevin's argument remains strong.

[Tung Yin]

Fed. No. 84, I should let Tony speak for himself, but it seems to me there are a couple of points here.

First, I'm in general agreement with Kevin that the credibility of a source is relevant to assessing how much investigation one must do of assertions reported by that source. I'm not familiar with the Lancet to be able to say anything about it, and I'd like to think that a major university like Johns Hopkins should be presumptively credible. At the same time, the Rathergate fiasco has demonstrated that even presumptively credible sources aren't. From herein out, I will not trust any story that is reported by Dan Rather on "60 Minutes," because Rather told us he was as confident as he has ever been on a story about the authenticity of the Killian memo, and well, we know how that turned out. Kevin hasn't said anything either way about Rather's credibility -- and I should quickly note, Kevin is under no obligation to do so -- however, I'd like to think that Kevin too agrees with me about Rather's utter unreliability right now.

(Again, the underlying substance of the story may be true -- Bush probably did get special treatment -- but there are two separate stories there: the political story and the media story.)

Second, the difference between 25,000 and 100,000 may or may not be significant -- though note that if Tony is right, it could be as few as 8,000 (within the 95 percent confidence level). Again, 8,000 is nothing to sneeze at, but the comparative difference between what Saddam plus the sanctions would have killed versus what our airstrikes supposedly have is the key issue. And it seems to me much easier to make the point that Saddam plus the sanctions may well have killed more than 8,000 people during the time that we're measuring.

Needless to say, each of those deaths is tragic, but if we're making policy arguments, we should have some reasonably solid grasp of the numbers we're talking about.

[A. Rickey]

Fed. 84:

So, let's recap. It doesn't matter if one posts highball headline numbers without a cursory review of the data. You're not really worried about that.

And of course, 100,000 is bad, and 25,000 is bad. So let's ask: what number isn't bad? What is the acceptable number of civilian casualties--which are relatively inevitable in any war, though one can do ones best to reduce them--at which Kevin's argument is no longer strong?

Now, if we really wanted to do that analysis, we'd have to make some kind of cost-benefit analysis. And that might very well cover more than 18 months post- (and pre-) invasion, might take into account other factors... whatever. The point is that your argument above either suggests that there is some number of acceptable casualties (below, it seems, 25,000), or no casualties are acceptable. In which case no military action is ever justifiable, which is a position, but you'd better take it without hesitation.

Now, once you have your number established--because, you know, Fed, we should know what we're aiming at here--the next question would be has that number been exceeded. And in order to do that, we'd need a study of more than dubious accuracy, at least if you wanted to make a solid policy argument.

That, of course, would deal with your point. My point is that when it comes to Prof. Heller, if he puts up a big headline number from a "reputable" source, you ought to check it, because he's pretty well "resigned" not to doing so himself. Indeed, he'll presume (charitably, that presumption is rebuttable) that it's true. That's all.

[Fed.No.84]

From herein out, I will not trust any story that is reported by Dan Rather on "60 Minutes".

Are you overstating here, or has that memo really done that much damage to Rather's credibility?

[Fed.]

So, let's recap. It doesn't matter if one posts highball headline numbers without a cursory review of the data. You're not really worried about that.

That's not what I said. I said, even if the numbers are wrong by 75%, then that's still a lot of dead civilians. Hypo: A tells B that C stole $100 from A's wallet and thus C is a thief. It turns out A forgot that he had spent $75 on a date earlier, and thus C only stole $25 from him. Does that undermine A's initial point, viz., that C is a thief?

And of course, 100,000 is bad, and 25,000 is bad. So let's ask: what number isn't bad?

It depends on whether there is a good reason to go to war. I come from the Von Clausewitz school of military thought, best exemplified by General Sherman's March to the Sea. Of course, the Civil War was just, and therefore, even a high number of civilian casulties is not troublesome. However, in an unjust war, one civilian casulty is too many.

[Fed.]

[T]he comparative difference between what Saddam plus the sanctions would have killed versus what our airstrikes supposedly have is the key issue.

I think there's a big difference. Let's say A kills ten innocent people. B kills A, but in the process kills 5 innocent people.

Issue: Has B saved 5 lives, or has B murdered 5 people?

Of course, I'm afraid this gets us back to the old Problem of the Train. If you're riding a train that is going to crash, do you do nothing, thereby letting the 5 passengers die; or do you divert the train to save the passengers, even though that means you'll kill the railroad worker.

[Tung Yin]

Yes, it does get to the problem of the train, and the usual explanations for why it's okay to turn the wheel don't really apply when we're talking about different types of killing. But people accept consequentialist arguments to some extent, which is where the difference between 100,000 and 25,000 or even 8,000 kick in.

Oh, and on "60 Minutes," yes, I'm serious. Rather's initial "defense" of the "reporting" was so ridiculously one-sided and pathetic that I can presume only that he really has lost the ability to see objetively. Is there any reason he should be viewed as having any credibility when reporting on a political story?

[A. Rickey]

With regards to your first hypo:
Hypo: A tells B that C stole $100 from A's wallet and thus C is a thief. It turns out A forgot that he had spent $75 on a date earlier, and thus C only stole $25 from him. Does that undermine A's initial point, viz., that C is a thief?

Unfortunately, the cognate here would be "C has caused civilian deaths through an invasion." But again, with regards to Bush, the idea that there have been some civilian casualties that wouldn't have existed otherwise is spectacularly uncontroversial. I suppose--given your "Clauswitzian" view--that you can claim that causing such civilian deaths is an evil in and of itself.

But lacking that Clauswitzian justification, there is a calculus which has to be applied. Which brings us to your second hypo. What Tung Yin mentions is exactly what the study is trying to measure. The "excess" deaths, after all, are a measure of mortality pre- and post-invasion. Because, you see, the equation there becomes very different if, instead of a 5/5 split, it's 30/5 or 5/30.

In other words, if you're using the "Clauswitz" out, then the report is irrelevant, Fed: no number is too low for you anyway. If like most people, however, you're not being black and white about the issue, the size of the number, and thus the accuracy of the report, matter. Somehow I doubt that Prof. Heller was making your argument when he posted his headline numbers.