Mr. Pierce,

The question of Iraqi deaths, and the findings of the Lancet study in particular, obviously remain hugely controversial.  The reason the controversy continues is precisely because the study has not been discredited in the way you allege.  As the New England Journal of Medicine concluded in a balanced review in 2008, there is so far no conclusive evidence that the Lancet study does or does not overestimate mortality.  Scholarly opinion remains sharply divided.

But my purpose was not to defend the Lancet survey as over and against other studies.  What I in fact wrote was that there are “several possible sources of significant bias in the Lancet study, which suggest that its figures should be treated with extreme caution.”  I then highlighted one of these sources, by no means suggesting that it was the only criticism that might be made.  Nevertheless, I argued, the three most important studies to date—while all limited and potentially flawed in different ways—provide converging support for “excess mortality” in the range of 400,000 lives if not higher.  I include the IBC’s tally in this statement for the following reasons (some of which were included in the article I originally sent to Commonweal but were omitted by the editors for considerations of space).

As I pointed out in my article, according to IBC co-founder John Sloboda, “Our best estimate is that we’ve got about half the deaths that are out there.”  The IBC now reports approximately 122,000 reliably documented violent civilian deaths.  If we accept Sloboda’s statement, this gives us a conservative estimate of 244,000 actual violent Iraqi civilian deaths to date

But why, we must ask, did Sloboda declare that English media reports (supplemented by some other statistics) have captured fully half of all war-related deaths in Iraq since 2003?  What is the empirical evidence for this claim?  Beyond the staggeringly high morgue figures I cited in my article for the period before the Shiite uprising began, a 2008 study in the British Medical Journal compared “report-based” methods of counting war deaths with survey methods in thirteen countries.  It found that “uncritical acceptance of media reports” in studies similar to the IBC’s underestimated actual war deaths by a factor of three.  If this holds true for the IBC’s monitoring of news stories from Iraq the actual number of fatalities would today stand at 366,000 civilian deaths from violence alone.  This number would not include Iraqi combatant deaths, nor deaths caused by the war but not by direct violence.

Perhaps it is your contention, however, that media reports of civilian deaths in the Iraq War were far more accurate and comprehensive than in other conflicts.  You begin your letter by suggesting that the IBC’s work was in no way hampered by lack of Western media access to violent areas or the necessity of traveling with military escorts.  In support of this notion you site the large number of international sources the IBC has drawn on, without giving any indication of how these sources might be weighted in the IBC’s work.  Even for those media outlets that had “embedded” journalists, you allege, it was “a small number of their reporters.”  You offer no supporting figures and so I will provide a figure for you.  The number of officially embedded foreign reporters at the beginning of the war was 800 according to USA Today. This is hardly “small.” 

Further, as David Ignatius wrote in the Washington Post this past year, “As violence spread throughout Iraq in late 2003 and 2004, and as insurgents employed kidnapping as one of their weapons, it became all but impossible for Westerners to travel freely. To move anywhere outside central Baghdad, it was wise to embed. American reporters typically embedded with U.S. units, spending a week or two with them.”  The Arabic and international media, Ignatius continues, had somewhat greater ability to move about the country and even in some cases were “embedded” with the insurgents.  Yet much of this reporting was not in English and so is not included in the IBC’s work.  Nor was this reporting without its perils.  The vast majority of reporters killed in Iraq were Iraqi reporters (117 compared with only two from the United States and 13 from Europe, according to the Committee to Protect Journalists).

The low end of the Lancet study’s confidence interval, you will recall, was 392,979 excess deaths as of June 2006.  The Lancet study—unlike the IBC—made no distinction between civilian and combatant deaths.  It also included excess mortality.  How, then, do you propose we take into account combatant deaths, excess mortality, and unreported killings in addition to the IBC’s figures of reliably documented civilian deaths, which you seem to accept?  More than 90% of those killed in violence, the Lancet researchers reported, were men, and nearly 60% of these were men of military age (between 15 and 44 years of age), even though this group only represented about a quarter of the total population.  This suggests that a high number deaths in Iraq have involved individuals who might be classified as “combatants” and so be systematically excluded from the IBC’s reporting.

Deaths from causes other than direct violence is also something that I did not discuss in detail in my article due to space limitations, although much more could have been said.  A study published in January 2011 in the International Journal of Environmental and Public Health, for example, found that infants in the city of Fallujah have birth defects at levels more than 11 times the normal rate.  The authors believe this mysterious and growing “epidemic of abnormalities” and cancers may be the result of depleted uranium rounds—munitions which were heavily used by U.S. forces in their assaults on the city in 2004 and which contain ionizing radiation and other metals that leave toxic residues in the environment.  “Excess mortality” might thus well continue for many decades to come in ways that are virtually impossible to quantify but that are no less real for this fact.  Recall, however, that the city of Fallujah was excluded entirely from the Lancet study.

The World Health Organization meanwhile declared in 2008 (contrary to your assertions that it had nothing unambiguous or consequential to say about excess mortality rates) that in the IFHS study, “The non-violent mortality rate increased by about 60%, from 3.07 deaths per 1000 people per year before the invasion to 4.92 deaths per 1000 people per year in the post-invasion period.”  Although the question of pre-invasion mortality rates is hotly debated as you observe, it is not at all unreasonable to apply the WHO’s figure of a 60% increase in non-violent deaths to its own baseline figures (which may be debatable but which cannot be dismissed merely by citing other estimates, all of which are open to criticism).  This yields, again, an entirely plausible figure of approximately 400,000 total excess deaths.

But if you think this figure is grossly inflated on grounds of personal incredulity or from a strong preference for a different baseline figure, let us simply cut this figure in half to 200,000.  Following the IBC’s trend-line we must still roughly double the figure to account for deaths post-2006.  And so we arrive at approximately 400,000 excess deaths again.  This is something you have failed to consider.  The IBC’s fatality figures (excluding the latest Wikileaks revelations) have more than doubled since June 2006 when both the Lancet study and the IFHS study ended.  This means that if media reporting can be used to establish trends of violence—which the IBC says is the real value of their work—the IFHS figure should be more than doubled now as well, yielding a total of approximately 300,000 violent deaths, excluding all other war-related mortality.  Similarly, even if we discard fully two-thirds of the deaths in the Lancet study on the grounds they were somehow “fabricated” or the result of sampling errors and main street bias we would still end up, following the IBC’s trend line, with a total number of deaths to date in excess of 400,000.

An estimate of hundreds of thousands of Iraqi lives lost is therefore by no means simply the low end of the Lancet study’s confidence interval.  If I had simply and uncritically accepted the Lancet study’s findings as you charge, then following the IBC’s trend line that more than doubles violent deaths since June 2006, I would have argued for an excess mortality figure today somewhere in the range 800,000 to 1.8 million.  I did not make such a claim but instead pointed out that all three major studies support a figure of hundreds of thousands of deaths.  This can very reasonably be inferred from the data from all of the surveys when placed alongside each other, even in an excessively conservative and cautious light.

Why, then, are you critical of the Lancet article alone without offering a single remark on the potential sources of underestimation or downward bias in the IBC’s work or in the article by Guha-Sapir and Degomme (which you cited from Wikipedia) that argues for a total “war-related death toll” of around 125,000 as of 2006?  The problem of bias can certainly cut both ways.  Unfortunately, it seems you are only interested in possible sources of overestimation.  It is hard not to conclude that your sole interest is in discrediting the higher figures of the Lancet study, without real concern for questions of accuracy or objectivity or balancing conflicting evidence.  At no point in your letter do you advance a positive argument for what you think is the best estimate of total excess mortality in Iraq, making clear your own assumptions and preferred methods or their limitations (although you allude at least twice to the superiority of “triangulation”). 

It is very easy, of course, to find potential sources of bias or error in any social scientific study—whether in Iraq, Darfur, or elsewhere—and many scholars spend entire careers simply using regression tools to critique the research of others without ever advancing a positive argument or generating original data of their own.  Skepticism of this sort certainly offers many professional rewards and few dangers.  It is, in a word, cheap.  In the face of some human and moral disasters, however, the path of endless skepticism and methodological debate is a luxury, it seems to me, that we simply cannot afford.

Mr. Osborn,

Thanks for the reply. You are introducing some new mistakes here which i will discuss below.

But first, I am not aware of the "balanced review" from 2008 which you say appeared in the New England Journal of Medicine. I'd be happy to read it if you care to provide a link. However, I would note right away that it would seem somewhat odd if your description were accurate, since one such piece of evidence is the IFHS report that was published in the New England Journal of Medicine in January of 2008, which concluded unambiguously that the Lancet survey overestimated. It is hard to see then how or why another piece in the NEJM would say there is no such evidence. I would also note that a review from 2008 would not have been able to consider evidence that has emerged since that time, such as the paper I referenced above (http://www.informaworld.com/smpp/section?content=a921401057&fulltext=713...), which presents considerable evidence of fabrication in the Lancet data, and which was published in 2010. There are several other things since that time which have also influenced this debate, such as the rare formal censure of the survey's lead author by AAPOR (http://tinyurl.com/yct4p23), or the thorough refutation of a poll-based estimate which was the only other source to suggest a similarly high number of violent deaths as the Lancet did (http://www.iraqbodycount.org/analysis/beyond/exaggerated-orb).

I don't think so much controversy continues now with regard to the Lancet survey. That is not to say there is a "consensus" against it. Its authors and a few of their friends continue to insist that it's right, and some left-wing blogs and such still do, but it seems clear that serious support for it has dwindled considerably over the last few years, down mostly to a handful of die-hards. Regardless, my point was not that nobody believed it credible, but I still think your review gave very minimal attention to a large amount of substantive criticism it has received, while heaping criticism on the others.

On to the new errors, You write that the three studies you cite "provide converging support" for an excess deaths figure of "400,000 lives if not higher", that is, you present this figure as a "conservative" minimum again, as if like the bottom of a confidence interval, a lower bound. I would say that this claim can not be sustained with any reasonable reading of the IFHS (and certainly not of IBC). As I look at the IFHS data, if you were to derive a figure for excess deaths, which the study did not do, there are two plausible approaches. The first is to use the raw figures from the survey ('unadjusted' figures) and this would seem to come to something like 270,000 excess deaths. Alternately, you could incorporate the upward adjustments they suggest with their violence estimate, and this would come to an adjusted estimate of 400,000 excess deaths. Thus, there is already a range of plausible estimates to draw. In addtion, both of these estimates would have error margins that extend below 270 and above 400. Then you might also consider the issue of the baseline rate, as I discussed last time, and the range of possible estimates would expand again. There just is no basis for the claim that the 400,000 figure is some kind of minimum ("if not higher", you say) from the IFHS, because this figure relies on using particular assumptions, and is not robust to all kinds of plausible adjustments in these assumptions.

You then say that IBC would somehow support this 400,000 ("if not higher") excess figure. This just makes no sense at all. You say that John Sloboda of IBC once suggested he thinks "about half" was their best estimate of their coverage. You say this leads to a "conservative estimate of 244,000". But there are several things wrong with this. First, this statement seems to come from an interview from about five years ago. It doesn't seem "conservative" to just extend this to the present, as if he were predicting coverage rates indefinitely into the future, but maybe it's reasonable. Second, if Sloboda is giving his "best estimate" that implies that he thinks the true number may be lower or higher than that, with his "best estimate" being the most likely in a range of possibilities. So where does "conservative" come in here? Again, you seem to be turning a central estimate into a lower bound. Third, based on their current total of (99,701 – 108,853) the "about half" idea would suggest something like 200-215,000. You say that they report 122,000, and therefore claim 244,000, but you're misreading the 122,000. That number is the result of including an estimated 15,000 additional deaths from the newly emerged WikiLeaks data. These newly discovered deaths would be filling in and reducing the very reporting gap Sloboda was talking about. You're adding them on top of it. Whatever the rate of underreporting in IBC before the WikiLeaks data, it is less after it, not the same.

Lastly, even if we accept a figure of something like 200-215,000 (or even your mistaken 244,000) this does not provide any support at all for 400,000+. In order for it do so we must assume there are a _minimum_ of 150-200,000 non-violent excess deaths to add on top of the (already non-conservatively doubled) IBC figures. The IBC figures themselves provide no basis for that assumption. They don't measure those kind of deaths and make no claims about them. So the assumption has to come from somewhere else. Where? The IFHS data suggests something in that range may well be true, but as above, this is hardly robust, and certainly not a lower bound. Turning to the Lancet data, that would suggest that 150-200,000 is too high for non-violent excess.

We then come to a very unfortunate citation on your part, and some other decidedly non-conservative inferences you draw from it. You cite an article from the British Medical Journal. You claim that this paper looks at methods like IBC and says they generally underport war deaths by "a factor of three", and you therefore derive a figure of "366,000", increasing your already mistaken figure of 244,000 by half again. First, this BMJ study and all of the claims you cite from it have been thoroughly refuted in a subsequent paper in the Journal of Peace Research. This paper shows that the BMJ authors misrepresent the  data they are criticizing (it is not based on "media reports" in the first place) and their claim of underreporting by a factor of three is "simply false". That BMJ paper is a complete mess, which, "signally failed to substantiate any of its major criticisms, while containing serious methodological and factual errors", to quote the conclusions of the JPR paper.

Here is a link to that paper:
http://www.hsrgroup.org/docs/Publications/Additional-Publications/Spagat...
There is more detail in an Appendix here:
http://www.hsrgroup.org/docs/Publications/Additional-Publications/Spagat...

These really just refute all of the major claims in that BMJ paper, so if you do read these, I doubt you would cite that paper again, let alone try to draw inferences from it about the Iraq war. However, putting that aside, the BMJ paper itself did actually touch on the Iraq war, and gave its own estimate based on its findings at the end. It gave a figure of 184,000, or about half of the figure you chose to infer from their groundless "factor of three" headline. This figure is discussed in section 3 of the Appendix linked above.

"Perhaps it is your contention, however, that media reports of civilian deaths in the Iraq War were far more accurate and comprehensive than in other conflicts."

That wasn't a contention of mine, no. If you're asking now, I would think that probably is true because of how high profile the Iraq war has been relative to most conflicts, and due to advances in communications technology over previous wars. However, as discussed above, the BMJ article has nothing to say about how comprehensive "media reports of civilian deaths" were in other conflicts. More broadly, I don't think there is much knowledge about how comprehensive media reports of deaths were in other conflicts because I don't think there's been much research on that question across anything like a broad range of past conflicts. As such, there's little reason why I would contend anything about media reporting in Iraq relative to previous conflicts. The paper I referenced above from informaworld discusses one prior example near the end, which compares a study of the Afghanistan war by Johns Hopkins researchers to a media-based count similar to IBC's approach and that comes out to around 65%: "The survey found 5576 killed. This compares to a media-based count of 3620 civilians killed for the same period." So that's one data point to consider.

"You begin your letter by suggesting that the IBC’s work was in no way hampered by lack of Western media access to violent areas or the necessity of traveling with military escorts."

No, I didn't make this suggestion, just as I didn't make the contention above.

"In support of this notion you site the large number of international sources the IBC has drawn on, without giving any indication of how these sources might be weighted in the IBC’s work."

No. I didn't give a weighting. Nor did you. Your article instead just repeatedly implies a heavy weighting on the work of "Western reporters" in particular (who are supposedly mostly trapped in a bubble in Baghdad), "embedded" reporters in particular, and a supposedly very limited range of sources or reporters outside of "Western" ones. My citation shows something quite different. The IBC website has quite a bit of discussion on this kind of thing which is relevant. Here are some links on this:

http://www.iraqbodycount.org/analysis/qa/assessment/5
This one shows sources on a particular day and the range of originating countries and languages for that day.

http://www.iraqbodycount.org/analysis/beyond/put-to-work/3
This one lists the top 15 sources used over a given period and says that four of these "are Iraqi or regional sources whose output originates in Arabic." Another is Chinese.

Another four near the top are "Western" news wires that have lots of reporters stationed all over the country. The top one there is Reuters:
http://www.pressgazette.co.uk/story.asp?storyCode=35752&sectioncode=1r
That article gives some overview of their work in Iraq. It shows that they are hardly isolated in Baghdad, and even though a "Western" agency, its output comes mainly from Iraqi reporters stationed around the country. I suspect that the other big wires work similarly. So talking of "Western" this and that doesn't seem to make so much sense here. Most of the biggest sources IBC uses are either Iraqi/Arabic sources themselves, or are Western wire sources whose staffs are mostly Iraqi/Arabic. Then there are a wide range of other sources with less casualty output than those, some of which may more resemble the "Western", trapped in Baghdad, stereotype you present, and other smaller Iraqi and regional sources.

Another IBC piece addressing this issue here:
http://www.iraqbodycount.org/analysis/beyond/state-of-knowledge/8

"The Arabic and international media, Ignatius continues, had somewhat greater ability to move about the country and even in some cases were “embedded” with the insurgents.  Yet much of this reporting was not in English and so is not included in the IBC’s work."

Ok. Then it should be pretty easy to list a few examples of such reported civilian deaths that are not included in the IBC's work. The links I've provided above show that IBC includes a lot of "Arabic and international media", so it won't actually be that easy.

As another piece from IBC says,

"Do the English language sources on which we rely miss sources that are carried by the Arabic language media? Although this is a presumption made by some of our critics, we have found little evidence of this. Two facts are significant. no-one has ever sent us an Arabic media report reporting deaths in Iraq we did not already know about from English language sources. Secondly, there are well-known Arab news media that publish some of their stories in English and are included among IBC's sources. There are also some agencies like the BBC that provide translations of key Arabic sources. We have rarely found deaths among the latter that have not been reported elsewhere."
http://www.iraqbodycount.org/analysis/qa/ibc-in-context/11

On the 'embedding' issue, you complain that I, "offer no supporting figures and so I will provide a figure for you.  The number of officially embedded foreign reporters at the beginning of the war was 800 according to USA Today. This is hardly “small.” "

I wasn't speaking about "the beginning of the war". I was speaking about the war as a whole. I would think such numbers were highest during the initial invasion and then dropped precipitously thereafter. But even so, I would think that 800 is still a fairly small portion of the total number of journalists operating in Iraq at the time. I also don't know how much of _casualty reporting_ of the type IBC uses actually came from embedded reporters. It may not have been very much, even during the invasion when the practice was most common.

"The Lancet study—unlike the IBC—made no distinction between civilian and combatant deaths.  It also included excess mortality.  How, then, do you propose we take into account combatant deaths, excess mortality, and unreported killings in addition to the IBC’s figures of reliably documented civilian deaths, which you seem to accept?"

IBC discuss the combatant issue here, which is where the 122,000 figure you used before came from: http://www.iraqbodycount.org/analysis/numbers/warlogs/

It says that total would go up to about 150,000 if adding Iraqi combatants, so about an 80/20 split.

The IBC data also shows a very high proportion of deaths being adult men, about 80% (11% women, 9% children), rather than the 90 you say the Lancet had, but i guess something like this would make sense since almost 100% of combatant deaths would be adult men. (http://www.iraqbodycount.org/analysis/numbers/nejm-2009/)

For excess mortality other than violence I think the information just isn't there to have any kind of reliable conclusion. I think the data that is available suggests these went up, not down, during the war but I think it's pretty hard to be confident in any particular figure for reasons I discussed earlier.

"A study published in January 2011 in the International Journal of Environmental and Public Health, for example, found that infants in the city of Fallujah have birth defects at levels more than 11 times the normal rate."

Yes, i saw this but I found some things about this study a bit dubious in terms of how they describe their methodology and the apparent strong opposition by the Iraqi government, essentially on the grounds that the teams doing this on the ground were part of the insurgency. However, things sort of like this have been reported anecdotally elsewhere too.

"Recall, however, that the city of Fallujah was excluded entirely from the Lancet study."

Well, no. You're confusing the 2004 and 2006 ones. Fallujah was included in the 2006 study. It was excluded partially from the 2004 study. Namely, it was excluded from the main excess death estimate, but was included in other findings in that study.

"The World Health Organization meanwhile declared in 2008 (contrary to your assertions that it had nothing unambiguous or consequential to say about excess mortality rates) that in the IFHS study, “The non-violent mortality rate increased by about 60%, from 3.07 deaths per 1000 people per year before the invasion to 4.92 deaths per 1000 people per year in the post-invasion period.” "

My assertion was that the IFHS didn't report any estimate of 400,000 excess deaths. You again change what I'm saying to argue with a straw man, so that he can be "contradicted". What you quote is from the Q&A to the IFHS report, but you truncate the quote. It goes on to say, "Further analysis would be needed to calculate an estimate of the number of such deaths and to assess how large the mortality increase due to non-violent causes is, after taking into account that reporting of deaths longer ago is less complete."

The broader quote shows at least two things. First, you were indeed in error in claiming that the IFHS reported 400,000 excess deaths. Period. Full stop. I want to point out that you never acknowledge this fact. Indeed, your reply never acknowledges any of the other factual errors I pointed out either. You don't acknowledge any of the points where you were clearly wrong, but instead just start arguing with me over various other things. I find this somewhat disturbing.

Second, the quote states that if they were going to make such an estimate that, "further analysis would be needed" which takes "into account that reporting of deaths longer ago is less complete". What to make of this? There's only one possible interpretation. This statement means that "3.07" is a "less complete" measure of deaths than is "4.92" for the respective periods. If so, this would mean that "a 60% increase" is too much. Some of that increase is not real. This is another of the assumptions open to interpretation in converting this data into an excess death estimate. Furthermore, any increase across two time periods is not necessarily _caused_ by any one factor. Not only is the "60% increase" not "unambiguous" in terms of possible varying completeness, but it would also be ambiguous as to how much of that increase was actually caused by - rather than occurred during - the war. The IFHS says nothing of this causation-correlation issue either. In the same Q&A they show they are aware of not drawing causation conclusions too loosely:

"Q: Does this estimate represent "excess" violent deaths - those attributable to the invasion?
A: No. It is an estimate of how many violent deaths occurred between the March 2003 invasion and June 2006. The study did not measure whether or not those deaths would have occurred had there been no invasion. However, the mortality rates for 2002 and early 2003 showed that mortality due to violent causes was low before the invasion."

So they are saying that their estimate of 151,000 is not an estiamte of violent deaths _caused by the war_. It is an estimate of violent deaths that occurred during the war. They make no claim about whether the war caused these deaths, or how many. They simply note that they measured a low rate before the war, informally suggesting that most are probably caused by the war. To make a claim about 'excess deaths due to war' is not just to compare a difference in two rates. It's also to make a claim about the cause of that difference, and one which excludes all other potential causes but the war.

You say, "it is not at all unreasonable to apply the WHO’s figure of a 60% increase in non-violent deaths to its own baseline figures (which may be debatable but which cannot be dismissed merely by citing other estimates, all of which are open to criticism).  This yields, again, an entirely plausible figure of approximately 400,000 total excess deaths."

I would agree it's not necessarily "unreasonable" to do it. But it's also not necessarily unreasonable to do it with a wide range of different plausible assumptions as well, and which would lead to a wide range of plausible figures. I didn't say that 400,000 excess deaths was not plausible. It's just that it is but one plausible way to interpret the data among many, so that a wide range of other figures are also plausible, including ones much lower than 400,000. Nor did i say the IFHS baseline figures can be "dismissed". My point was that again there are a range of other plausible baseline figures that can't be dismissed either, such as perhaps the Lancet's, which would make most of those 400,000 excess deaths disappear.

My point was not to say that 400,000 is not plausible. I would say that it is. But it's one plausible way to interpret the data. There are other plausible ways to interpret the data as well. My point was to say that there is no basis for presenting that 400,000 as if it were some kind of "conservative" minimum or lower bound, as you were doing.

"But if you think this figure is grossly inflated on grounds of personal incredulity or from a strong preference for a different baseline figure, let us simply cut this figure in half to 200,000.  Following the IBC’s trend-line we must still roughly double the figure to account for deaths post-2006.  And so we arrive at approximately 400,000 excess deaths again."

Two problems with this. First, "cut this figure in half" again may be one plausible approach, but it still does not give you a minimum figure as you'd been claiming prior. To point out just one reason, the 400,000 includes this "upward adjustment" stuff that IFHS did, but that none of the other studies do. Without these adjustments, the central figure already drops to something like 270,000. The standard error margins on this probably dip toward 200,000 (half). This is without any discussion of what we think about the baseline rate, or other assumptions that are built into this kind of estimate.

Second, there is no basis for following IBC's trend-line with an "excess deaths" estimate that consists substantially of non-violent deaths, because variation in non-violent mortality may not correspond to violent civilian death rates. Again, it is certainly one plausible assumption that they do. But it is also plausible that they don't. Deaths from non-violent causes might go down even as violent deaths increase. Indeed, that link I provided about the Congo estimates last time shows that this actually seems to happen in many conflicts. A curious point about the Lancet data is that the excess deaths from non-violent causes go down during about the first year and a half of the war, according to that data, and then go back up again during the second half of the period to become positive again by the end. If you accept that trend line, the trend for the two types of deaths have no correlation at all, so why would we assume that non-violent mortality trends after those surveys correspond to the IBC trend line?

Again, it is not that these particular assumptions are not plausible ones, but there are other plausible ones as well, so we can certainly arrive at "400,000" by plugging in those particular assumptions, but we can not arrive at "400,000 or higher" this way.

Likewise, you say, "even if we discard fully two-thirds of the deaths in the Lancet study on the grounds they were somehow “fabricated” or the result of sampling errors and main street bias we would still end up, following the IBC’s trend line, with a total number of deaths to date in excess of 400,000."

But why "two-thirds"? Even if we take the IFHS central estimate of 150,000 we must discard three-fourths of the Lancet's 600,000 violent deaths. If we accept the 125,000 conclusion of CRED that i cited last time we need to discard more than three-fourths. And again, these are 'best estimates' from these groups, not lower bound minimums.

"An estimate of hundreds of thousands of Iraqi lives lost is therefore by no means simply the low end of the Lancet  study’s confidence interval. "

"Hundreds of thousands" means 200,000 or more, and is not the low end of the Lancet interval.

Likewise, you say that you "pointed out that all three major studies support a figure of hundreds of thousands of deaths." But again you are changing the story. You said they support a claim that 400,000 is a minimum figure. They don't.

"Why, then, are you critical of the Lancet article alone without offering a single remark on the potential sources of underestimation or downward bias in the IBC’s work or in the article by Guha-Sapir and Degomme (which you cited from Wikipedia) that argues for a total “war-related death toll” of around 125,000 as of 2006?"

I noted major criticism of the Lancet article because your article ignored it all. I also did not focus on potential 'downward bias' in IBC because you went to great lengths to focus on (and in my view exaggerate) this. As to the CRED piece I think their conclusions have the potential for both upward and downward bias.

"It is hard not to conclude that your sole interest is in discrediting the higher figures of the Lancet study,"

My interest was in correcting factual errors in your article (all of which you ignore in your reply) and to balance what was in my view a rather biased, and in several ways misleading, overview of the evidence which hyped all kinds of criticism of sources like IBC and IFHS as being too low, while ignoring literally all the substantive criticism of the Lancet article. I further think that it is the evidence that discredits the Lancet figures, and if people are able to see this evidence (rather than having it hidden and not mentioned), they will have a much better understanding of the issue you were trying to write about. In short, my interest was in "facing the question of Iraqi civilian deaths as honestly and objectively as possible".

Mr. Pierce,

In my article and in my earlier letter to you I argued that there are very good reasons to believe excess mortality in Iraq exceeds 400,000 deaths based on the best evidence we have from all of the studies, none of which can simply be dismissed by way of appeal to scholarly consensus.  You now make clear that you actually largely agree with me, even though you would have us discard the Lancet study entirely.

You now allow for a “plausible” range of excess deaths from the data in the IFHS study of between 270,000 and 400,000.  Even by a conservative reckoning (taking the lower end of this interval), we are thus already speaking of hundreds of thousands of deaths—not including any deaths that have occurred since June 2006.

There comes a point when prolonged debate over statistical methods, where human lives are the statistics we are talking about, begins to degrade everyone concerned.  I fear we may be reaching this point (I include myself) and so this will be my final note.  Below are my responses to what I take to be your most substantive arguments and questions in your recent posting.  I leave you with the final reply if you desire it.

Sincerely,

Ronald Osborn

___

1.     The balanced review I cited from 2008 appeared in the January 31 issue of NEJM and was written by Catherine and John Brownstein.  It is entitled “Estimating Excess Mortality in Post-Invasion Iraq”.  You should be able to find it without trouble on JSTOR or through another scholarly archive.  They write: “The [Lancet] survey methods have been scrutinized, and observers have put forward convincing arguments both that it does and that it does not overestimate mortality.”

2.     You note that a more critical article has been published more recently.  What you do not mention is that the paper you cited is by an individual who has been closely connected with the Iraq Body Count.  It is, in other words, very much part of the acrimonious and ongoing inter-study fight I mentioned in my article.  This does not mean Spagat’s work should be dismissed.  It does mean we must approach it with caution no less than the Lancet’s work and not assume that it is by any means the final word.

3.     The censure of one of the Lancet survey's authors by his university had absolutely nothing to do with the team’s findings or statistical methods.  It involved concerns by Johns Hopkins over whether the anonymity of some responders had been adequately protected according to the increasingly strict rules many universities have adopted for human subject testing.  I fail to see why this is relevant to our discussion.

4.     You write that I “gave very minimal attention” to criticisms of the Lancet article “while heaping criticism on the others.”  In fact, I did not “heap” criticism on any of the studies, all of which could be subjected to a great many criticisms not included in my article.  I readily concede that I did not delve into the methodological debates surrounding the Lancet study which would have required an article of its own (and you may be interested to know that the article I originally submitted to Commonweal was more than 1000 words longer than the one that was published, such were the constraints the editors and I were working with).  I did, however, make clear that there is a highly controversial and still ongoing debate about Iraqi deaths, and I shared with readers some of the key criticisms of all of the studies—including the most important criticism of sampling biases in the Lancet study.

5.     You make a good point that we should not include the latest Wikileaks figures in any doubling of the IBC’s figures following Sloboda’s estimate that the IBC has captured fully half of all violent civilian deaths.  I accept your argument.  The questions remain: Why should we trust Sloboda’s estimate?  How do we now factor in combatant deaths?  And how do we factor in non-violent excess mortality?  You have offered no suggestions for how to address these problems.  In fact, no body count method can answer these kinds of questions, which is why survey methods are needed, deeply problematic and vulnerable to criticism though they all will be.  There are, to date, two surveys that have been published in leading peer-reviewed journals.  There is still no scholarly consensus on the reliability of either and it is not simply a few obdurately irrational individuals (“die-hards”) who continue to take the Lancet article seriously.

6.     You argue that we cannot extend the validity of Sloboda’s estimate that the IBC captured half of all deaths since it comes from an interview that is several years old.  What you do not say is how the age of his statement in all likelihood would affect its reliability.  One of the major criticisms of media-based methods of counting fatalities is that over time what we witness is media fatigue.  As the public loses interest in stories of Iraqi deaths, news agencies redeploy their reporters elsewhere.  Stringers are no longer hired.  Stories begin to appear with less frequency.  Unless counter-balanced by new sources of information or new political events that generate new levels of public interest and media engagement, the IBC’s work would therefore in all likelihood become less and less accurate over time.  This may have something to do with the IBC’s decision in 2007 to begin admitting single-source reports (which I failed to catch in my article and which I appreciate your correcting in your first letter).

7.     You write that the “BMJ study and all of the claims you cite from it have been thoroughly refuted” in an article in the “Journal of Peace Research” (note: it was in fact the Journal of Conflict Resolution).  The article you refer to was co-written, once more, by one M. Spagat, whose every latest publication is apparently an irrefutable blow to whatever came before.  It may be that you have carefully studied both of the articles in question as well as their datasets and so are able, from a standpoint of detached objectivity, to render a verdict of “thorough refutation.”  But thorough proofs and thorough refutations are both rare to nonexistent in the social scientific literature I am familiar with.  As I wrote in my article, human affairs continue to elude reduction.  The debates thus tend to be interminable and almost always deeply inconclusive at the level of methodological and theoretical disagreements.  Hence, if an article appears next year strongly critiquing Spagat et al. I will not leap to the conclusion that he too has been “thoroughly refuted.”  That is not how social science works—especially not on the basis of articles published only one year apart.

8.     I do indeed think that the IBC’s work is weighted in important ways towards Western media and I think you think this as well.  One reason is because of the much greater resources of Western media outlets and their ability to send reporters to cover foreign events.  Another is because of the much greater interest among Western media consumers (relative to, say, China, Japan, or other non-Arab countries) in the Iraq War as an event.  In addition, there is the glaring fact that the IBC only considers English language reports unless stories have been translated (without itself employing its own team of translators).  So what we are really concerned with is the relative importance of Western media—which I trust we both agree is very important indeed—alongside Arab and Iraqi media in the IBC’s work.

9.     Regarding Arab and Iraqi media, I have nowhere denied the importance of Iraqi reporters, whether as stringers for the Western press or as journalists for Iraqi news agencies, in the IBC’s work.  What I would challenge is the idea that Iraqi reporters did not face similar dangers and obstacles as Western reporters, including lack of access to some of the most violent parts of the country throughout the war.  As I stated in my last note, 117 Iraqi reporters were killed in contrast to 2 Americans.  I fail to see how anything I have written about media coverage in Iraq is unreasonable or irrelevant, let alone somehow a “stereotype.”

10.  Since contrition is what you have asked for, I acknowledge and repent of my error in writing that the IFHS “reported an additional 250,000 deaths from nonviolent causes.”  What it plainly reported was that deaths from nonviolent causes had increased by 60%.  It also reported what its baseline figures were.  Others did the math.  Yet you now somehow parse WHO’s words about pre-invasion mortality rates being “less complete” to mean that 60% “is too much.”  That is not what “less complete” means here at all.  What they are saying is that we need more solid information about pre-invasion death rates to speak confidently about excess mortality but that working with the best figures we have we can still infer some results, even though this was not the focus of their study and so is not something they will discuss or defend in their report.

11.  A side note about the violence in Iraq in broad and unscientific comparative perspective with reference to your first posting:  In 2010, the Lancet published the results of a study on excess mortality rates in Darfur, which was based on data from several surveys to compensate for gaps in individual survey data.  You have argued that this method is radically different in principle from what I described as random cluster sampling and you favorably cited the study as an example of responsible social scientific research on excess mortality.   That article found that between 2004 and 2008, approximately 300,000 people had died in the Darfur region because of the conflict, including from increased health risks and diseases.  Darfur has a population of approximately 6 million.  Iraq has a population of approximately 30 million.  The conflict in Iraq has now lasted twice as long as the period covered in the Lancet study of Darfur.  It has involved interreligious and interethnic dimensions that many have described as forms of ethnic-cleansing.  It has unfolded in one of the most highly weaponized and militarized environments in the world (I will not labor you with statistics on tons of explosives dropped or numbers of combatants involved, although I believe 1.5 million U.S. soldiers alone have served in Iraq, so a quarter of the entire population of Darfur).  Perhaps, though, the violence in Iraq has been not only less in absolute terms but miniscule in relative terms compared with Darfur.  Perhaps—although it is an argument that does beg credulity.

12.  You write, “there is no basis for presenting that 400,000 as if it were some kind of ‘conservative’ minimum or lower bound, as you were doing.”  But if you carefully read what I wrote you will not find the words “minimum” or “lower bound” anywhere in my conclusions.  The language I used throughout was a language of best estimates, likelihood, and probability based on an attempt to take all of the conflicting studies seriously but also critically.  I have offered circumstantial and logical reasons for this conclusion in my article as well as in my last note to you (and statistical regression is not the only way to arrive at estimates or to make meaningful statements about what appears most probable or likely).  I therefore stand by what I wrote.  The best evidence we have suggests that the number of Iraqis who have died as a result of the U.S. invasion and the sectarian violence it unleashed is probably more than 400,000.  I have nowhere claimed that a lower figure is impossible or that my article proves this number beyond dispute.  If you would like to argue that a lower figure is more probable then by all means do so and others will then be able to weigh your argument.

13.  Unfortunately, you continue to strongly resist making any clear statement about what you believe the most likely number of deaths is.  Your comments help to highlight the inability of social scientific methods to lead us to incontrovertible conclusions.  I am also grateful to you for engaging with me at length since the debate has helped to sharpen my thinking.  But you have also made contentious if not factually incorrect statements (notably, regarding what the IFHS says and does not say about excess mortality).  You have also hedged yourself in numerous places behind highly equivocal language that does not expose yourself to any risk and that does not move us one step closer to an understanding of the magnitude violence in Iraq.  For example, you write, “As to the CRED piece I think their conclusions have the potential for both upward and downward bias.”  Of course it does.  All quantitative research has “the potential for both upward and downward bias.”  That is a sentence that literally tells us nothing either about the CRED study or about your own deepest intuitions of how many people have died in Iraq from all of your reading (and where empirical certainty is not possible, intuition too has a role to play in public reasoning).

14.  In conclusion: We clearly agree that facing the question of Iraqi deaths is a moral and intellectual imperative.  Any disagreement we have over questions of methodologies should in no way overshadow the staggering reality of Iraqi suffering and the massive numbers of deaths—hundreds of thousands by every indication.  This human reality was the substantive point of my article.  You acknowledge that there are entirely rational, credible, and plausible arguments we might make in defense of the view that the death toll is above 400,000.  I have made precisely such an argument.  If you disagree with this conclusion, I would once more invite you to put forward a positive argument of your own of what you think the excess death toll probably is and what comparative evidences lead you to this conclusion.  If, on the other hand, you actually agree with me that the war-related death toll is probably 400,000 or greater but feel I have failed to adequately make the argument or relied on unreliable data, I would invite you to now help me make the argument stronger. The alternative is not sober caution but a studied agnosticism and epistemological skepticism that does not do justice to the facts we know or to the lives of the people of Iraq.

Generally, I find the focus of this kind of article quite frustrating.  After the first 6 sentences or so, one can understand that we're aiming at proving the Iraq war unjust.  Why not simply say so from the beginning?

While I'll acknowledge that I haven't tracked your views over time, Mr. Osborn, I will comment that I generally understand Commonweal to have a rather..liberal..approach.  One that irritates much more than it educates.

These critiques would be more credible if I felt that staff and writers for Commonweal had a distinct interest in promoting Catholic faith at all times.  Sadly, my general perceptions have been otherwise.  Somehow, it's great to chastise national leaders about Iraq, but when it comes time for personal accountability in living all of Catholic faith on a daily basis, most people fall oddly silent.

Pages worth of statistics will not return anyone from the dead and lackluster faith won't bring about just wars or lack of wars in the first place.

As a footnote to my comments moments ago, I think I should say this:

Just War theory as I have read it tends to imply that one person or a group of people can make a worthwhile assessment of the virtue of any war at some point in time.  It also tends to imply that such a group makes an assessment based on all that led up to the fighting, what happened during the war, and the aftermath.

Given that US documents tend to be classified for about 40 years, I think it unlikely that anyone can make a competent decision regarding the justice, or lack thereof, for any war in any worthwhile time frame.  Even then, one side will almost always view a war as "just", while at least on other will declare it "unjust".  Oftentimes, the distinction will depend on who "won" the war.

I do not reject the theory summarily, but maintain a rigorous skepticism regarding whether anyone can really declare a conflict to be just or not.

Mr. Osborn, you write:

“You note that a more critical article has been published more recently.  What you do not mention is that the paper you cited is by an individual who has been closely connected with the Iraq Body Count." Well, if so, I didn't mention this because it would have been an ad hominem and non-sequitur, in other words, irrelevant.

"The censure of one of the Lancet survey's authors by his university had absolutely nothing to do with the team’s findings or statistical methods.  It involved concerns by Johns Hopkins over whether the anonymity of some responders had been adequately protected according to the increasingly strict rules many universities have adopted for human subject testing.  I fail to see why this is relevant to our discussion."

I never mentioned a censure by a Lancet study author's university. Such a censure did take place in 2009, but I didn't mention it. I think you are confusing this with my reference to the censure by AAPOR.

Now that you bring it up though, I would say the JHU censure does have some relevance to the discussion though. What it shows is that the study was designed, approved and reported to do one thing, but the researchers on the ground in Iraq actually did something else, and the authors repeatedly made misleading statements about this before it was exposed. They continually said that one thing happened on the ground (that no unique identifiers were collected), while what their field teams actually did is something else:
http://lancetiraq.blogspot.com/2009/02/burnham-sanctioned.html

I didn't bring this up because this particular censure alone does not necessarily show the results on mortality are wrong, but it certainly adds to the long list of other doubts that already hang over the results. As one commentator who'd previously supported the study concluded, "That we now know that another crucial detail, the collection of identifiable information, deviated from the published record, and that the authors failed to correct the public record on the matter until forced to, raises questions about what other aspects of the study may not have been conducted as described."
www.zcommunications.org/new-doubts-regarding-the-lancet-iraq-mortality-study-by-stephen-ssoldz.pdf

"You write that I “gave very minimal attention” to criticisms of the Lancet article “while heaping criticism on the others.”  In fact, I did not “heap” criticism on any of the studies, all of which could be subjected to a great many criticisms not included in my article."  

I stand by my description and criticism as correct. Of course you could have heaped even more criticism on the others than you did, but that's kind of beside the point.

"There is still no scholarly consensus on the reliability of either and it is not simply a few obdurately irrational individuals (“die-hards”) who continue to take the Lancet article seriously."

I think that by now it mostly is.

"You argue that we cannot extend the validity of Sloboda’s estimate that the IBC captured half of all deaths since it comes from an interview that is several years old.  What you do not say is how the age of his statement in all likelihood would affect its reliability. "

Yes I did not say this because I don't know how the age would effect it and nor do you. That is the very problem I was pointing out. You go on to present one theory on why it might have gotten worse over time, and then you pretend that this is "likely". I could present other theories that might suggest it has improved over time, but I tend to think this is kind of pointless here. My point stands. Extrapolating Sloboda's five year old claim to the present is problematic and not "conservative".

"You write that the “BMJ study and all of the claims you cite from it have been thoroughly refuted” in an article in the “Journal of Peace Research” (note: it was in fact the Journal of Conflict Resolution).”

Yes, it was JCR, not JPR. My mistake. You go on to basically reject the idea that the BMJ article has been refuted, apparently not on the basis of reading the works I cited, but on the grounds of a general claim that "thorough refutations are both rare to nonexistent in the social scientific literature". Maybe they are rare, but this is very clearly one such case. You should read the papers.

"Since contrition is what you have asked for, I acknowledge and repent of my error in writing that the IFHS “reported an additional 250,000 deaths from nonviolent causes.”  What it plainly reported was that deaths from nonviolent causes had increased by 60%.  It also reported what its baseline figures were.  Others did the math.  Yet you now somehow parse WHO’s words about pre-invasion mortality rates being “less complete” to mean that 60% “is too much.”  That is not what “less complete” means here at all."

Thank you for acknowledging your error on the IFHS. However, your claim above about my supposed "parsing" is preposterous. That is the only thing "less complete" can mean there. The authors write, "reporting of deaths longer ago is less complete". That can only mean exactly what I described: that their measurement of the pre-war rate is "less complete" than their measurement of the post-war. There is no other possible meaning, and your denial of this is ridiculous. If what they state there is true it can only mean that the 60% increase "is too much".

"You write, “there is no basis for presenting that 400,000 as if it were some kind of ‘conservative’ minimum or lower bound, as you were doing.”  But if you carefully read what I wrote you will not find the words “minimum” or “lower bound” anywhere in my conclusions."

Again, ridiculous. You repeatedly used phrases like, "400,000 lives if not higher" and "conservative" etc., painting it as a minimum. However, you seem to agree now that it is not a minimum so we need not keep beating this to death.

On Darfur, you cite an estimate of 300,000 and you try to relate this to "violence in Iraq", but this is not an estimate for violent deaths in Darfur. Indeed, according to its authors, less than 20% of those deaths were due to violence, which would suggest a toll lower than 60,000. If you want to compare this to violent deaths in Iraq it is this figure you'd use, not 300,000. As for the roughly 250,000 non-violent deaths implied here, this may or may not be comparable to Iraq at all. Iraq is a much more developed country than Sudan, for example, and how such deaths might work in one conflict may be entirely different in another. Such deaths can go down in some conflicts while up in others.

Moreover, I did not cite this as "an example of responsible social scientific research on excess mortality". I didn't offer a judgment of it, I just said that it "has mostly been accepted" in public discourse as you also said in your article, but noted that the basis of this is not a "random cluster survey", though I think it includes some such surveys among its sources. Likewise, in your last reply you claimed that I "allude at least twice to the superiority of “triangulation”", but I also did not offer any judgment of "triangulation". I just used the word because that is how the researchers have described that kind of approach.

"Unfortunately, you continue to strongly resist making any clear statement about what you believe the most likely number of deaths is."

I have made several clear statements that I do not believe I have any sound basis to offer any particular estimate of the number of "excess deaths". Thus, I do not offer one. I could of course offer one, but I think it would be a largely pointless exercise in arbitrary assumption-making. I think we know more about violent deaths. The IBC (with the WikiLeaks estimates included), say 122,000 civilians, or 150,000 if including combatants. I consider this a reliable lower bound on the numbers killed. It would probably be somewhat higher than this, and one could make a range of plausible speculations about that, but knowing that at least 150,000 have been killed is already more substantive than what we know about "excess deaths". For one, I'm not aware of any data on excess deaths after 2006. So we would have to assume almost five years worth of non-violent mortality trends out of whole cloth. And excess deaths are not like violent deaths, in that they are not necessarily cumulative over time, and can be somewhat counter-intuitive in this way. For example, assume that non-violent deaths did go up during the period of the IFHS and the Lancet by something between 50,000 and 250,000. But then assume that after the sectarian war wound down by 2008, the non-violent trends fell back down to something below the 2002 level. That would mean as we go forward in time from 2008 we would not be adding more deaths on top of the 50-250,000, we would be subtracting deaths off of it. As such, the overall excess death toll from non-violent causes by 2011 could actually be a lower number than it was in 2006. I of course have no idea if something like that scenario is true, but it's not an implausible one.  

I also reject your claim that I've used supposedly "highly equivocal language". That I didn't go into the kind of detail you would have liked about what I think of the CRED figure, does not mean I was equivocating. I think their figure is very plausible, and I don't see any good reason to think it is clearly upward biased, or clearly downward biased. Though they offered that figure before the IFHS emerged, it independently comes to a very similar conclusion, which adds some authority to it in my view. One big area to me which suggests upward bias is they start by analyzing the 3 sources (IBC, ILCS and Lancet) and conclude on what they consider to be a best estimate for Baghdad from these. But what they do then is build the geographic distribution from Lancet on top of this Baghdad figure, so they implicitly wind up making Baghdad only "average", building in a huge number of deaths outside Baghdad, while all the evidence points strongly against Lancet's  distribution, which suggests that if their best estimate for Baghdad is correct, their total estimate must be exaggerated.