<< ACTIVITY INCREASED ~ MAIN ~ OFF THE BUSES >>
METHODOLOGY FLAWED
Lancet latest:
Researchers at Royal Holloway, University of London and Oxford University have found serious flaws in the survey of Iraqi deaths published last week in the Lancet.
Professor Michael Spagat of Royal Holloway’s Economics Department, and physicists Professor Neil Johnson and Sean Gourley of Oxford University contend that the study’s methodology is fundamentally flawed and will result in an over-estimation of the death toll in Iraq.
Among other Lancet critics: Paul Bolton, a professor of international health at Boston University; Stephen Apfelroth, professor of pathology at the Albert Einstein College of Medicine in New York City; and mortality studies expert Richard Garfield. Labor’s Kevin Rudd puts the death toll at 50,000, some 605,000 below the Lancet study’s estimate.
From the first link:
During email discussions between the Royal Holloway-Oxford team and the Johns Hopkins team conducted through a reporter for Science, for an article published in the journal on Friday 20 October, it became clear that the authors of the study had not implemented a clear, well-defined and justifiable methodology. The Royal Holloway-Oxford team therefore believes that the scientific community should now re-analyze this study in depth.
Any thoughts about this, dipole?
Posted by The_Real_JeffS on 2006 10 22 at 01:24 PM • permalinkI thought peer review was supposed to take place before publication. I guess the authors and publisher were too busy working under pressure to meet a deadline to bother with the traditional niceties.
Posted by Bruce Rheinstein on 2006 10 22 at 02:07 PM • permalinkThe more I read about the appropriateness of this methodology when applied to violent deaths the more inclined I am to paraphrase Bentham’s judgement regarding human rights, that this survey “is nonsense on stilts”.
What’s most disturbing is that the board of the Lancet is allowing the editor to drag this journal through the mud.
Posted by dover_beach on 2006 10 22 at 07:29 PM • permalink“In other words, the analysis was flawless, but its starting point was suspect. Bolton noticed the massive number of Iraqi casualties, called “excess” deaths because they exceeded what would be expected in peacetime, hinged on accepting a single fact—that the rate of death before the war had been as low as 5.5 people per 1,000 each year. In a country of 26 million, that translated into approximately 143,000 deaths annually.”
“To Bolton, that seemed low, especially in comparison to other nations with more advanced health care. In the United States, for instance, the mortality rate is 8.3, and in the European Union it is 10.1.”
Hogwash. There isn’t anything unusual about a death rate of 5.5/1,000 in a nation like Iraq (where the average age of the population is much lower than that of America or Europe).
There are endless problems with Roberts’ “studies” (for one thing he is a fundamentally dishonest person with a political agenda, and he doesn’t have a lick of hard evidence to back up what he’s saying…and only a fool would take the word of people like that), but, that isn’t one of them.
Posted by Dave Surls on 2006 10 22 at 07:37 PM • permalinkHi JeffS #3. I have only seen 2nd hand reports of this. People have used highly abstract stuff about fractals and power laws in financial modelling. I got the impression this might be something similar, involving the way streets branch off into smaller streets. If they have some such model maybe they could check it against map data and quantify any bias in the survey’s selection process.
It seems like a reasonable thing to look at. Maybe they can come up with a better sampling method.
It’s not so clear how all this would affect the results. But I’m just speculating as I haven’t seen their work.
People have used highly abstract stuff about fractals and power laws in financial modelling. I got the impression this might be something similar
Yeah. Fractal analysis. Much like the coastline of a continent, there have been infinite deaths in Iraq since Saddam was removed.
Posted by Rob Crawford on 2006 10 22 at 09:10 PM • permalinkHi Grimmy #14. The authors do describe several unpleasant incidents that took place during the surveys. Also I suspect it is extremely difficult to fake or tamper with such a large quantity of complex data without leaving some sort of fingerprint.
But your suggestion at least has the merit of explaining a 10x error, which nothing else I’ve seen does.
“The authors do describe several unpleasant incidents that took place during the surveys.”
Yeah, I heard the hotel they were staying in ran out of tonic water.
Posted by Dave Surls on 2006 10 22 at 11:35 PM • permalinkAnyone want to take bets on how long it takes Lambert to slime them?
Posted by niobium2000 on 2006 10 22 at 11:36 PM • permalinkAt least Rudd is realistic on death numbers. It was the one thing he said in that interview which made sense.
Apart from that he was just rubbish.
I soooo want to vote Labor in the next election to register my displeasure at the way the Telstra sale was handled, (for the sale - against the way they stuffed it up), but I just don’t think I’ll be able to…
It’s not so clear how all this would affect the results. But I’m just speculating as I haven’t seen their work.
I wondered if you were still seeing Roberts as someone who couldn’t be dishonest.
Not to say that he is being dishonest, mind you, although that’s possible. Delusional comes to mind. But he could still be wrong, and I am not accepting the Lancet at face value, not with all the questions being raised.
But I will say that Robert’s credibility is not looking as solid as it once was. If, in fact, it ever was solid.
Posted by The_Real_JeffS on 2006 10 23 at 12:13 AM • permalink“Well, I took that $20,000 and I flew to Jordan. I paid someone to smuggle me into Baghdad. He was an officer in Saddam’s military.”—Les Roberts
Real honest.
An honest to God traitor.
Posted by Dave Surls on 2006 10 23 at 12:58 AM • permalinkJeffS #21 - I really don’t have an opinion on Roberts as far as the conduct of the research goes. I don’t know him from a bar of soap (as the saying goes here in AU). Clearly he is not a dispassionate observer, but I don’t see that as a reason to discredit him.
It seems like a fairly large number of people are involved in processing the data, including graduate students. I just do not find a conspiracy among all these people plausible.
Also, contrary to what the Holloway/Oxford people are saying, it seems the raw data not been destroyed. As I said above, I think it would be hard to fake this stuff.
Personally I find Tim Lambert’s blog pretty informative on this. There are usually good arguments both for and against in the comments, as well as new links.
People have used highly abstract stuff about fractals and power laws in financial modelling
Survey tatistics don’t work that way. You follow the rulebook. That’s why it’s almost a black and white question whether these guys did it right or not.
A critical thing is, if the sample is not representative, then everything you do after that is worthless, especially if you don’t realise it’s not representative. In the linked article, one of the problems they talk about is “main street bias”. Apparently, the surveyers chose streets that ran directly off main streets for convenience. But in Iraq, living near the main street means living closer to the violence, and therefore means you are more likely to die. This means they surveyed households with the highest death rates in every cluster they visited.Posted by daddy dave on 2006 10 23 at 01:12 AM • permalink#14 Grimmy, they didn’t need to fake the data. It was doomed to vastly overestimate before it began.
I think the results are rubbish, but I don’t think they invented the data. Given all the attention and scrutiny, there would be signs that massive fraud was committed, unless they were criminal masterminds. You’re talking about a large team, local employees on the ground, flights, phone conferences, money trails, accountants, multiple computers and other equipment…
Making all that up would almost be more trouble than just collecting the data in the first place.Posted by daddy dave on 2006 10 23 at 01:23 AM • permalink#24 Daddy Dave - the Holloway/Oxford people are physicists. They claim not just this survey but the rulebook is biased.
What’s the evidence that the selected households are unusually near the main street? I would think that depends on the street layout. You would have to simulate the process on a street map. I guess that is what the Oxford people want to do.
Also what is the basis for your claim that living near the main street means living near the violence? I don’t think Clint Eastwood in ‘Fistful of Dollars’ is a good guide.
the Holloway/Oxford people are physicists.
#26, how did you get that? From Tim’s post:
Professor Michael Spagat of Royal Holloway’s Economics Department
Unlike a lot of other professions, most economics courses demand that students undertake a lot of study in statistics and econometrics. I did four years of it myself. I dare say he knows what he’s talking about.
Posted by Art Vandelay on 2006 10 23 at 03:09 AM • permalinkdipole, you also ask “What’s the evidence that the selected households are unusually near the main street?”. If you had of read the article, you would have found this:
The study suffers from “main street bias” by only surveying houses that are located on cross streets next to main roads or on the main road itself. However many Iraqi households do not satisfy this strict criterion and had no chance of being surveyed.
The authors of the study themselves admitted that they chose houses near main streets. This could introduce bias into the sample.
A second, more interesting source of bias is that they did not survey anyone living in rural areas. From the stats I’ve seen, 7 million Iraqis live in such areas. This would definitely skew the results.
Posted by Art Vandelay on 2006 10 23 at 03:16 AM • permalinkArtVandelay - OK, some of them are physicists. I am not questioning the competence of these people. Maybe they have a good point. But I find it hard to see through the fog of contradiction and second-hand reporting.
For example it is not clear to me that only ‘houses that are located on cross streets next to main roads or on the main road itself’ get surveyed. Where to the survey authors actually say that? They do talk about a ‘list of main streets’. Without access to that list I don’t see how you can even say what they mean by a main street. Yet these econo/physics guys are suddenly the experts on Iraqi urban planning.
Also I think there could be as good or better chance of those mysterious unreported casualties happening in a back street as near the main street, so you could also argue for a bias in the other direction.
I take your point about the rural population. if true, that’s worth a maximum 25% off the total however.
dipole, if you had actually read the study, you would have found this (on the second page!!)
The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed. For this study, a household was defined as a unit that ate together, and had a separate entrance from the street or a separate apartment entrance. [p. 2]
Initially, I used to think that you had something to add on this issue, however, as time has passed by, it has become quite clear to me that you’re just another boring troll.
Posted by Art Vandelay on 2006 10 23 at 04:03 AM • permalinkBut that’s ambiguous. At an intersection do you continue around the corner or along the same street? If the street ends do you go around the corner or come back on the other side? Forty houses seems a long stretch on a non-main street. I admit I don’t know what Iraqi towns/villages look like.
Apologies for failing to mentally challenge you.
dipole, if you can’t understand what the problem is with their approach, then there’s no point debating this with you.
Posted by Art Vandelay on 2006 10 23 at 06:34 AM • permalinkGiven the state of education in the western world today, I wouldn’t be surprised if someone just stuffed up and put a decimal point in the wrong place.
Posted by mr creosote on 2006 10 23 at 07:50 AM • permalinkI’ve been following and contributing to the discussion of this paper over on a couple of statistics blogs. While not rejecting it out of hand the reaction of professional statisticians to this paper has been rather cautious.
The cluster sampling method used is a perfectly valid way of getting estimates. If the underlying assumptions are valid and the sample is a representative one then then they should get a reasonable estimate. If!
The survey was designed to be able to detect a doubling of mortality. Quite a coarse effect. Remember that this survey has been carried out in dangerous conditions and the safety of field staff has been a major concern. Thus the design is less efficient than what they would have been able to use in a non-combat zone. They have used fewer larger clusters than ideally they should have. Probably because of field staff safety considerations. The effect of such a design will be less sensitivity and wider confidence intervals. Still if their assumptions were valid the survey should have been capable of fulfilling its aims.
If the survey was carried out as described there are some potential sources of bias but it hard to see them inflating the estimate by a factor of ten. Some of these potential bias sources are surveying only urban areas, the main road bias, under-sampling of households in apartment buildings and the possibility of double counting of deaths.
The main things that have led to doubts about the paper are the very low non-response rate and the huge difference between these estimates and other estimates.
0.8% of households refused to participate and 0.9% were absent. This is astonishingly low especially since there was no revisiting of households. And a team of four would visit 40 households in a day. Presumably they split into two teams of one man and one woman each of which visited 20 households. This gives them about half an hour with each household. They have to explain the purpose of the survey, gain cooperation, get the information and death certificates in this time. And the description implies that either the head of the household or his spouse was available for interview in every house that responded. The description of what was done and their results seems too good to be true.
It looks like something is being left out of the description of what was done. Like some others have suggested I suspect that the survey teams contacted someone in the sampled street beforehand. Now usually letting people know beforehand that they are in a survey is a good idea. It greatly reduces non-response rates. But in this case it is easy to see the potential for bias and opportunities to corrupt the survey. The authors have also admitted that the Iraqi field staff doing the interviews were opposed to the coalition. I suspect that something went wrong in the field. It could have been from bias or it could have been from ignoring the protocols in order to reduce the risk to field staff.
The Lancet paper gives an estimate of 601,000 violent Iraqi deaths since the invasion. This includes both combatants and non-combatants. The Iraq Body count gives an estimate of about 50,000 non-combatant deaths. The Iraq Body Count includes police and deaths from crime but not insurgents or army. Thus the total announced Iraqi deaths would be higher but I don’t know much higher. 60,000 – 70,000? The Iraq Body Count will be an underestimate because of unannounced deaths. But how many of these will there be? The deaths least likely to be officially recorded would be assassinations by gunfire and deaths from air strikes. Deaths from car bombs and other explosive devices are likely to be recorded. There just do not seem to be the number of deaths from these causes that the Lancet paper estimates. What the Lancet paper does do is give us reason to believe that the admitted deaths from gunfire are likely to be severe underestimates.
In most cases a death certificate was available. This suggests that the local mechanism for issuing death certificates is intact. But have these local figures been transmitted to the central authorities? It is quite likely that at least at certain times they have not. This is what defenders of the John Hopkins study are claiming. Perhaps but it needs to be checked out rather than just asserted.
I’m afraid that the total Iraqi deaths might be uncomfortably large. 100,000 would not surprise me. 600,000 would. I hope the real figure is lower.
Posted by Lloyd Flack on 2006 10 23 at 07:57 AM • permalinkPersonally I find Tim Lambert’s blog pretty informative on this. There are usually good arguments both for and against in the comments, as well as new links.Personally, I find Tim Lambert to be a very strange fellow with odd ways of expressing his opinions. And biased as well.
Frankly, dipole, I find your arguments to be disingenuous and obtuse. For example, your confusion over what is (or is not) a main street. That’s a key point in the methodology that was supposed to be used, yet you argue in a silly manner (re #31, where you call a quote on the matter from the Lancet study “ambiguous”!!!).
That you think Tim Lambert has any credibility does not speak well for your cognitive abilities. Your opinions, as expressed here, merely reinforce that. Henceforth, I am simply going to ignore your comments on this study—there’s already enough noise in the signal.
Posted by The_Real_JeffS on 2006 10 23 at 09:30 AM • permalinkDang! Early morning blogging is a pain. Apologies for the strike out on #38, should have been a quote.
PIMF PIMF PIMF!!!!!
Posted by The_Real_JeffS on 2006 10 23 at 09:31 AM • permalinkLOL, PW! Yes, that’s so, but it’s unintentional, and, alas, I can’t take credit for the humor.
Posted by The_Real_JeffS on 2006 10 23 at 07:35 PM • permalink#35 Lloyd Flack
I suspect that something went wrong in the field.
I agree. You note that the field workers admitted to being anti-coalition, and that there is evidence that the surveys were conducted in impossibly fast time.
Here’s my guess (are you still here Grimmy?): the field workers cherry-picked the data. They knew what theire western masters really wanted to hear, and they also had an agenda of their own.Posted by daddy dave on 2006 10 23 at 11:56 PM • permalinkThey knew what theire western masters really wanted to hear, and they also had an agenda of their own.
And the JH team swallowed it completely. As you say, Roberts and his people heard what they wanted to hear.
Posted by The_Real_JeffS on 2006 10 24 at 01:39 AM • permalinkI still hold to the idea that there’s nothing to prove that anyone ever knocked on an actual door in this entire operation…except when they gave the secret code knock to let the “data makers” know that coalition forces were in the street.
Any of y’all ever had the “pleasure” of standing guard over a pest hole of 3rd worlders who were “doing politics”?
Demographics, records, “studies” the whole nine yards, I’ve seen it created from scratch in sweatshops by wild-eyed rant-o-chanters before.
Is there really anything beyond “because they said so” to back them up? Has anyone attempted to actually reinterview one of the interviewed data pieces? Has anyone even attempted to get a eyesball on the data getters to prove they even exist?
Never mind, ArtVandelay, JeffS, blogstrop. At least you’re in good company. Steven E. Moore and a bunch of Oxford physicists (not forgetting that economist guy of course) for starters.
#44 Grimmy,
If they made it up it will show up when someone looking for evidence of fabrication examines the raw data. You can’t create a random sequence out of your own head. You need a random number table or random number generator. If you try to do it without such help you will create detectable patterns mostly through trying too hard to to make it look random. You’ll end up with too few runs of the same number.Like Daddy Dave I think it’s biased sampling rather than fabrication. I don’t know exactly how they stuffed it up (but I can guess). There are indications of dubious sampling practices. Perversely these are indications that some sampling was done rather than the whole thing being a complete fabrication.
The simplest explanation with the minimum amount of villainy is usually the right one.
#45 Dipole,
I agree with most of what you’ve been trying to point out. This is a dubious but plausible seeming paper. There too much that is reasonable in it for us to be able to reject it out of hand. Nevertheless I think they stuffed it up.The method described should give reasonable estimates.
The estimates are way out of line with any other estimates and it is hard to see why that should be the case for some of the causes of death. (I’m particularly thinking of car bombs and IEDs.)
There are signs of some dodgy sampling.
We can’t just say the numbers look wrong and try to shout it down. It has to be analyzed carefully and to do this we cannot a priori assume that it is wrong.
Posted by Lloyd Flack on 2006 10 24 at 09:45 AM • permalinkGimme some lap tops, local phone books, city maps, some sort of demographics/concensus (or failing that the relevant pages from the publicly available CIA World Book, a few monkeys that can read and type and a few weeks.
500(ish) names picked from a phone book with the addresses listed as pegged on a map. Make up the rest to fit what I think the numbers should be.
There’s no way in hell you or anyone else is going to actually check up on any of it, because it’s not worth dieing for you to do so. And that’s what it would MOST DEFINATELY cost anyone trying to double check. He knows that full well and anyone with half a functional clue would too.
Of course I dont know that’s how it was done. But, all this bullshit about how complicated it would have been is pure bullshit. Maybe something like that would be hard to pull off in a stable, industrial 2nd or 1st world country…but I’ll tell you point blank, that kind of shit happens constantly in 3rd world areas. Anywhere there’s a UN grant that can be qualified for by a “study”, there’s studies being manufactured. I’ve watched it happen.
Why do I keep bringing this up?
1. IF they really went door to door as claimed, there’d have been deaths. IF there had been deaths it would have been ALL OVER the media, print, tv, radio, blogs, non stop…why? Because it would have been put forward as proof that the US/Coalition/Iraqi Gov was trying to squash/squelch/eliminate dissenters. That’s a fucking given.
The faces, names and gruesome details of the door knockers deaths wasn’t front-line news for days and days, therefore I have zero reason to believe door knockers were put into the street.2. The person behind this is an overt and obvious propagandist. Nothing about what he did was based in any sort of integrity or honor. What kind of moron would I be if I just assumed that he actually ever intended to really take risks when it is too damn easy to simply manufacture the results?
Any time something is proven to be based in bullshit and created by liars, only idiots and naive morons will assume that any part of it is based in any sort of attempt at truth.
No matter how you dance and weave, you can not justify either the baseline nor the end product. What the fuck does that tell you?
What you tell the rest of us is that you are so desperate for there to be some salvageable “truthiness” to it that you’ll continue to try to massage it into something palatable.
“There’s no way in hell you or anyone else is going to actually check up on any of it, because it’s not worth dieing for you to do so. And that’s what it would MOST DEFINATELY cost anyone trying to double check.”
I don’t know about that one. UNDP conducted a survey of some 21,000 Iraqi households back in 2004, and none of the surveyors were killed, as far as I know, anyway.
BTW, UNDP estimated that somewhere on the order of 24,000 Iraqis had died in the first year of the war, as compared to the roughly 275,000 Iraqi deaths (if memory serves, that was the actual number Roberts’ survey came up with) Roberts’ strange and unusual initial survey claimed for the same time period.
Posted by Dave Surls on 2006 10 24 at 01:40 PM • permalinkDave, I’m not one to credit anything UN with any more honesty or integrity than this Lancet thing.
Anyway, I am not saying there’s any proof that my suggested events are accurate, only that there’s damn near nothing that can be solidly examined that proves that a survey ever took place.
There is much to suggest that it never did.
The numbers dont jive with anything rational, therefore could be simply manufactured.
There were no loud cries of US/Coalition/Iraqi Gov invovlement in the deaths and/or kidnappings of any of these workers, as there most assuredly would have been if any had actually died or gotten kidnapped.
The suggestion that strange folk could safely wander around in the more heavily contested neighborhoods without bloody repercussions is, to the best of my understanding of the situation there, beyond beyond.
Even “insurgent friendly” door knockers would suffer some attrition simply due to the random nature of the general outrages perpetrated against the citizenry by these “insurgents” and the fact that this “insurgency” is really nothing more than hostile foriegn arabs and muslims in competition with criminal gangs and saddamite loyalists. There’s no HQ or “supreme boss” that any door knockers could get full clearance from when it comes to safe passage and such in “insurgent held areas”.
Also, if one was to even go after such clearance from any of those elements, then they’d have to be known as friendly to those elements, in which case the entire project is, from start to finish, a piece of treasonous propaganda and if it is such, then why the hell go through all the trouble and risk of actually collecting data that is so entirely easy to manufacture from the safety of a hotel room?
Maybe they did actually take to the streets and through the luck of the insane, they got away without huge losses in their own ranks.
If so, then they are doubly stupid and even more dumbass, becuase such data is too easy to manufacture in an environment where there is no possibility of serious double checks by other oranizations or interested persons.
Hey Tim, when are you going to admit that you don’t have the slightest clue of what you are talking about? How long will it take you to recognize that Richard Garfield is not, as you repeately claimed, a critic of the Lancet study?
Here’s what Garfield really thinks:
I am shocked that it is so high, it is hard to believe, and I do believe it. There is no reasonable way to not conclude that this study is by far the most accurate information now available.
Well, Roberts et al could clear up a lot of this by just producing the data. That is, if they actually have data. Fred Kaplan seems to think it may have been destroyed. How conveeeeeenient, if so.
Hey, if one desperately wants to believe in the Tooth Fairy, one can always find evidence to convince oneself.
I find it rather amazing at what appears to be a lot of ducking and dodging by Roberts, et al when presented with tough questions.
And just how long did that ‘peer review’ take, anyway?
Posted by JorgXMcKie on 2006 10 26 at 10:44 PM • permalink
Page 1 of 1 pages
Members:
Login | Register
| Member List
Researchers at Royal Holloway, University of London and Oxford University have found serious flaws in the survey of Iraqi deaths published last week in the Lancet.
Oh, what would those guys know? They’re not even running for office!