Log in

No account? Create an account
06 September 2010 @ 08:14 pm
Bad Science  
We saw Ben Goldacre talk at Eastercon though, I think, I already had his book, Bad Science, following a recommendation from lil_shepherd. It is more carefully structured than his talk was and has more space to explore its argument, which is a definite plus because he never quite got to his point at Eastercon.

In general the book walks through a progression of bogus science starting with the obvious (the example used here are detox products which behave the same way whether in contact with a human or not) and moving through ever more sophisticated forms of bad science, from trials which are uncontrolled, trials which are not blinded to trials which perform essentially unfair comparisons. In the course of this there is a completely fascinating chapter on the placebo effect and a look at experiments that have shown that some placebos are more effective than others.

A running theme throughout the book is media complicity in the promotion of bad science. I know (at least one) of the journalistic friends of my acquaintance feels Goldacre is being unfair to the media in his arguments and painting them as straightforward villains when the reality, in his own phrase, is a little bit more complicated than that. I think actually Goldacre's essential argument here is firstly that evidence-based medicine is woefully underrepresented in our education system, given that the vast majority of science stories in the media are about health. He couples this with an observation that most journalists, and in particular generalist journalists (as opposed to science correspondants) are arts graduates and tend to view science as an argument between opposing authority figures with the evidence as a sideline. Therefore the accuracy of a science story depends on the reputation of the scientist and not on the actual evidence presented. It is certainly pretty damning that so little of the reporting on the MMR health scare went anywhere near a paper's science correspondant. Goldacre makes the point that, despite the media's current interest in presenting Andrew Wakefield as an out-and-out villain, MMR isn't safe because Andrew Wakefield was unethical, it is safe because all the evidence points to it being safe. Andrew Wakefield's character should be irrelevant - although the arena became complicated later, it should have been obvious to anyone with a basic grounding in scientific trials that he was making vastly inflated claims for a study involving only twelve, self-selected subjects.

The book itself stands primarily as a call to teach the basics of evidence-based medicine in schools: what constitutes a trial, what constitutes a trial against control, why trials should be blinded, the problems with applying statistics post-facto, and so on. Goldacre returns repeatedly throughout to the theme of checking the papers for yourself and an insistence that, by and large, these are easy to understand with only a few basics in experimental design and analysis. It is unfortunate therefore, that in some of the later chapters where he is looking at more sophisticated forms of bad science he does actually have to tangle with some fairly complicated subject matter. Early on he makes the point that you should read the evidence in a scientific paper first and only then read the conclusions since these are often written to play up or down certain results. However he then berates the opponents of MMR for ignoring the conclusions in a Cochrane review* and instead focusing on the review's internal criticisms of the various trials it compares.

I didn't always agree with Goldacre and I think he could be accused of simplifying issues in the face of his own assertion that things are nearly always a little bit more complicated than they seem. But the book is immensely readable and I can't disagree with his basic thesis that we should invest more effort into education on the basics of evidence-based medicine especially in face of the media and public obsession with health scares and miracle cures. As primer in how to spot bad science in the medical field, you could do a lot worse than starting here.

Incidentally (though I'm here using Goldacre as an authority figure), three facts jumped out at me from the book. Firstly Vitamin C has been shown to have no preventative effect on the common cold (Vitamin C has always been my least favourite vitamin, possibly because I dislike oranges, so I felt a smug satisfaction at this news). Secondly I've been slowly reading my way through McGee's On Food and Cooking - one of the molecular gastronomy bibles. McGee is big on antioxidants, Goldacre is not. This casts something of a shadow over the whole molecular gastronomy program since it would seem that their scientific approach to cookery is as vulnerable to the lure of quackery as wider journalism. Lastly Goldacre mentions, in his chapter on MMR, that there was a similarly unfounded Whooping Cough vaccine scare in the 1970s. I can actually remember suffering through Whooping Cough and being told I hadn't been vaccinated against it because of the risk of brain damage.

*Cochrane reviews perform meta-analyses, i.e. they analyse the results of all available trials on some treatment. claraste who does systematic reviews of medical literature professionally was rather aghast when I told her Goldacre was implying that lay people should read Cochrane reviews since they are (so she tells me) written to a very rigid template and assume a lot of technical knowledge on the construction of meta-analyses. It is for this very reason that Goldacre insists more attention should have been paid to the conclusion, but at this point you've reached a level of complexity (when should you give more weight to paper's conclusions and when to what you can understand of the evidence presented) that, I suspect, will cause a lot of people to retreat into "I just don't understand" territory.

This entry was originally posted at http://purplecat.dreamwidth.org/17637.html.
philmophlegm: Skeptical Environmentalistphilmophlegm on September 6th, 2010 08:18 pm (UTC)
I should like Dr Goldacre, I really should. He seems to talk a lot of sense. He even went to Magdalen.

But then at the same time, I come back to something a friend of mine from Magdalen said: Ben Goldacre is one of three people (the other two being Stephen Fry and Brian Cox) that people follow on Twitter to 'prove' they are clever. And if you ever read the comments on his blog, you'll see that there is some truth in this.

Mind you, I'd love someone to do the same thing for economics - a 'Bad Economics' if you will. Freakonomics and Superfreakonomics go in that direction, but are more informative than argumentative, and only cover micro-, never macroeconomics.
philmophlegm: cyberleaderphilmophlegm on September 6th, 2010 08:44 pm (UTC)
Oh, and I have this niggling worry that he and Dr Evan Harris work for the Scientific Reform Society...

louisedennislouisedennis on September 6th, 2010 09:08 pm (UTC)
To be honest (as someone who gets most of her news from the Today program) I'd not come across him until Lil mentioned the book on her blog. I've now signed up for his blog feed so I shall reserve judgment until I've read more than one of his posts!

I did feel the book skirted a fine line between being readable and over-indulging in polemic. I suspect that with more exposure to his writing I might have stronger opinions about which side of the line it fell.

Bad economics would be great. I've had people try to explain short-selling and naked short-selling to me several times. I think I've grasped how one of them works, but I'm not sure which. As for the rest of economics - I think I've just about got a grip on how supply and demand interact in general terms....
bunnbunn on September 6th, 2010 08:45 pm (UTC)
Goldacre is quite amusing and has some good things to say, but he's SUCH a drama queen. So sarky in a 'how could anyone think such a thing, they are all SUCH IDIOTS and don't deserve to live!' kind of way.

The other day there was some news story that mentioned him in a manner he thought incorrect, so he posted a comment correction: when the comment had not been published only a few hours later he threw a hissy fit on his blog. Unimpressive in a man that wants people to check references and validate their views, I thought (particularly as the comment was eventually published in what seemed like an entirely reasonable timescale...)

There are some podcasts somewhere of his stuff on placebos they are quite interesting.
louisedennislouisedennis on September 6th, 2010 09:10 pm (UTC)
I can imagine he would be quite difficult to work with on a day-to-day basis. I didn't find the book too sarcastic, though I did think the later chapters over-estimated the simplicity of some of the points he was making, especially in the context of what had gone before.
lukadreaminglukadreaming on September 6th, 2010 09:03 pm (UTC)
Suffice it to say that Goldacre may have interesting and valid points to make, but I was singularly unimpressed by his performance at Eastercon and by a number of interviews I've read with him. I suspect he's starting to believe his own publicity. And yes, his broad brushstroke view of the nasty media doesn't tally with mine *g*.

The issue of science graduates not going into journalism is one that does need highlighting. Sure, there are some who want to cover health, but that's not the same thing at all -- they veer towards the health and beauty field.
louisedennislouisedennis on September 6th, 2010 09:16 pm (UTC)
Well we don't produce enough science graduates full stop*. So it's not a huge surprise that not many of them end up in the media. But I think Goldacre is correct in thinking that if more of the basics of medical trials got taught in schools we would probably see less nonsense in our papers and on the TV screens. In fact I've often thought that science teaching in schools (meaning, of course, the way I was taught science at my school) has an unfortunate tendency to focus on the body of knowledge produced by science as opposed to the methodology and that only reinforces Goldacre's point that science gets presented as opinions put forward by authority figures who have arrived at those opinions by mysterious and opaque mechanisms.

It wouldn't avoid the sneakier bad science creeping in, but would hopefully stop the whole nation getting its knickers in a twist over a study involving 12 subjects.

*from a certain point of view anyway.
lukadreaminglukadreaming on September 6th, 2010 09:22 pm (UTC)
And it's not helped by the way journalists are trained -- get an opinion from an authority figure, then balance that with another authority figure who doesn't agree . . .
louisedennislouisedennis on September 7th, 2010 09:18 am (UTC)
Well journalists, in general, don't give a lot of space to flat earthers, holocaust deniers or even rich department store owners who think the Duke of Edinburgh connived with the Secret Service to murder Princess Diana. So there is a level at which they bring professional judgment to bear on the underlying likelihood of the position being put forward so I'd sort of hope, at least naively, that a better grounding in medical evidence would help them select their stories.

Similarly a better grounding would allow them to ask more searching questions. I don't know if you came across the Toads leaving Aquila story (some toads that were being studied near Aqulia vanished in the days preceding the Earthquake). It was mentioned on the Today programme and John Humphrys is quite intelligent enough and sufficiently unafraid of putting interviewees on the spot, to make the points that correlation does not imply causation, and that you are on dodgy ground drawing strong conclusions from noticing something after the fact that you weren't initially looking for - fairly basic points which wider basic scientific literacy would sort of require a journalist to make (if that makes sense). In this case the scientist involved has done nothing more than say that there needs to be more work, I hasten to add, which is precisely what Wakefield should have done, instead he chose to make medical recommendations for alternative forms of vaccine and so on and so forth.

I think also, since at some level we always have to invoke authority figures, I'd expect journalists to shine the same sort of light onto the false claims, and networks of complicity that can exist in "science" as they do to politicians. Again wider scientific literacy would have made a paper blush to claim that a man who had gained a PhD via a correspondance course with one of America's dodgier universities and who was operating out of his garden shed was the UK's foremost expert on MRSA. As it stands that kind of understanding that not all PhD's are equal and not all "labs" (to be generous) work to the same standards is sufficiently lacking in the public consciousness that it doesn't appear to occur to journalists to check that kind of thing.
Kargicq: Neuromancerkargicq on September 8th, 2010 09:08 pm (UTC)
Hm, I think the problem is more fundamental than that. "Sun" journalists are ferociously bright by all accounts, even if they are arts graduates (grin); I'm sure they could have figured out who was an expert or not if they actually cared. But they are not (per se) trying to find out the truth; they are trying to sell as many papers as possible while not being sued. So "OMG hospitals full of MRSA" was a good story; presumably the lawyers advised that "UK's foremost expert on X" was not a protected title and could legitimately be claimed by anyone.

Neuromancer (UK's foremost expert on fiscal policy. No, honestly.)
philmophlegm: adamsmithphilmophlegm on September 8th, 2010 10:25 pm (UTC)
"UK's foremost expert on fiscal policy. No, honestly."

You can write some articles for my economics blog then!
louisedennislouisedennis on September 9th, 2010 11:51 am (UTC)
Flicking back through the book, I see that the MRSA scare was almost entirely a Tabloid thing. But the MMR reporting was across papers including the broadsheets which may be why I was thinking the MRSA reporting filtered higher up the feeding chain.

There's clearly a class of paper which has no real interest in the truth and that's a much wider problem than in the public understanding of sicence, but I like to think there are a lot that still do and these are making mistakes which you'd hope were easily avoidable, though I seem to have picked the wrong example here.
fredbassettfredbassett on September 7th, 2010 08:26 am (UTC)
Mr FB always raves about that book, and the bits I've read of it were interesting. But he came over as such a complete tosser at Eastercon that I now find it hard to see past that.
louisedennislouisedennis on September 7th, 2010 09:02 am (UTC)
I didn't find him particularly off-putting at Eastercon, though I can see why Luka was annoyed. He was clearly playing to the audience but then all entertainers do and his talk was just as much about entertainment as it was about education. I guess it's a case of YMMV.
fredbassettfredbassett on September 7th, 2010 11:12 am (UTC)
He just came over as someone firmly convinced of his own wonderfulness, but he wasn't bloody bright enough to check his equipment worked and then run to time! I despair of lecturers - or even entertainers - who bugger up on the basics.
Kargicq: Neuromancerkargicq on September 8th, 2010 09:14 pm (UTC)
I didn't go to Eastercon, but I am pretty convinced of BG's wonderfulness from what I have read. Amazing to read someone talking sense about science in a national newspaper; couldn't believe it the first time I came across it. His rants are mainly aimed at targets like pharma company who bury evidence and manipulate data, or people who persist in saying HIV doesn't cause AIDS. So I'll need a lot of convincing that he isn't a hero :-)

fredbassettfredbassett on September 9th, 2010 09:02 am (UTC)
I just found it very hard to get beyond the way he came over as an opinionated, arrogant little jerk to actually listen to what he was saying, which was a shame, as I agree that he does present well in writing.
louisedennislouisedennis on September 9th, 2010 11:54 am (UTC)
It must be said I've been a little surprised at the hostility to BG in the comments to my post. He comes across as vehemently opinionated (not always in a good way) and something of a smart-alec, but you need a larger-than-life persona to be a columnist. But he also does appear to strive to avoid knee-jerk responses and broad-brush condemnations and, as you say, talks a lot of sense.
sophievdennis on September 7th, 2010 10:29 am (UTC)
You are right that BG underestimates the difficulties of a lay person/Arts Graduate understanding original medical literature. My experience of looking directly at reports in medical journals is often one of near total incomprehension brought on by a combination of their specialist medical and statistical terminology. Neither of these formed any significant part of my science and mathematical education, which was hardly lacking. I find the conclusion and abstract are often the only parts I can understand. Therefore I am totally reliant on the researchers' interpretation of their findings.

As such I think he is at times, as you say, overly harsh on journalists. By the same count, BG obviously has a good point regarding science education and not leaving science reporting in the hands of generalist journalists with humanities degrees who, however intelligent they are, have no grounding in the technical detail of the subject.

There is probably also a tangential point to be made about the value of a baccalaureate type qualification at 18 which would require a combination of both arts and sciences. GCSE-level science and maths are still fairly basic (and, if I recall, often "wrong" or at least very over-simplified), but are the point at which many highly-educated and intelligent people stop their studying of such subjects.
louisedennislouisedennis on September 8th, 2010 11:33 am (UTC)
I do wonder if the problem isn't so much that people don't continue in science beyond GCSE (although in wider economic terms it would be good if more people did) but that we don't start teaching the methodology of science early enough...
philmophlegmphilmophlegm on September 8th, 2010 10:43 pm (UTC)
There's an interesting discussion waiting to take place on someone's LJ pages on what 17 and 18 year olds should study. I interviewed a school-leaver the other day for a job at JOLF as a trainee accounting technician. His three A-levels were Maths, Further Maths and Physics. He would have been a stronger candidate with a broader range of thinking. Sophie's suggestion of something like the Bacc would help, but would that mean that new undergraduates weren't as skilled in their chosen subject as people doing the standard three A-levels? Maybe a decent compromise would be to do one specialist subject plus a general qualification.

I would like to see more people know about economics (part of the reason for my recent economics posts) and that is something that could be compatible with teaching younger children about scientific methodology, bias, scepticism, peer review etc. First year of secondary school would be a good time to run lessons on 'How to think' that could cover all of these areas.
louisedennislouisedennis on September 9th, 2010 08:51 am (UTC)
We have a "citizenship" requirement in the national curriculum don't we these days? I would have thought some sort of critical thinking component would fit well there...

There is also some kind of critical thought A level these days, but when I was an Admissions Officer I was told to ignore both it and General Studies.