An unfortunate sweet tooth for sensationalism

When I taught a science for non-science majors at Cornell University, my favorite lecture was on science in the popular media.  My learning objective was straight forward: I wanted students to pick up a newspaper, flip to the science section, and detect bullshit.

This is such an important life skill because at this point in time, blogs, newspapers, and magazines print unscientific bullshit.  They do it because it sells.  I have never seen an article so neatly package all of the problems that lead to this state of affairs than John Bohannon’s recent “Chocolate Diet” charade.  If you haven’t read it yet, please do.  I’ll wait.

Amazing, right?

An echo of the article, still up on BodyandSoul.com.au
An echo of the article, still up on BodyandSoul.com.au

The study

Here’s the breakdown: Diana Löbl and Peter Onneken are working on a “diet science” documentary, and want to show how diet news can get reported.  So they enlist Johannes to conduct a shoddy study.  Participants will either make no dietary changes, go on a diet, or go on a diet with a bar of chocolate.  By only including 15 people in their study, and measuring a wealth of factors, they were guaranteed to find that the chocolate dieters fared better in some way than the other groups.  The finding would be meaningless, but it would be “statistically significant” – within the confines of their prohibitively small study.  They lucked out, and the 5 chocolate eaters lost weight.

How to Lie With Statistics, by Daryl Huff.
How to Lie With Statistics, by Daryl Huff.

Next, they needed to publish their study somewhere.  Fortunately for Johannes, it’s getting harder and harder to distinguish “fake” publishers from real ones.  Real journals will conduct real peer review, where third-party scientists must evaluate the article before it can be published.  Due to the intentionally poor design of this study, it (hopefully) wouldn’t pass peer review in any credible journal.

A fake publisher, on the other hand, will not conduct review.  Instead, it will collect a publication fee, then immediately accept and publish the article without review.  Bohannon’s paper was easily accepted into the International Archives of Medicine without any peer review.

Once the paper was published, it took off in the science and diet sections of many, many magazines, blogs, and newspapers.  You can easily understand why- “Chocolate helps you lose weight” is the sort of headline The Daily Mail and its ilk dream about.

The Assignment

Back to my students.  To consider a news piece credible, students first needed to ask three questions.

1.  Is the finding reported by a scientist?  Who?  Does the scientist work for a biased source?

2.  Was the finding published in a peer-reviewed journal?

3.  Can you find the article?  Do the conclusions the authors draw in the abstract reflect what is reported in the newspaper?

How would my students fare if they were given this article?  It turns out, not so well.  Here are my answers, working from Bohannon’s press release.

Question 1:  Is the finding reported by a scientist?  Does the scientist work for a biased source?

Johannes Bohannon is presented as the research director of the nonprofit Institute of Diet and Health, a fictional institute.  They have a website that is intentionally minimalist.  It doesn’t look incredibly legitimate, but it also doesn’t scream crazy.  I imagine most students would give it a pass, despite the possibility that, say, this fictional institute is funded by Nestle.

Question 2:  Was the finding published in a peer-reviewed journal?

Yes of course it was!  Here’s the real problem predatory journals present to the common reader going forward.  The International Archives of Medicine (IAM) is a legitimate enough sounding journal.  The Journal’s website has an ISSN, citation metrics, and assurances that it is included in the JournalGuide whitelist of reputable titles.  As Bohannon reports, this journal used to be reputable, as it was published under BioMed Central, a trustworthy open-access publisher.

In short, one would need to do some serious digging to realize the journal is bogus.

Question 3.  Can you find the article?  Do the conclusions the authors draw in the abstract reflect what is reported in the newspaper?

Yes and yes.  This is an easy one, because Bohannon’s wrote both the press release and the article.

The solution

It’s clear that we have a problem here.  What prevents someone from doing poor work (or fabricating work) and “publishing” the work in predatory journals?  Nothing.  And unless some sort of watchdog agency is set up to accredit peer-reviewed journals, this will not change any time soon.  If real scientists can’t distinguish a quality journal from a predatory one, what chance do non-scientists have?  As it is, I find Beall’s list of questionable publishers to be the most useful resource for investigating journals.  But this is a list built by a science librarian who saw a problem, and can’t be expected to be definitive or exhaustive.  The JournalGuide, cited on IAM’s website, appears to be a reasonable resource in its own right (it’s listed in this Nature article on resources for evaluating journals), but obviously it’s not perfect if IAM is listed.

If policing publishers isn’t the answer, then what can be done?  Bohannon’s piece makes one thing abundantly clear: the popular media did not do their homework on this piece.  Many magazines ran the story without contacting Bohannon at all.  If he was contacted, he was never contacted by someone who knew enough about science to ask the right questions.  In short, science journalism lack skepticism.  It seems that most magazines will run an article without asking the three questions I listed above.  And even if they do, it’s clear that a more careful tact must be taken in the future to prevent spreading misinformation.

Until that day arrives, I urge you, the reader, to bring a healthy dose of skepticism to every new miracle finding you read, especially related to human health, sexuality, and diet.  Do your homework, because no one else is doing it for you.

The Tannosome

An article posted on the Scientific American blog caught my attention recently.  Researchers have discovered the existence of a new organelle, the tannosome, in plants.  It is this organelle that produces tannins, of great interest to wine lovers.  The scientific article in question, posted to my scoop.it, was officially published in the Annals of Botany on September 25th 2013, but it barely created a ripple in the news cycle.  Why not?

Organelles are extraordinarily important in biology.  An organelle is, by definition, a membrane-bound compartment within a (eukaryotic) cell that performs some specialized function.  Much like each organ within the body performs a specific task, organelles within the cell include the DNA-replicating and transcribing nucleus, the “energy factory” mitochondria, and the “storage center” vacuole.  (Most) plants possess chloroplasts, organelles which house the photosynthetic machinery required to harness energy from light.

A diagram of the organelles in a typical animal cell.

Tannins are known as secondary metabolites.  This term covers a huge range of array of chemically diverse molecules, lumped together for not being “primary” metabolites, such as proteins, sugars, or ribonucleic acids (such as DNA).  Often, but not always, secondary metabolites are important for “non-primary” processes, such as herbivory defense, pheremones, or pigmentation.  Tannins can be found in most plants, and may function in herbivory defense, pathogen defense, and UV-protection.  Humans are quite fond of tannins for their role in flavoring beverages, particularly wine.

The most digestible evidence they offer for the tannosome being a novel organelle is a set of TEM images (below).  On the left (A) is a standard chloroplast, and the middle and right images (B, C), show the differentiated form at different stages.

Image
From Brillouet 2013

Prior to this publication, there were reports of Tannins being found in chloroplasts- but no one had the idea that these were specialized unique organelles.  Similarly, the authors point out that chloroplasts still produce a number of secondary metabolites (phenolics, for example) despite this new organelle.  From what I can gather, the authors don’t make a case for this organelle producing anything other than tannins.

So why the media silence?  One article notes that with a lead-in to tea and wine, it really should have caught some buzz.  Not to mention the discovery of a new organelle (albeit derived from a known organelle) is noteworthy in anyone’s book, even if it doesn’t have obvious implications past tannin biosynthesis.  For whatever reason, it just isn’t the discovery that ignites people’s interest.  Science news is generally dominated by medicine, astronomy, sex, nutrition, and the cute or gross.  Perhaps if the finding had more implications for alcohol production or consumption, the media would have taken notice.  Maybe when tannosomes are harnessed to produce a new altered-tannin  wine, the world will take notice.  But without this study, one would not even know where to begin such a task.

Sources:

Brillouet JM, Romieu C, Schoefs B, Solymosi K, Cheynier V, et al. (2013) The tannosome is an organelle forming condensed tannins in the chlorophyllous organs of Tracheophyta. Ann Bot 112: 1003–1014.

Organelle image: from wikimedia commons, http://upload.wikimedia.org/wikipedia/commons/thumb/1/11/Animal_Cell.svg/800px-Animal_Cell.svg.png