Reimagined and Revamped. Fighting the spread of nonsense often feels like a Sisyphean task. However, the joy is in making the information available, not the hope of conversion.

Idea for advertizing regulation

Its not hard to find nonsense being advertised as science. The list is repeated over and over again at this blog and at others. Why is it allowed? Why can there be some smarmy looking guy on TV late at night talking about the size and firmness of, yeah...not that...but poop!?! How come Kevin Trudeau can pedal complete nonsense and claim conspiracy theories to promote complete nonsense?

Even though the FDA regulates what can be said about a product regarding health benefits, it truly doesnt seem to be enough. Vaccine scares get started by results of poorly conducted, small population studies. The list is truly endless.

My poor wife told me that Teflon, one of the most inert and robust materials ever created is dangerous and could harm our kid. I sighed (never do that, its a bad response). I was not at all frustrated at her, I was frustrated at the state of things. She read in some crappy magazine and from her mother than when a pan is scratched that the teflon may come off and poison our baby. Meanwhile when I actually look at studies concerning this very thing a) nothing comes off of these pots when they are heated and used and b) even if it did, teflon will just pass through the digestion system, they even use teflon as the control in many studies that involve studying the toxicity of materials (citation).

So, what do we do? Here is a proposal, I am totally open to comments and criticism on the system. Its simply a graded system. If you cite the study you have to cite the grade. We'll start at the bottom.

Grade C study
This study is small in scope and population. Further it is non-blinded Often these are the studies most cited becuase they tend to have bias associated with them and generally show the result the folks conducting it want. Almost all woo is supported by grade C studies. Anything that uses a population of 100 or less would be a Grade C study. At best, positive results from a grade C study, would lead to a grade B study. There would be no reason to repeat a grade C study. People who use grade C studies as proof of their claim can be rightly laughed at.

Grade B study
Clearly the conclusions of a grade B study should be relied upon more than a grade C study. A grade B study could come in more than 1 form. For example, it could have way more people in it and be only single blinded or truly randomized, or have some other level of bias removal. Alternatively it could still be a relatively small study, but be double blinded. In either case the results can be relied upon to a degree in which I think you could even allow advertizing for it.

Grade A study
A grade A study is double blinded and has many participants (in the multiple thousands or tens of thousands) that are randomized and selected from a truly representative population. It is likely that very few studies would be grade A, but we can expect that those that are can truly be relied upon to tell the layperson whether not or something is true to the best of our knowledge. This would be the kind of study that drug makers would strive for to truly prove that their drug or treatment is better than placebo or some competing alternative.

I'm not saying I have all the kinks worked out on this, nor that this is a perfect system (what system is?). For example, where does meta-analysis fit in here? I suspect all meta-analysis would fall into Grade B even though the populations can be quite large.

If we could have a system like this, and perhaps a certifying body, maybe the science-weary or science-phobic could have a good tool by which to wean their facts from.

Do you like this idea? Are more grades required? Do you think I have the qualifications right for this? How would you go about getting a system like this started? Let me know.

P.S. please don't post an alarmist list of sites that claim Teflon is dangerous. I know, I know, its perhaps not the Teflon but the residue from manufacturing, etc etc. I've read all that. Dont buy cheap shit cookware, don't microwave non-microwavable plastic, and dont cook with saran wrap on your food. You'll be fine.


On 10/10/08, 9:23 AM , krelnik said...

There actually is such a rating system that was proposed by Alejandro Jadad and his colleagues at Oxford University. It is called the "Oxford scale" or the "Jadad scale" and it rates studies from 0 (very poor) to 5 (rigorous). It is discussed on page 134-135 of the excellent new book "Trick or Treatment" by Ernst & Singh.

Jadad has written several papers on the topic of grading studies that have further info on this.