Have you ever heard the term “evidence-based?” It’s often used by health experts, authorities (or the media) as code for ‘You really gotta listen to this information because it’s, like, totally true and proven by SCIENCE!” Use the term evidence-based and you immediately claimed the high intellectual/moral ground these days.
I want to give you a little object lesson on why you should maintain a healthy skepticism when someone tells you their information or program is “evidence-based.”
I’m a writer by trade, but a researcher by inclination, and I have become involved with academic researchers on various levels (including being co-author of one published academic paper and one book chapter). So I’ve seen how “evidence” is used, and in some cases, misused.
A few weeks ago I was one of many people who criticized an Australian study that claimed to show that sleep training (AKA Ferberizing, controlling crying, controlled comforting) had no long-term ill effects on children’s development and mental health. The chorus of voices decrying the flaws in that study continues to grow. Currently, I am part of a group working on an article that will, not to put a fine a point on it, thoroughly trash this study for the piece of crap that it is. We plan to send it to Pediatrics, the high-profile journal that published the article.
Honestly, this study has so many flaws I don’t see how any journal could have considered it publishable. The biggest single one is that the researchers can’t actually say what any of the parents in the study did in terms of sleep training their babies (or not). All they know is that one group of parents were advised to try either controlled crying or “camping out” (a gentler, more gradual form of sleep training where parents stay in the room in the early stages of the training), and the other group did not get structured advice on sleep training (although they could have picked up that advice elsewhere.).
One thing we do know is that 40% of the parents in the so-called treatment group chose not to follow the recommended advice. As for the other 60% the researches didn’t report data on what they did, only on what method they chose. Some could have changed their minds. Some might have had relatively easy and quick success, others may have given up on sleep training after a night or two and some might have listened to their baby cry night after night in a dogged attempt to follow the advice they were given. We don’t know. And we don’t know what the parents in the control group did either. Some of them may well have used various sleep training methods for all we know.
The only thing we know is that there were no differences between kids of the parents in two groups five years later: no differences in children’s mental health, patterns or sleep behaviour.
So what can one reasonably conclude from that?
Actually you could conclude one thing. Allowing parents to muddle their way through sleep problems on their own is no better or worse for children than giving them advice which they may or may not have used.
But that didn’t stop the authors, who, I’m told, are very influential in Australia, from claiming that their study is solid “evidence” that sleep training techniques don’t cause harm. Therefore, as they stated in the conclusion to their journal article, “position statements, policy documents, and training curricula” should be updated to “reflect our findings that behavioural sleep training techniques are… safe to use in the long-term.” As I said before, they have proven nothing of the sort. Unfortunately, many, many researchers, medical professionals, media reporter will take them at their word.
I’m used to seeing researchers trying to make the findings look more important than they really are. What really makes my blood boil is the way these researchers are suggesting that their sketchy study should dictate how maternal nurses are trained and what advice should be given to parents who report infant sleep problems.
I doubt that the authors of this study are stupid people. At least, I can’t imagine that they are fool enough to not know how shaky their study is. Yet still they submit it as solid “evidence.” Shameful. If I were the supreme world commissar of research protocols these people would all be out on their asses.
The journal is culpable too, because, by publishing a study that quite clearly pretends to be something it isn’t, they are contributing to what is essentially a lie.
My colleagues and I will do our best to expose the flaws in the study in hopes that it will not be used as the basis for advice to professionals and parents.
Here’s the question for Mom and Dad. Do you think that the health professionals and parent advice crafters who are given the researchers’ spin on this study will all take the time to analyze its quality before following the authors’ advice to update their policies, practices and training manuals?
I doubt it. Most will assume that the findings and advice are solid because they come from established researchers, with PhDs, who got their research published in one of the world’s alpha pediatric journals. Thus the lie will be repeated over and over again by people who don’t realize what they are doing.
That’s how experts lie to you folks. They don’t always lie, of course, but sometimes they do.
Don’t you wish it was easier to tell the difference?