Doctor David B. Allison is the current dean, distinguished professor, and provost professor at the Indiana University School of Public Health-Bloomington. Prior to Indiana University, Allison was a distinguished professor, Quetelet Endowed Professor, and director of the NIH-funded Nutrition Obesity Research Center (NORC) at the University of Alabama at Birmingham (UAB). Allison was appointed director of the NORC in 2003 and served until 2017. Allison has published more than 500 scientific papers with research interests including obesity and nutrition, quantitative genetics, clinical trials, statistical and research methodology, and research rigor and integrity.
In addition to his primary appointments, Allison is a co-director for two NIH-funded “Short Courses” on obesity research held in Birmingham, Ala., during the summer. Dr. Allison’s “Short Course on Mathematical Sciences in Obesity Research” is going on its fifth consecutive year, while the “Short Course on Strengthening Causal Inference in Behavioral Obesity Research” is coming up on its fourth consecutive year. These interdisciplinary courses convene a cadre of expert faculty members who teach on various aspects of obesity research, covering economics, epidemiology, statistics, genetics, and much more. These courses are oriented toward investigators who want to increase the rigor in their approach to obesity research, and they bridge various disciplines in which obesity research is performed. Allison took the time to answer a few questions regarding the ability to better approach obesity and nutrition research.
You have noted that the rigor of obesity research has been lacking. Has there been a shift in recent years?
I don’t have unequivocal data as to whether there has been a shift in recent years in obesity research or research overall. I have a hypothesis though, which is when you look within any one journal, research is getting ever more rigorous, whereas when you look across all journals, it may be getting less rigorous because of the influx of new journals.
If you take a journal like the American Journal of Clinical Nutrition, they keep getting more rigorous. That is in part because the editor-in-chief, Dr. Dennis Bier, has a very strong commitment to that, and he has built an associate editorial board who shares the commitment. Meanwhile, other journals keep springing up that are not as rigorous. So for the overall quality of the literature, I am not sure if it is going up or down, since you have these competing factors.
One of the things my colleagues and I are writing a paper on now is the childhood obesity intervention literature, which seems to be particularly susceptible to distortion. We hypothesize that this has to do with feelings of zealousness – the idea that childhood obesity is such a problem and it must be addressed. To come out after an expensive and effortful intervention and say, “Guess what, I did an intervention and it just didn’t work, so let’s move on.” People just don’t want to say that. They want to instead say, “But it must work, we can’t tell people not to do this, especially if we don’t have something better, so let’s twist and bend and ‘find a pony in there’.” We see a lot of “spin” in these things, and that is an area where things seem to have become worse.
What are your suggestions to researchers in the field to increase conscientiousness in limiting and being transparent about shortcomings in the quality of the research produced?
I think there are different aspects to it. Some aspects to it, and perhaps related to what I was saying about the childhood obesity literature, is that people, often again with good intentions, are bending the truth. I think that we need to continually remind ourselves that we are scientists and reflect on why we all got into science in the first place. To be a scientist means to pursue truth through the scientific method. We have to affirm that speaking the unvarnished truth is an uncompromisable imperative. Commitment to one’s identity as a scientist is something to be held dear.
Then, I think there are some things that are more skill-level. Many errors I see – and partially because this is my expertise, so I see what I understand and know about – are statistical errors. One of the challenges is that the norm for many years was, and still is, that many scientists should be able to conduct their own statistical analyses. Physicians are generally not trained with this mentality, because they get very little training in statistics. They accept that they will need to go to a statistician – most at least. Whereas if you are trained in a field like nutrition, psychology, physiology, or biology, you get a PhD in that, and you get one or two statistics courses as you earn your PhD, often taught by that same department. The person who teaches that course tends to not be a professional statistician, but rather a physiologist, biologist, or nutritionist who knows a little statistics. What you are getting is kind of an intelligent amateur who is running the statistics for professional research.
If you think about that – it’s the equivalent of me saying that I need to get a kidney surgery and I say, “Well, I have an anatomy book. I know approximately where my kidneys are. I have a bottle of hand sanitizer. I can get some rubber gloves and a pocket knife, and I can do it myself.” Well, no. Just because you have an anatomy book and you know where the kidneys are and you understand the idea of surgery doesn’t mean you are a professional surgeon, and we wouldn’t have you do it. Why take a different view about statistics? Part of what we are currently exploring, since statisticians are in limited supply, is how we can get more professional statisticians to be involved with more papers, and how can we create a culture and an economic situation that would permit that.
Is there an overreliance on observational research in nutrition/obesity studies? If so, why is this the case?
I think there is sometimes a reliance on observational studies for situations in which they are not what I would call “probative.” For example, you can think, “Well, maybe Pokémon GO is going to reduce obesity levels.” No-one has ever looked at it, so sure, go ahead and do an observational study. Do people who start using Pokémon GO lose weight or gain less weight? And that is fine, there is nothing wrong with that. You might even want to replicate it once or twice. But if you say, “Well, now we’ve done that, so let’s do 20 more of those,” then you need to wonder why you need the next 20. Maybe you need one more to confirm it, but not 20 more. What you see is people not shifting out of the observational and into the experimental when it is called for. For example, breakfast consumption, fruit and vegetable consumption, things like that – when people continue to grind on the observational literature long after it is really useful.
You have noted that you see many errors in obesity and nutrition related meta-analyses. How would you caution investigators in interpreting these papers?
I would say, interpret with a grain of salt, particularly if there isn’t a professional statistician on there. The issue is that there is software out there, where it is seemingly easy. You plug in a few numbers, and it spits out a meta-analysis for you. The problem is, you need to know what numbers to plug in. That is where the problem, the challenge, and the mistakes often occur. Particularly, these mistakes seem to occur around variances. I would caution anybody who is going to do a meta-analysis who thinks, “Oh, meta-analyses are easy. I can just get a grad student to grab some papers, write the numbers down, and plug them in some public software.” I would caution people not to do that, but to have a professional statistician involved.
You have published articles criticizing the statistics and assumptions of various academic papers, resulting in their retractions. Can you theorize why these papers are being published in the first place? What are some mistakes that you see most often?
Why they get published in part is because we don’t really have a good system for vetting papers. Many people seem to think that peer review is that system, but I don’t think it was ever realistic to expect that peer review can be the true gatekeeper of papers and can eliminate all mistakes – or even most mistakes. I think peer review just checks if a paper belongs, and then you receive advice. But the peer reviewers don’t have the time and the ability to go through everything the author and investigator did to see if it is correct. That must fall on the investigators themselves. I think many investigators let a lot slip through – some intentional and some unintentional. I think we need to work on both of those things.
What would you recommend to young researchers in the field of obesity and nutrition who would like to improve their ability to identify poor methods and conclusions?
I would say to take our short courses. Read very widely, including interdisciplinary work. Read some work on the physiology of obesity, the genetics of obesity, engineering approaches, computational approaches, nutritional, psychology, medical, and economic approaches, so that you have a broad base to compare things to. I would say to talk broadly and question everything. Question yourself. Question your own ideas. Those are all important things to do.
This is part two of a two-part interview with Dr. David Allison.
