Posts

, ,

The Need for Increased Rigor in Obesity and Nutrition Research: A Q&A with Dr. David Allison

Doctor David B. Allison is the current dean, distinguished professor, and provost professor at the Indiana University School of Public Health-Bloomington. Prior to Indiana University, Allison was a distinguished professor, Quetelet Endowed Professor, and director of the NIH-funded Nutrition Obesity Research Center (NORC) at the University of Alabama at Birmingham (UAB). Allison was appointed director of the NORC in 2003 and served until 2017. Allison has published more than 500 scientific papers with research interests including obesity and nutrition, quantitative genetics, clinical trials, statistical and research methodology, and research rigor and integrity.

In addition to his primary appointments, Allison is a co-director for two NIH-funded “Short Courses” on obesity research held in Birmingham, Ala., during the summer. Dr. Allison’s “Short Course on Mathematical Sciences in Obesity Research” is going on its fifth consecutive year, while the “Short Course on Strengthening Causal Inference in Behavioral Obesity Research” is coming up on its fourth consecutive year. These interdisciplinary courses convene a cadre of expert faculty members who teach on various aspects of obesity research, covering economics, epidemiology, statistics, genetics, and much more. These courses are oriented toward investigators who want to increase the rigor in their approach to obesity research, and they bridge various disciplines in which obesity research is performed. Allison took the time to answer a few questions regarding the ability to better approach obesity and nutrition research.

You have noted that the rigor of obesity research has been lacking. Has there been a shift in recent years?

I don’t have unequivocal data as to whether there has been a shift in recent years in obesity research or research overall. I have a hypothesis though, which is when you look within any one journal, research is getting ever more rigorous, whereas when you look across all journals, it may be getting less rigorous because of the influx of new journals.

If you take a journal like the American Journal of Clinical Nutrition, they keep getting more rigorous. That is in part because the editor-in-chief, Dr. Dennis Bier, has a very strong commitment to that, and he has built an associate editorial board who shares the commitment. Meanwhile, other journals keep springing up that are not as rigorous. So for the overall quality of the literature, I am not sure if it is going up or down, since you have these competing factors.

One of the things my colleagues and I are writing a paper on now is the childhood obesity intervention literature, which seems to be particularly susceptible to distortion. We hypothesize that this has to do with feelings of zealousness – the idea that childhood obesity is such a problem and it must be addressed. To come out after an expensive and effortful intervention and say, “Guess what, I did an intervention and it just didn’t work, so let’s move on.” People just don’t want to say that. They want to instead say, “But it must work, we can’t tell people not to do this, especially if we don’t have something better, so let’s twist and bend and ‘find a pony in there’.” We see a lot of “spin” in these things, and that is an area where things seem to have become worse.

What are your suggestions to researchers in the field to increase conscientiousness in limiting and being transparent about shortcomings in the quality of the research produced?

I think there are different aspects to it. Some aspects to it, and perhaps related to what I was saying about the childhood obesity literature, is that people, often again with good intentions, are bending the truth. I think that we need to continually remind ourselves that we are scientists and reflect on why we all got into science in the first place. To be a scientist means to pursue truth through the scientific method. We have to affirm that speaking the unvarnished truth is an uncompromisable imperative. Commitment to one’s identity as a scientist is something to be held dear.

Then, I think there are some things that are more skill-level. Many errors I see – and partially because this is my expertise, so I see what I understand and know about – are statistical errors. One of the challenges is that the norm for many years was, and still is, that many scientists should be able to conduct their own statistical analyses. Physicians are generally not trained with this mentality, because they get very little training in statistics. They accept that they will need to go to a statistician – most at least. Whereas if you are trained in a field like nutrition, psychology, physiology, or biology, you get a PhD in that, and you get one or two statistics courses as you earn your PhD, often taught by that same department. The person who teaches that course tends to not be a professional statistician, but rather a physiologist, biologist, or nutritionist who knows a little statistics. What you are getting is kind of an intelligent amateur who is running the statistics for professional research.

If you think about that – it’s the equivalent of me saying that I need to get a kidney surgery and I say, “Well, I have an anatomy book. I know approximately where my kidneys are. I have a bottle of hand sanitizer. I can get some rubber gloves and a pocket knife, and I can do it myself.” Well, no. Just because you have an anatomy book and you know where the kidneys are and you understand the idea of surgery doesn’t mean you are a professional surgeon, and we wouldn’t have you do it. Why take a different view about statistics? Part of what we are currently exploring, since statisticians are in limited supply, is how we can get more professional statisticians to be involved with more papers, and how can we create a culture and an economic situation that would permit that.

Is there an overreliance on observational research in nutrition/obesity studies? If so, why is this the case?

I think there is sometimes a reliance on observational studies for situations in which they are not what I would call “probative.” For example, you can think, “Well, maybe Pokémon GO is going to reduce obesity levels.” No-one has ever looked at it, so sure, go ahead and do an observational study. Do people who start using Pokémon GO lose weight or gain less weight? And that is fine, there is nothing wrong with that. You might even want to replicate it once or twice. But if you say, “Well, now we’ve done that, so let’s do 20 more of those,” then you need to wonder why you need the next 20. Maybe you need one more to confirm it, but not 20 more. What you see is people not shifting out of the observational and into the experimental when it is called for. For example, breakfast consumption, fruit and vegetable consumption, things like that – when people continue to grind on the observational literature long after it is really useful.

You have noted that you see many errors in obesity and nutrition related meta-analyses. How would you caution investigators in interpreting these papers?

I would say, interpret with a grain of salt, particularly if there isn’t a professional statistician on there. The issue is that there is software out there, where it is seemingly easy. You plug in a few numbers, and it spits out a meta-analysis for you. The problem is, you need to know what numbers to plug in. That is where the problem, the challenge, and the mistakes often occur. Particularly, these mistakes seem to occur around variances. I would caution anybody who is going to do a meta-analysis who thinks, “Oh, meta-analyses are easy. I can just get a grad student to grab some papers, write the numbers down, and plug them in some public software.” I would caution people not to do that, but to have a professional statistician involved.

You have published articles criticizing the statistics and assumptions of various academic papers, resulting in their retractions. Can you theorize why these papers are being published in the first place? What are some mistakes that you see most often?

Why they get published in part is because we don’t really have a good system for vetting papers. Many people seem to think that peer review is that system, but I don’t think it was ever realistic to expect that peer review can be the true gatekeeper of papers and can eliminate all mistakes – or even most mistakes. I think peer review just checks if a paper belongs, and then you receive advice. But the peer reviewers don’t have the time and the ability to go through everything the author and investigator did to see if it is correct. That must fall on the investigators themselves. I think many investigators let a lot slip through – some intentional and some unintentional. I think we need to work on both of those things.

What would you recommend to young researchers in the field of obesity and nutrition who would like to improve their ability to identify poor methods and conclusions?

I would say to take our short courses. Read very widely, including interdisciplinary work. Read some work on the physiology of obesity, the genetics of obesity, engineering approaches, computational approaches, nutritional, psychology, medical, and economic approaches, so that you have a broad base to compare things to. I would say to talk broadly and question everything. Question yourself. Question your own ideas. Those are all important things to do.

 

This is part two of a two-part interview with Dr. David Allison.

Advocacy for Health Research

By: R. Alex Coots
For scientists, the benefits of nutrition and health research are immediately apparent. It’s easy for us to see how the general public and policymakers alike can benefit from a better understanding of health and nutrition. Few of us would argue that we need less health research or fewer grants, but this is exactly what’s been happening since the NIH budget doubling ended over a decade ago. Decreasing budgets means fewer studies, and fewer studies means less progress on today’s pressing health problems. To help address this problem, I spent a day on Capitol Hill with professors, patients, and other stakeholders to advocate for a more sustainable and predictable funding schedule for health research.
Given the abundance of high quality research institutions in New York, I thought it would be easy to get legislators to support science. How wrong I would be. At best, congressional staff received us with apathy and at worst, hostility. During one particular meeting, an elected representative went so far as to say “All you people want is more and more and more rather than try to make what you have go farther.” And this was said by someone who co-sponsored the 21st Century Cures Act!
What became clear to me during the meetings was that the science profession was not viewed as one that provides answers to today’s most pressing questions; rather, it was viewed as just another (expensive) special interest group. While many scientists do advocate for use of scientific information in the formation of policy, not many of us advocate for the resources we need to carry out our work. Professor Lawrence Goldstein at UCSD has previously advocated for a phone call with each grant written and each grant reviewed. I’d extend this model to include a call with each paper published so that policymakers can hear the scientific progress being made in their district or state. Ensuring that scientific information is used in policy formation is only part of the advocacy battle. We scientists must ensure that our discoveries are limited by our imaginations, not by a lack of grants.
,

The Path to Policy: ODS Interview

Interview with NIH Office of Dietary Supplements Director Dr. Paul Coates

By: R. Alex Coots

Academia is changing.

Today’s universities increasingly rely on adjunct faculty to teach courses and reserve the coveted full-time academic position for the science superstars. This phenomenon, coupled with decreasing paylines from funding agencies, makes a science career especially challenging to pursue. And that’s not even considering the project difficulties!

The problem has become so pressing that even the NIH has realized it. New initiatives, such as the BEST Innovation Award, aim to ensure that graduate students and post-docs have increased opportunities to expand their skill sets for a future outside of academia.

Policy is one of the many areas that nutrition experts can serve. The current Director of the Office of Dietary Supplements (ODS), Dr. Paul Coates, successfully made the transition from bench research as a geneticist to a career in science policy. He spoke with me about his career and transition to ODS.

What motivated your interest into policy?
I was curious. For all these years, I had been funded to do research by the NIH and other organizations, but what I concentrated most on was my own research. I was pretty naïve when I came to the NIH, not knowing what life was like for people who worked on the government side. There were plenty of them like me, PhD’s in one setting or another, who had come to the NIH to work as extramural program directors.

What are the important skills or knowledge that someone should have when moving into policy?
One of the things I understood was the importance of making connections. My first job at the NIH was focusing on diabetes research efforts. I learned how to work with other people within an institute, and then gradually in other institutes and beyond to achieve common goals. I think the art of science policy is knowing who else works in this field that you can benefit from, and flip it around and ask “How can I help other people benefit from working together with them?” Recognize the talent that’s out there in other organizations.

What advice would you give to students?
You need to pay your dues as a scientist first. You need to understand the scientific method. You don’t have to spend an eternity in science, but you must have spent some time doing it. Author publications and write grants. My observation is that the people best prepared for this kind of experience “get it” about what a scientist does. They must be prepared to critically analyze data and know what to look for in the literature to inform policy.

What types of projects do the AAAS and Milner fellows work on?
The AAAS Science and Technology Policy Fellowship is beautifully designed to encourage people at different levels of experience in science to work closely with federal agencies to learn about the science-to-policy transition. In ODS, we’re recent partners in that program. Fellows are engaged in projects that my office works on. We have a very active role in translating science into policy, but also in identifying research needs.

The Milner fellowship has a different side to it. Jointly funded by ODS and the Beltsville Human Nutrition Research Center, the Milner fellowship brings in one or two people per year for a two-year stint that will allow them to conduct research in one of the labs at Beltsville. At the same time, they participate at ODS in work on science policy.

How do you see ODS changing in the future?
ODS is getting a little older. A fairly urgent challenge is identifying people who can come up behind us and continue to identify opportunities for research—particularly those that have public health implications— and be committed to help tackle them.